presentation
DESCRIPTION
TRANSCRIPT
April 10, 2023 IDS SEPG
PROCESS … The GLUE that holds
Software Projects Together
A Focus on Software Testing
Dennis A. MorrisonSenior Principal EngineerDD(X) System [email protected]
April 10, 2023 IDS SEPG
Presentation Outline
• Software Process Overview
• Software Testing Overview
• Source Code Control
• Trouble Reporting and Bug tracking
• Summary
April 10, 2023 IDS SEPG
Software Phases(Waterfall Model)
Requirements
Design
Conceive
Code
Test
Build
Deploy
SupportDeliver
April 10, 2023 IDS SEPG
Software Phases(Process Glue)
Requirements
Design
Code
Test
Deploy
Support
Requirements Review – Are the Requirements testableArtifacts – Review documentation; Requirements placed under documentation control
Design Review – Does the design support requirements? Artifacts – Review documentation
Code Review – Does the code support requirements and implement the design??Artifacts – Review documentation; Code under source code control
Test Review – Test plans Test Results – Did the tests pass?Artifacts – Review documentation; Test plans; System Trouble Reports (STR)
Test Review – Customer Acceptance Test Test Results – Did the tests pass?Artifacts – Review documentation; STRs
Customer Survey – Is the system meeting customer expectationsArtifacts - Software Updates; STRs
April 10, 2023 IDS SEPG
Software Phases and Loops
Requirements
Design
Code
Test
Deploy
Support
Phases where most rework occurs
Design Errors
New feature requests or feature refinement
Software Bugs
Software Bugs
New Feature Request
Requirement Clarification or changes
April 10, 2023 IDS SEPG
Software TestingAn Art or a Science ?
• Testing is a phase of software development that can become, or seem, endless– Classically, the most expensive part of the project
– Schedule overruns are often attributed to the testing cycle
– Testers are usually squeezed into smaller and smaller test cycles
• How much testing is enough?• How do you know when you are finished testing?• How good is the product when you are finished?
April 10, 2023 IDS SEPG
Testing Has Many Stages
Test
Unit Test White Box Testing
Integration TestGrey Box Testing
System Test Black Box Testing
Acceptance TestSubset of Black Box Testing
April 10, 2023 IDS SEPG
• Phase = Unit Test
• Testing is focused on internal operations of the software component– Inputs/outputs
– Logical operations
– Limit testing
– Limited functional testing with stubs
• Testing methodologies– Formal testing with test plan and review
– Ad-hoc random
– Typically, tests are manually executed
Unit Test = White Box Testing
April 10, 2023 IDS SEPG
• Phase = Integration Test• Testing is focused on functional aspects of an integrated
set of software components– Inputs/outputs– Logical operations from outside of the software components– Limit testing (induced error paths from external stimuli)– Stubs are replaced by real software as they become available
• Testing methodologies– Formal testing with test plan and review– Semi-automated and manually executed– Automation yields regression test capability
Integration Test = Grey Box Testing
April 10, 2023 IDS SEPG
• Phase = System Test• Testing is focused on system level requirement verification
– User Interface (Human to Computer Interface)– Requirement verification– Limit and normal operational testing– Stress and longevity testing
• Tests are executed on target hardware and software• Testing methodologies
– Formal testing with test plan and review– Typically, tests are semi automated and manually executed– Automation yields regression test capability
System Test = Black Box Testing
April 10, 2023 IDS SEPG
• Phase = Acceptance Test• Testing is focused on externally visible features
– User Interface (Human to Computer Interface)– Operational focus– Interfaces with external systems in customer domain
• Other collaborative systems• Customer network, etc.
• Tests are executed on target hardware and software usually at the customer’s site
• Testing methodologies– Formal testing with test plan and review– Typically, tests are manually executed
Acceptance Test = Subset of Black Box Testing
April 10, 2023 IDS SEPG
Some Software Testing Models• Developer level testing
– Applicable to small teams
• Community testing– Beta versions of a product are released to a volunteer
community for functionality and user level testing• Used by gaming community prior to product releases
• Microsoft uses this approach as one of their test phases
• Collaborative testing– Developers and customers test together
• Project level testing– Used by most large projects
April 10, 2023 IDS SEPG
When Should Software Testing Begin?
• The software test cycle should begin during the project requirements phase
• Requirements review should result in the first level of test specifications
• Similar to the “Agile” software development process, test cases should be written and reviewed before the software is developed
April 10, 2023 IDS SEPG
When is Testing Finished?
• When the schedule says you’re finished?
• When you run out of money?
• When the boss says to release it?
• When the tester’s say you’re done?
• When the customer says you’re done?
• When the law suits are filed?
April 10, 2023 IDS SEPG
Requirements Verification• Raytheon uses requirement verification and
statistical mechanisms to determine when testing is complete
• Requirements are decomposed and allocated to software components
• Requirements are “sold off” with during formal testing events with stakeholders (customers) present
April 10, 2023 IDS SEPG
Something you have
Token Reader
Something you know
Pin Reader
Something you are
Biometric Reader
Requirement Decomposition
Req 1
Req 1.3Req 1.2Req 1.1
Req 1.3.2Req 1.3.1Req 1.2.1Req 1.1.2Req 1.1.1
Operational RequirementParent requirement
decomposes into three child requirements The system shall
provide three factor user authentication
April 10, 2023 IDS SEPG
Requirements Role in Testing
• Software is tested against requirements– Before a parent requirement can be “sold off”,
all children requirements must be fulfilled– Before a child requirement can be “sold off”,
all of its siblings must themselves be “sold off”– The failure of any requirements will prevent the
parent from being fulfilled
April 10, 2023 IDS SEPG
Something you have
Token Reader
Something you know
Pin Reader
Something you are
Biometric Reader
Requirement Decomposition
Req 1
Req 1.3Req 1.2Req 1.1
Req 1.3.2Req 1.3.1Req 1.2.1Req 1.1.2Req 1.1.1
Operational RequirementParent requirement
decomposes into three child requirements The system shall
provide three factor user authentication
Req 1Req 1
April 10, 2023 IDS SEPG
Requirements Traceability• Software test cases are built from system use cases
– Scenarios (rainy day and sunny day) are used to test various threads through the software
– Every scenario is traced to parent requirements• A scenario may have more than one parent requirement
– Test cases must demonstrate interface level testing• All system and software level interfaces are documented and tracked• These are known as IRLLs, IRLEs, and IRLMs
– IRLL = Interface Requirement Logical Level– IRLE = Interface Requirements Electrical Level– IRLM = Interface Requirements Mechanical Level
• Test cases and requirements are stored in DOORS
April 10, 2023 IDS SEPG
Software Source Code Control• The ability to save and retrieve versions of
software is a key component to success in medium to large software projects– Software can be “versioned” for new features
or software load lines– Changes made to software modules due to bug
fixes can be isolated– Enables developers to work on next release
while maintaining current versions– Software builds can include or exclude changes
April 10, 2023 IDS SEPG
Using Source Control SystemsSource Code
Version 1
Source Code Control System
Source CodeVersion 1
Put (ModA V1)get(ModA V1)
Source CodeVersion 2
put(ModA V2)
Mod A-V1.1Mod X-V7Mod Z-V2
SW BuildSpecification
SoftwareBuild
Source CodeVersion 1.1
Source CodeVersion 1
get(M
odA V
1)
put(M
odA
V1.
1)
New Feature
Bug Fix
April 10, 2023 IDS SEPG
Trouble Reporting and Bug Tracking• Software bugs are documented using formal mechanisms
– CMTS – Change Management Tracking System (MRs)– WSTR – Web System Trouble Report (STRs)
• Bugs are considered addressed when software has been modified, tested, and submitted to the load-line
• Bugs are considered fixed when they have been verified• Bug fixes are pulled into the software load via the source
code control system• Bug reports and time based metrics guide management in
software release decisions
April 10, 2023 IDS SEPG
Software Defect Trend Example
20
60
90
110117
132
144
152 155 157160 162
28
14
24
5460
6670
85
95
115
145
18
52
76
86
63
7278
82
70
62
45
17
0
20
40
60
80
100
120
140
160
180
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Cumulative Reported Cumulative Closed Delta
Steep Increase in bugs found
Bug discovery rate slowing
Developer push to close bugs
Developers on vacation?
End Game push. Nearing delivery
Closing the final gap
April 10, 2023 IDS SEPG
Test Methods
• Manual– Usually applied during unit test
• Semi Automated– Applied during integration test
• Fully Automated– System test
April 10, 2023 IDS SEPG
Test Automation• Test automation can provide a project with
huge dividends• Test scripts
– Drive test automation engines– Can be used in unit testing– Can be reused by multiple test organizations
• Regression tests can be constructed from reusable test scripts
April 10, 2023 IDS SEPG
Metrics Drive Behavior• An organization or individual’s behavior is
directly related to the metrics they are measured by– Defects found and fixed– On time delivery– Productivity
• SLOC counts• Defect rate at delivery
– Complexity indices– Customer satisfaction
April 10, 2023 IDS SEPG
Best Practices?• No real silver bullets• Process is the glue that holds a project
together• The right solution depends on the project
but is often a combination of methods/processes and tools
• Just remember that customer satisfaction trumps all!!
April 10, 2023 IDS SEPG
BACKUP SLIDES
April 10, 2023 IDS SEPG
In the world of software development, process is the “glue” that holds the project and team together. Too little, the pieces fall apart; too much and it can become a sticky quagmire resulting wasted effort, time, and money. The challenge facing many projects is finding just the right balance point on the process continuum. As a project progresses through its life cycle, the balance point will shift and the project will need to continuously review and adjust to maintain optimal performance. A typical software project life cycle is comprised of six distinct phases: Requirements, Design, Code, Test, Deployment, and Support. Most projects place a significant amount of emphasis on the front-end phases of the life cycle e.g. requirements, and design. However, industry data clearly demonstrates that the largest percentage of a projects budget is spent during the final phases, namely test, deployment, and support. These phases encapsulate much of the “rework” experienced by the project. This talk will describe the process philosophies and tools used by Raytheon to tackle large-scale multisided software development projects.
ABSTRACT
April 10, 2023 IDS SEPG
Tools
• The right tool can result in huge pay backs
• Source Code Control Tools
• Trouble Report tracking
• Stress testing
• Performance testing
April 10, 2023 IDS SEPG
Testing Methodologies• Ad-hoc
– Non structured testing with or without test artifacts
– No test plans
– Non repeatable tests
• Formal testing– Test plans
– Test Artifacts
– Defect tracking
April 10, 2023 IDS SEPG
Software Phases(Spiral Model)
Spiral Development:
Experience, Principles, and Refinements Spiral Development Workshop
February 9, 2000
Barry Boehm, edited by Wilfred J. Hansen
April 10, 2023 IDS SEPG
The NATO Sea Sparrow Surface Missile System (NSSMS) is a quick reaction,
point defense system designed to protect the ship it is on. It is very,
very effective against enemy aircraft, low flying sea skimming missile and
even bomb laden inflatable boats. It was first developed by Raytheon in
the '70's and installed on over 150 ships world-wide. In the mid '90's the
Navy contracted Raytheon to re-architect the system so that it would be
capable of firing the Navy's newest missile the Evolved Sea Sparrow Missile
(ESSM). The Navy's brand new nuclear powered aircraft carrier , the USS
Ronald Reagan, has 6 Seasparrow radars & 3 Seasparrow missile
launchers. The movie shows the Navy doing Qualification Testing on these
systems.
April 10, 2023 IDS SEPG
Software Phases(Process Glue)
Requirements
Design
Code
Test
Deploy
Support
Requirements Review – Do the requirements support the customer’s intent? Are they testable? Are they complete (sunny day/ rainy day)Artifacts – Review documentation; Requirements placed under documentation control
Design Review – Does the design support requirements? Artifacts – Review documentation
Code Review – Does the code support requirements and implement the design??Artifacts – Review documentation; Code under source code control
Test Review – Test plans – Do the tests adequately exercise the requirements?Test Results – Did the tests pass?Artifacts – Review documentation; Test plans; System Trouble Reports (STR) or Modification Requests (MR)
Test Review – Customer Acceptance Test Test Results – Did the tests pass?Artifacts – Review documentation; System Trouble Reports (STR) or Modification Requests (MR)
Customer Survey – Is the system meeting customer expectationsArtifacts - Software Updates; STRs