class 1

04/10/2304/10/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved

11

CES 592 Telecommunications CES 592 Telecommunications System Product VerificationSystem Product Verification

Sonoma State UniversitySonoma State University

Class Lecture 1 : IntroductionClass Lecture 1 : Introduction

Fall 2004Fall 2004


22

InstructorsInstructors

France AntelmeFrance Antelme Ario BigattiniArio Bigattini Jaseem MasoodJaseem Masood Steven WoodySteven Woody Coordinator: Prof Ravi KumarCoordinator: Prof Ravi Kumar

Website:Website:


33

GeneralGeneral All lecture material will be downloadable from the All lecture material will be downloadable from the

website the day after the lecturewebsite the day after the lecture Instructors will be available for the half an hour prior Instructors will be available for the half an hour prior

to the onset of lecture every week unless by prior to the onset of lecture every week unless by prior arrangement via e-mail.arrangement via e-mail.

Class mark will be made up of three projects each Class mark will be made up of three projects each counting 20% and a final project counting 40% of the counting 20% and a final project counting 40% of the overall mark.overall mark.

Course guidelines are posted on the website.Course guidelines are posted on the website. This is a new course and there may thus some This is a new course and there may thus some

changes from the original course curriculum as the changes from the original course curriculum as the semester progressessemester progresses


44

Telecom Product Verification Telecom Product Verification IntroductionIntroduction

Overall High Level Flow ChartOverall High Level Flow ChartElements of Product VerificationElements of Product Verification

–Hardware VerificationHardware Verification–Software VerificationSoftware Verification


55


This course focuses on the following:This course focuses on the following: Verification of a new Telecom product prior to Verification of a new Telecom product prior to

release from Engineering Development to Production. release from Engineering Development to Production. We do We do notnot consider Production Testing to be a consider Production Testing to be a component of Product Verificationcomponent of Product Verification

“ “Black Box” Box testing techniques with some Black Box” Box testing techniques with some application of “Grey Box” Testing techniquesapplication of “Grey Box” Testing techniques

A A practicalpractical approach to testing techniques and approach to testing techniques and methodologiesmethodologies


66

Telecom Product Verification Telecom Product Verification Testing techniquesTesting techniques

Black Box Testing:Black Box Testing: Also known as Also known as functional testingfunctional testing

A A softwaresoftware testing technique whereby the internal workings of testing technique whereby the internal workings of the item being tested are not known by the tester. For example, the item being tested are not known by the tester. For example, in a black box test on a software design the tester only knows in a black box test on a software design the tester only knows the inputs and what the expected outcomes should be and not the inputs and what the expected outcomes should be and not how the program arrives at those outputs. The tester does not how the program arrives at those outputs. The tester does not ever examine the programming ever examine the programming codecode and does not need any and does not need any further knowledge of the program other than its specifications.further knowledge of the program other than its specifications.

Reference http://www.webopedia.com/TERM/B/Black_Box_Testing.htmlReference http://www.webopedia.com/TERM/B/Black_Box_Testing.html


77

Black Box TestingBlack Box Testing

Equipment Under

Test(EUT)

External stimulus

External stimulusEx

tern

al st

imul

us

Exte

rnal

stim

ulus

Physical layer:• Input signal• Optical• Line

Operating Environmental:• Vibration• Temperature

Additional Stresses:• Environmental

Software functionality:• Logical layer• Control layer• Management layer

Output

Ou

tput

Output

Ou

tput

Entire EUT (“Black Box”) is subjected to stimuli – not individual circuit cards within the unit.


88

Telecom Product Verification Telecom Product Verification Testing TechniquesTesting Techniques

White Box Testing:White Box Testing: Also known as Also known as glass boxglass box, , structuralstructural,, clear boxclear box and and open boxopen box testing testing. . A A softwaresoftware testing technique whereby explicit testing technique whereby explicit knowledge of the internal workings of the item being knowledge of the internal workings of the item being tested are used to select the test data. Unlike tested are used to select the test data. Unlike black box testingblack box testing, white box testing uses specific , white box testing uses specific knowledge of programming knowledge of programming codecode to examine outputs. to examine outputs. The test is accurate only if the tester knows what the The test is accurate only if the tester knows what the program is supposed to do. He or she can then see if program is supposed to do. He or she can then see if the program diverges from its intended goal. White the program diverges from its intended goal. White box testing does not account for errors caused by box testing does not account for errors caused by omission, and all visible code must also be readable.omission, and all visible code must also be readable.

Reference: http://www.webopedia.com/TERM/B/Black_Box_Testing.htmlReference: http://www.webopedia.com/TERM/B/Black_Box_Testing.html


99

Telecom Product Verification Telecom Product Verification Testing techniquesTesting techniques

Grey Box Testing:Grey Box Testing:A A softwaresoftware testing technique whereby a combination testing technique whereby a combination of Black Box testing and white box testing of Black Box testing and white box testing approaches are utilized. A partial knowledge of the approaches are utilized. A partial knowledge of the internal workings of the item being tested is known to internal workings of the item being tested is known to the tester. Limited stimuli are applied to the internal the tester. Limited stimuli are applied to the internal workings of the item under test and the remaining workings of the item under test and the remaining test focus is on the Black Box approach applying test focus is on the Black Box approach applying inputs to the item under test and observing the inputs to the item under test and observing the outputs.outputs.


1010

The Product Development CycleThe Product Development Cycle

System Spec

Software

Spec

New Product

Idea

New Product

Idea

Software Development SW Unit Test

HW-SWIntegrationHardwareSpec Hardware

DevelopmentHW Unit T

est

Product Verification

Engineering Developmentfunctions

Product Verificationfunctions

Customer & marketDriven inputs

Product LineManagement &

Engineeringinputs

Release to

manufacture


1111

Product Verification PhaseProduct Verification Phase

Formal Product Verification Phase

Software Verification

HW-SWIntegration

Test

HW Compliance & Agency approvals

Release to Production

HW Stress Testing

HW Standards/Reqts testing

Software Verification

VolumeProduction


1212





1313


We will also apply the terms We will also apply the terms “Black Box”“Black Box” and and “White Box”“White Box” testing loosely to testing loosely to Hardware Hardware VerificationVerification i.e. The entire Equipment under i.e. The entire Equipment under test is subjected to external stimuli and the test is subjected to external stimuli and the resulting outputs will be verified according to resulting outputs will be verified according to the product requirements and standards.the product requirements and standards.


1414


Hardware Verification:Hardware Verification: Unit Test PhaseUnit Test Phase First prototypes are typically tested by the HW First prototypes are typically tested by the HW

development Engineering team development Engineering team The Hardware feature is tested at the The Hardware feature is tested at the subs-ystemsubs-ystem level to level to

ascertain performance of all subsystems to the design ascertain performance of all subsystems to the design specification. These are areas such as transmitters, specification. These are areas such as transmitters, receivers, PLL’s, busses, Microprocessors and memory receivers, PLL’s, busses, Microprocessors and memory

Analogous toAnalogous to Software “White Box ”testing Software “White Box ”testing approachapproach Usually execute a full a subset of formal test plan and, in Usually execute a full a subset of formal test plan and, in

addition execute a series of sub-system level design tests addition execute a series of sub-system level design tests to ensure that all critical subsystems are functioning within to ensure that all critical subsystems are functioning within the design specification and design parameters.the design specification and design parameters.

Successful completion of this Test Phase as measured Successful completion of this Test Phase as measured against the against the exit criteriaexit criteria for this phase, result in entry into for this phase, result in entry into the the formal test phaseformal test phase


1515


Hardware Verification:Hardware Verification: Formal Test PhaseFormal Test Phase Equipment Under Test (EUT) is verified against the Product, Equipment Under Test (EUT) is verified against the Product,

Industry and Regulatory requirementsIndustry and Regulatory requirements The Hardware EUT is tested at the platform or system level The Hardware EUT is tested at the platform or system level

i.e. All constituent hardware features (“cards” or “circuit i.e. All constituent hardware features (“cards” or “circuit packs”) integrated into a functional system chassis packs”) integrated into a functional system chassis

Analogous toAnalogous to Software “Black Box”testing Software “Black Box”testing approachapproach Verify that the entire Verify that the entire SystemSystem (i.e router, switch etc.) (i.e router, switch etc.)

functions as per the design specificationsfunctions as per the design specifications Usually execute a full set of the tests required to ensure Usually execute a full set of the tests required to ensure

functionality according to the Hardware feature functionality according to the Hardware feature specificationspecification

Successful completion of this Test Phase as measured Successful completion of this Test Phase as measured against the against the exit criteriaexit criteria for this phase, result in release of for this phase, result in release of the Hardware platform to Productionthe Hardware platform to Production


1616

Elements of Hardware VerificationElements of Hardware Verification

Hardware Verification

Compliance &

Agency Approvals

Stress TestingHALT/HASS

Standards basedTesting

Physical LayerLogical Layer

EMCSafetyNEBSTelecom

Design Stress TestingAccelerated Life-cycle Testing

(Beyond normal operating limits)Where does it break?


1717


Hardware Verification:Hardware Verification: Standards based testing-Standards based testing- Equipment Under Test (EUT) is verified by own Equipment Under Test (EUT) is verified by own test organization (i.e. self certified) against the Product, Industry requirements for performance of test organization (i.e. self certified) against the Product, Industry requirements for performance of the following:the following:

Physical layer– Transmitters

Eye pattern, Pulse mask Line frequency Line Build-out Rx Sensitivity, Dispersion/Reflection power penalty Laser line width, Side Mode suppression etc Tx Power output Jitter generation and jitter tolerance

– Receivers Rx Sensitivity, Dispersion/Reflection power penalty

– Synchronization systems ( Where applicable) Wander (<10Hz) & Jitter (>10Hz) performance: MTIE/TDEV Interface Jitter/Wander (Optical and Line)

Logical and Higher Layers:– Framing– Alarming


1818


Hardware Verification:Hardware Verification: Compliance Testing:Compliance Testing: Equipment Under Test (EUT) Equipment Under Test (EUT) is verified against the Industry and Regulatory requirements and certified is verified against the Industry and Regulatory requirements and certified compliance by compliance by independent laboratories:independent laboratories:

EMI/EMS/ESD: – EM radiated & Conducted emissions– EM radiated and conducted susceptibility– Electrostatic Discharge tolerance

Safety: e.g. Underwriters Laboratory etc.

Telecom interface: i.e performance certification for inclusion in the Public network

Power and grounding: Verification of Power and Grounding layout to regulatory requirements

Environmental: Very coarse Industry environmental operation/storage under hi/lo temp, airborne contaminants, Shock and vibration, Flame resistance, Heat generation etc


1919


Hardware Verification:Hardware Verification: Stress Testing:Stress Testing: Equipment Equipment Under Test (EUT) is subjected to external stresses Under Test (EUT) is subjected to external stresses to excite failures to excite failures

HALT: Highly Accelerated Lifecycle Tests: EUT subjected to stress to determine failure modes

HASS: Highly Accelerated Stress Screen: EUT is subjected to elevated stresses in order to ensure that there is high yield in production

The point of this type of test activity is to use elevated stresses in order to excite failure modes of very low probability of occurrence in a short period of time. The failure mode is remedied and the remedy is used to drive higher yields in production and lower customer failure rates


2020





2121


Software Verification Test Phases Defined:Software Verification Test Phases Defined: PlanPlan the test strategy the test strategy DesignDesign the test cases the test cases ExecuteExecute the test cases the test cases ReportReport test results test results

Testing schedules are always a balance Testing schedules are always a balance between quality and time-to-marketbetween quality and time-to-market


2222


Test ExecutionTest Execution Phases Defined Phases Defined Unit TestingUnit Testing Integration TestingIntegration Testing Feature TestingFeature Testing Systems TestingSystems Testing


2323


Software Verification: Software Verification: Unit Test PhaseUnit Test Phase First actual testing performedFirst actual testing performed Can be performed by developer on own Can be performed by developer on own

codecode Tests the new feature in isolation from Tests the new feature in isolation from

other featuresother features White boxWhite box testing testing approachapproach Check for memory leaks, resource Check for memory leaks, resource

depletiondepletion Fault Insertion testingFault Insertion testing


2424


Test ExecutionTest Execution Phases Defined Phases Defined Unit TestingUnit Testing Integration TestingIntegration Testing Feature TestingFeature Testing Systems TestingSystems Testing


2525


Software Verification: Software Verification: Integration Test PhaseIntegration Test Phase Should be performed by an independent testerShould be performed by an independent tester Tests major functionality of the new feature Tests major functionality of the new feature

immediately after the code branch has collapsed immediately after the code branch has collapsed into the main branchinto the main branch

Should verify end-to-end functionality Should verify end-to-end functionality as soon as as soon as possiblepossible

Should be quick and frequent test – automated Should be quick and frequent test – automated daily/weekly “sanity” or “smoke” testdaily/weekly “sanity” or “smoke” test


2626


Test ExecutionTest Execution Phases Defined Phases Defined Unit TestingUnit Testing Integration TestingIntegration Testing Feature TestingFeature Testing System TestingSystem Testing


2727


Software Verification: Software Verification: Feature Test PhaseFeature Test Phase Verifies the new feature meets all Verifies the new feature meets all

requirementsrequirements Positive and Negative test casesPositive and Negative test cases Performed with a structured test plan, and Performed with a structured test plan, and

also with free-form exploratory testingalso with free-form exploratory testing Manual and Automated testingManual and Automated testing Black box testingBlack box testing approach approach


2828




2929


Software Verification: Software Verification: System Test PhaseSystem Test Phase Performed in a customer environmentPerformed in a customer environment Use-case testing to perform typical customer Use-case testing to perform typical customer

operationsoperations Large scale testing “Scalability” (multiple nodes)Large scale testing “Scalability” (multiple nodes) Interoperability testing with other vendor Interoperability testing with other vendor

equipmentequipment Stress testing “Availability” (max traffic Stress testing “Availability” (max traffic

throughput, simultaneous traffic types, bit errors, throughput, simultaneous traffic types, bit errors, alarm storms…)alarm storms…)

Security vulnerability testing Security vulnerability testing Performed with a structured test plan, and also Performed with a structured test plan, and also

with free-form exploratory testingwith free-form exploratory testing


3030



More test phases?More test phases? Field Testing / Beta TestingField Testing / Beta Testing Testing User DocumentsTesting User Documents Regression TestingRegression Testing Acceptance TestingAcceptance Testing Code review / walkthrough / inspection Code review / walkthrough / inspection

(Bug prevention vs. bug detection or “Quality Assurance” (Bug prevention vs. bug detection or “Quality Assurance” vs “Testing”)vs “Testing”)


3131


Software testing – project management considerations


3232


Project Management Considerations: Project Management Considerations:

Test ExecutionTest Execution Defect trackingDefect tracking Bug scrubs and prioritizationBug scrubs and prioritization Loadbuild frequencyLoadbuild frequency Code coverage analysisCode coverage analysis Location for test resultsLocation for test results Metrics for test executionMetrics for test execution Test phase entry and exit criteriaTest phase entry and exit criteria Test case execution time vs. test failure investigation timeTest case execution time vs. test failure investigation time Test plan / test case / test script updatesTest plan / test case / test script updates


3333


Project Management ConsiderationsProject Management Considerations

There is overlap in the test execution roles, but at some point, There is overlap in the test execution roles, but at some point, the software must be tested by a software tester, not the the software must be tested by a software tester, not the software developersoftware developer

Software Development vs. Software TestingSoftware Development vs. Software TestingFunctional Perspective – “Does it work?”Functional Perspective – “Does it work?”

vs.vs.Vulnerability Perspective – “Can it be broken?”Vulnerability Perspective – “Can it be broken?”

The job of the software tester is to want the software to fail, The job of the software tester is to want the software to fail, expect it to fail, and to find its failures.expect it to fail, and to find its failures. (adapted from Kaner, (adapted from Kaner, Falk & Nguyen)Falk & Nguyen)


3434


Project Management ConsiderationsProject Management ConsiderationsSoftware Testing can be performed in parallel (concurrently) Software Testing can be performed in parallel (concurrently)

with software development, or it can be performed with software development, or it can be performed serially after software development.serially after software development.

Best approach is a combination of both: testing starts when Best approach is a combination of both: testing starts when coding starts, more testing people added when code coding starts, more testing people added when code design is complete. Automated tests are built as software design is complete. Automated tests are built as software is built. (more about test automation later)is built. (more about test automation later)

Result is quicker detection and correction of bugs, for this Result is quicker detection and correction of bugs, for this release and future releases. Goal is always to find bugs release and future releases. Goal is always to find bugs earlier in the development process.earlier in the development process.


3535


Project Management ConsiderationsProject Management ConsiderationsWhen making a trade-off decision between When making a trade-off decision between

software testing depth vs. testing breadth, software testing depth vs. testing breadth, remember it is better to have some information remember it is better to have some information about all features, rather than no information about all features, rather than no information about some features.about some features.

““One percent of the bugs in Microsoft's software One percent of the bugs in Microsoft's software cause half of all reported errors with 20 percent cause half of all reported errors with 20 percent of bugs responsible for 80 percent of the of bugs responsible for 80 percent of the mistakes,”mistakes,” Microsoft Chief Executive Steve Microsoft Chief Executive Steve Ballmer, Oct 3, 2002Ballmer, Oct 3, 2002


3636



Which is better, to verify that all features work as Which is better, to verify that all features work as intended (positive test cases) or to verify all intended (positive test cases) or to verify all failure scenarios (negative test cases)?failure scenarios (negative test cases)?

There are always more tests that could be There are always more tests that could be performed.performed.

Constantly re-evaluate your test plans to make Constantly re-evaluate your test plans to make sure you are doing the sure you are doing the rightright tests. tests.


3737



No amount of testing can prove that the No amount of testing can prove that the software is bug-free. Software testing can software is bug-free. Software testing can only prove that bugs exist.only prove that bugs exist.

Q&AQ&A


3838

Telecom Product VerificationTelecom Product VerificationIntroductionIntroduction

ResourcesResources

Testing Computer Software, Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyenby C. Kaner, J. Falk, and H. Nguyen

Black-Box Testing: Techniques for Functional Testing of Software and Black-Box Testing: Techniques for Functional Testing of Software and Systems, Systems, Boris Beizer, Wiley, 1995Boris Beizer, Wiley, 1995

Managing the Testing Process, Managing the Testing Process, Rex BlackRex Black

Classic Testing Mistakes, Brian MarickClassic Testing Mistakes, Brian Marickhttp://www.testing.com/writings/classic/mistakes.pdfhttp://www.testing.com/writings/classic/mistakes.pdf

Software QA / Test Resource CenterSoftware QA / Test Resource Centerhttp://http://www.softwareqatest.com/index.htmlwww.softwareqatest.com/index.html


3939

SpacerSpacer

class 1

Documents

unlikeblack box testing

production testing

white box testing approaches

new telecom product

product requirements

asglass box

black box approach

test data