guiding the way to better software testing - rbcs, inc · guiding the way to better software...

The ISTQB Advanced SyllabusGuiding the Way to Better Software Testing

http://www.rbcs-us.com/

ISTQB Ad d S ll bISTQB Advanced SyllabusGoals of this presentationp

Familiarize you with the ISTQB program and certifications in generalExplain the Advanced level certification and its benefitsIll t t th ki d f biliti did t t d t t Illustrate the kinds of abilities candidates must demonstrate to gain the certificate

Topics to coverAn overview of the ISTQB program and results so farAn overview of the ISTQB program and results so farThe structure and origin of the Advanced SyllabusThe Advanced Test Manager The Advanced Test AnalystyThe Advanced Technical Test Analyst

Each of the last three topics includes a training course excerpt, an example of the kind of problem a certificate-h ld l d l iholder can solve, and a sample exam question

ISTQB Advanced Syllabuswww.rbcs-us.com

Copyright (c) RBCS 2008



ISTQB O iISTQB Overview

Program and ResultsProgram and Results

K Q ti T AddKey Questions To AddressWhat is tester certification? What is tester certification? What are the ASTQB and ISTQB? What are the levels of certification? What is the impact of certificationp

On the tester?On the organization?O t e o ga at o ?On the testing profession?




S ft T t C tifi tiSoftware Tester CertificationTester certification confirms through objective Tester certification confirms, through objective, carefully-designed exams, the professional capabilities of software testersISTQB’s approach

Practical, real-world focusedSupports a career path with levels of certificationSupports a career path with levels of certificationPromotes the software testing professionRepresents the distilled wisdom of many experts including practitioners consultants trainers and academicspractitioners, consultants, trainers, and academicsTraining is not required to take exams

Last two points in particular distinguish ISTQB’s



approach from other testing certifications


ASTQBASTQBComposed of recognized expertsp g pMix of practitioners, consultants, trainers, and academics

Patricia McQuaid, President (academic)Joe Gance, Vice President (practitioner)Randy Rice, Treasurer (trainer)Debra Friedenberg, Technical Advisory Group Chair (practitioner)Rex Black, ISTQB Representative (consultant)Rex Black, ISTQB Representative (consultant)Taz Daughtrey (academic/practitioner), Jerry Everett (trainer), Andrew Pollner (trainer), and Wayne Middleton (trainer), DirectorsLois Kostroski Managing DirectorLois Kostroski, Managing Director

Develops and administers examsAccredits trainersParticipates in ISTQB activities



Participates in ISTQB activities


ISTQBComposed of 41 National B d th d bl d i

ISTQBBoards, more than doubled in scope in the last four yearsBased in Brussels, Belgium, EURex Black, PresidentE ik V d l Vi Erik van Veenendaal, Vice PresidentHorst Pohlmann, TreasurerChris Carter, SecretaryA ll i l h i i tiA collegial, sharing organizationWorking parties (composed of National Board delegates) define syllabi (bodies of knowledge), glossary, processes, etc.glossary, processes, etc.These working parties distill the collective experience and wisdom represented across boards (>2,000 person-years)




F d ti C tifi tiFoundation CertificationEntry-level certification: 0+ years of experienceEntry level certification: 0 years of experienceGoals

Ensure a broad understanding of the fundamental best practices and key concepts in software testingpractices and key concepts in software testingProvide a foundation for professional growth

Syllabus/body of knowledge coversF d t l f t ti t ti i th ft lif l Fundamentals of testing, testing in the software lifecycle, static techniques, white-box and black-box test design, test management, and testing toolsSyllabus-based training courses are typically 3-5 daysSyllabus-based training courses are typically 3-5 days

Status: Version 2007, an incremental improvement on version 2005, released, with exams and training courses widely available



courses widely available


Ad d C tifi tiAdvanced CertificationMid-level certification: 5+ years experiencey pGoals

Ensure an understanding of advanced best practices and key concepts in software testing by committed test professionalsp g y pSupport on-going professional growth

Syllabus/body of knowledge coversAdvanced behavioral (black-box) testing for tester analysts, test d a ced be a o a (b ac box) test g o teste a a ysts, test automation and advanced non-functional testing for technical test analysts, and sophisticated test management conceptsSyllabus-based training courses are typically 5 days for each

d l (th d l t t l)module (three modules total)Status: Version 2007, a merge and expansion of the older Practitioner and Advanced syllabi, is released. Exams and accredited training are running against this version now



accredited training are running against this version now.


E p t C tifi tiExpert CertificationGuru-level certification: 8+ years experienceGuru level certification: 8+ years experienceGoals

Ensure consistent understanding and execution of proven cutting-edge techniques by seasoned test professionalsLead the software testing profession

Syllabi/bodies of knowledge may coverSyllabi/bodies of knowledge may coverTest process improvement, test automation, test management, and moreSyllabus based training courses will be offeredSyllabus-based training courses will be offered

Status: A working party is developing new expert syllabi. We expect the Expert Test Manager syllabi an



Improving Test Processes syllabi this year.


Vi li i th L l f C tifi tiVisualizing the Levels of CertificationExpert Expert Expert ExpertExpert [TBD]

Expert [TBD]

Expert [TBD]

Expert [TBD]

…

Advanced Technical

Advanced Functional

Advanced Manager

Foundation

Relative size of figures indicates expected relative numbers of potential certificate



holders (not to scale). There will be more than four expert certifications.


ISTQB b th N bISTQB by the Numbers




ISTQB C tifi ti G thISTQB Certification Growth




V l f C tifi tiValue of CertificationTo the testerTo the tester

Demonstrate mastery of the best practices and key concepts in the fieldAdvance career and opportunities in competitive pp pmarket

To the organization Ensure better testing for better software and lower gcosts of poor qualityAchieve consistency and re-usability of testing

To the professionTo the professionBuild on our best work and stop going in circlesDefine the profession and what professional testers know




A E pl f C tifi ti ROIAn Example of Certification ROIOne RBCS client ran a Foundation course as a One RBCS client ran a Foundation course as a pilot in one of its many offices in the USThe week after the course, a tester applied a technique he’d learned to reduce the regression test set from 800 test cases to 300That is a 60% reduction in regression test effort That is a 60% reduction in regression test effort, which accounts for most of their testingThis client now requires all new testers to hold qthe Foundation certificate, and is gradually training its entire staff of testers





ISTQB Ad d S ll bISTQB Advanced Syllabus

Structure and OriginStructure and Origin

ISTQB Ad d S ll b 2007ISTQB Advanced Syllabus 2007Developed by a team of 15 authors spanning 10 Developed by a team of 15 authors spanning 10 countriesSixty-nine primary reviewers spanning nine countriesFinal review and approval by 33 National BoardsDistills over 2 000 person years of experienceDistills over 2,000 person-years of experienceThe ISTQB and the authors are the source of the syllabus which is used by permission as the basis for y y pall accredited training courses (including RBCS’ courses)




Ad d S ll b A thAdvanced Syllabus AuthorsGraham Bath*+ Klaus Olsen Graham Bath +Rex Black*+Sigrid Eldh+

Klaus Olsen Randy Rice*+Jürgen Richterg

Bernard Homès (chair)+Jayapradeep Jiothis

J gEric Riou Du CosquerMike Smith+

Paul Jorgensen*+Vipul KocherJ d M K *

Geoff Thompson (vice chair)*+Erik Van Veenendaal*+Judy McKay*+ Erik Van Veenendaal*+

* Indicates an author who has also written or edited one or more books on testing.



+ Indicates an author who has 20 or more years of software or systems experience.


F d ti S ll b Th Si Ch ptFoundation Syllabus: The Six Chapters1 Fundamentals of testing1. Fundamentals of testing2. Testing throughout the software

lif llifecycle3. Static techniques4. Test design techniques5. Test management5. Test management6. Tool support for testing




Ad d S ll b Th T Ch ptAdvanced Syllabus: The Ten Chapters1 Basic aspects of software testing1. Basic aspects of software testing2. Testing processes3. Test managementg4. Test techniques5. Testing of software characteristics6. Reviews7. Incident (defect) management8 St d d d t t i t8. Standards and test process improvement9. Test tools and automation10 People skills (team composition)



10. People skills (team composition)


Ad d S ll b b th N bAdvanced Syllabus by the NumbersNumber of pages: 114Number of pages: 114Number of learning objectives

Test Manager:64 t t l• 64 total

• 43 K2, 12 K3, 9 K4Test Analyst:

29 t t l• 29 total• 18 K2, 7 K3, 4 K4

Technical Test Analyst: • 44 t t l• 44 total• 25 K2, 11 K3, 8 K4

Referenced books: 21Referenced standards: 7




L i Obj tiLearning ObjectivesThe Foundation and Advanced exams are based on l i bj i hi h h h ld b bl learning objectives, which state what you should be able to doThe learning objectives are at four levels of increasing sophisticationp

K1: remember basic facts, techniques, and standardsK2: understand the facts, techniques, and standards and how they inter-relateK3: apply facts, techniques, and standards to your projectspp y , q , y p jK4: analyze facts, techniques, and standards, and adapt or select them for your project

For the Advanced exams, the entire Advanced syllabus is implicitly covered at the K1 levelp yEach module (test manager, test analyst, technical test analyst) has its own set of K2, K3, and K4 learning objectives defined against each chapter




Ad d E Q tiAdvanced Exam QuestionsThe lower levels, K1 and K2, are covered implicitly The lower levels, K1 and K2, are covered implicitly within higher level questionsUnlike the Foundation exam, the Advanced exams are heavily focused on K3 and K4 questionsare heavily focused on K3 and K4 questionsMany exam questions consider a scenario

Scenario describedSequence of questions about scenario askedSimulates real-world use of various advanced testing concepts

h d ll b l blThe Foundation syllabus is also examinableCross-section questions, including joining Foundation and Advanced sections, are allowed



,


Ad d T i i CAdvanced Training CoursesMust cover all learning objectives defined Must cover all learning objectives defined for the module (test manager, test analyst, or technical test analyst)M b l fi d l (f li Must be at least five days long (for live, instructor-led courses)Must include real-world examples for all Must include real-world examples for all K2, K3, and K4 learning objectivesMust include realistic exercise for all K3 and K4 learning objectives (which must be done in class for live, instructor-led courses)courses)




T i i C B kdTraining Course BreakdownTest Manager Test Analyst Tech Test Analystg y y

Chapter Hours Percent Hours Percent Hours PercentIntro 1.0 3% 1.0 3% 1.0 3%

1 2.5 7% 0.5 1% 0.5 1%2 2.0 6% 3.0 9% 3.0 9%3 18.7 53% 2.0 6% 2.0 6%4 0.0 0% 18.0 51% 15.5 44%5 0.0 0% 3.5 10% 4.0 11%6 2.0 6% 3.0 9% 3.0 9%7 1.3 4% 2.0 6% 2.0 6%8 2.0 6% 0.0 0% 0.0 0%9 1.5 4% 1.5 4% 3.5 10%

10 4.0 11% 0.5 1% 0.5 1%



Total 35.0 100.0% 35.0 100% 35.0 100%


R i t f Ad d C tifi tRequirements for Advanced CertificatesTo earn a certificate one must:To earn a certificate, one must:

Hold a Foundation Level certificate issued by an ISTQB-recognized Exam Board or Member BoardHave appropriate experience in software testing or development between 2-5 years testing or development, between 2-5 years, depending on degree held and certificate(s) sought

h d h h llSubscribe to the Code of Ethics in the syllabusAccredited training is recommended but not requirednot required




ISTQB T t EthiISTQB Tester EthicsPUBLIC- Certified software testers shall act consistently with the public interest.CLIENT AND EMPLOYER - Certified software testers shall act in a manner that is in the best interests of their client and employer, consistent with the public interest.PRODUCT - Certified software testers shall ensure that the deliverables they

d ( h d d h ) h h h f lprovide (on the products and systems they test) meet the highest professional standards possible.JUDGMENT- Certified software testers shall maintain integrity and independence in their professional judgment.

A AG C f d f d l d h ll b bMANAGEMENT - Certified software test managers and leaders shall subscribe to and promote an ethical approach to the management of software testing.PROFESSION - Certified software testers shall advance the integrity and reputation of the profession consistent with the public interest.COLLEAGUES C ifi d f h ll b f i d i f COLLEAGUES - Certified software testers shall be fair to and supportive of their colleagues, and promote cooperation with software developers.SELF - Certified software testers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession



practice of the profession.



Ad d T t MAdvanced Test Manager

Goals Excerpt and Sample Exam Goals, Excerpt, and Sample Exam Question

B i Ad d T t MBeing an Advanced Test ManagerYou should be able to:

Define the overall testing goals and strategy for the systems being testedPlan, schedule and track the tasks D ib d i th ti iti Describe and organize the necessary activities Select, acquire and assign the adequate resources to the tasksSelect, organize and lead testing teamsOrganize the communication between the members of the Organize the communication between the members of the testing teams, and between the testing teams and all the other stakeholdersJustify the decisions and provide adequate reporting information here applicable information where applicable

Advanced Test Manager exams (and courses) focus on these main conceptsLet’s look at sample course content and an exam question



Let s look at sample course content and an exam question…


Ri k B d T tiRisk-Based TestingRisk: the possibility of an undesired outcomeRisk: the possibility of an undesired outcome

Product or quality risksProject or planning risks

The level of risk varies often depending on:The level of risk varies, often depending on:Likelihood Impact

In risk based testing testing responds to risk:In risk-based testing, testing responds to risk:Allocation of effort, test sequencing, prioritization of defect repairProviding mitigation and contingency responses Reporting test results and project status Reporting test results and project status

These responses to risk occur throughout the lifecycle:Reduce risk by running tests and finding defects R l i k d i k l l b d i f i



Re-evaluate risk and risk levels based on new information


A l ti l Ri k B d T t St tAnalytical Risk-Based Test StrategyThe higher the risk the more test effortThe higher the risk, the more test effortThe higher the risk, the earlier the test coveragegTotal level of quality risk reduced as testing continuesResults reported in terms of residual riskTest triage (e.g., due to schedule compression) done in risk ordercompression) done in risk orderMore robust than requirements-basedBest when blended with reactive strategies to



Best when blended with reactive strategies to detect missed risks


H Ri k B d T ti S l P blHow Risk-Based Testing Solves Problems Insufficient time: All testing is time-boxed; Insufficient time: All testing is time boxed; have to prioritize and triageCoverage questions: All test coverage, g q gmeasured as a percentage of what could be tested, is 0%; choose smart subsetP ifi ti St k h ld i l t Poor specifications: Stakeholder involvement fills gaps in documentsEnd-game compression: Provides means to End game compression: Provides means to drop tests intelligentlyRelease decisions: Can address residual risk



rather than bug and test counts


Hi t f A l ti l Ri k B d T tiHistory of Analytical Risk-Based TestingEarly 1980s: Separately Boehm and Beizer develop Early 1980s: Separately, Boehm and Beizer develop risk-driven spiral lifecycle and risk-driven integration, precursors of iterative and agile lifecyclesMid 1980s: Separately, Beizer and Hetzel declare risk as driver of testing but leave out mechanisms1990s: Separately Black Craig Gerrard and Redmill 1990s: Separately, Black, Craig, Gerrard, and Redmill develop similar approaches for quality risks analysis and risk-based testing2000s: Risk-based testing (in various forms) in wide usage




Ri k M tRisk ManagementRisk management includes three primary activities:Risk management includes three primary activities:

Risk identificationRisk assessment or analysisRisk mitigation or risk controlRisk mitigation or risk control

These activities start in sequence, but are overlapping and iterative due to continuous risk management Ri k id ll i l d ll j Risk management ideally includes all project stakeholders, though some stakeholders may act as surrogates for other stakeholdersTest analysts bring particular expertise to risk management due to their defect-focused outlookLet’s look at these activities more closely…



Let s look at these activities more closely…


Ri k Id tifi tiRisk IdentificationWhether for product or project risks, we can identify risks via: p p j , y

Expert interviewsIndependent assessmentsUse of risk templatesProject retrospectivesRisk workshops and brainstormingChecklistsC lli t iCalling on past experience

The broadest range of stakeholders yields the most complete, accurate, precise risk identification Risk identification can Risk identification can

Stop at risk items Look downstream to identify potential effects of the risk item (FMEA)



( )Look upstream at the source of the risk (Hazard Analysis)


Ri k A l i A tRisk Analysis or AssessmentRisk analysis or assessment studies the identified Risk analysis or assessment studies the identified risksRisks are categorized, using ISO 9126 or other quality categories categories Risks are assigned a level of risk, often based on likelihood and impact

Likelihood arises from technical risk Impact arises from business risk

The level of risk is determined either quantitatively q yor qualitatively

Typically the level of risk is determined qualitativelyEither way, unless statistical data is used, the level of risk



y, ,reflects stakeholder opinions and consensus


Ri k C t lRisk ControlFour options for risk controlFour options for risk control

Mitigation: reduce likelihood and/or impact beforehand

d f hContingency: prepare to reduce impact after the factTransference: transfer impact of risk to another ppartIgnore/accept: hope for the best

E h ti h b fit t iti t Each option has benefits, opportunities, costs, and potentially additional risks to considerPoorly done risk control can make matters



Poorly done, risk control can make matters worse!


E i F ti l Q lit Ri k A l iExercise: Functional Quality Risks AnalysisRead the HELLOCARMS System Read the HELLOCARMS System Requirements DocumentPerform an informal quality risks analysis in Perform an informal quality risks analysis in groups of 3-5, identifying risks for functional quality characteristics only, using the q y y gtemplate shown earlier

Spend 30 minutes identifying quality risksSpend 15 minutes assessing the level of each risk

Discuss




S pl E Q tiSample Exam QuestionAn organization follows a requirements-based test g qstrategy for most of its projects. Which of the following is the best example of modifying the test approach for a project based on an understanding of pp p j grisks?A. Past performance issues lead to an increased

effort on performance testing.p gB. Test estimation is based on the number of pages

in the requirements specification.C. Test execution is outsourced to a testing company C. Test execution is outsourced to a testing company

based on a low-cost bid.D. Unit test effort is limited to ensure early

commencement of system test execution commencement of system test execution.





Ad d T t A l tAdvanced Test Analyst


B i Ad d T t A l tBeing an Advanced Test AnalystYou should be able to:

Implement the test strategy with a focus on business domain requirementsAnalyze the system based on user quality expectations and apply that analysis to the testing to be doneapply that analysis to the testing to be doneEvaluate the system requirements to determine whether the business objectives can be met by that systemPrepare and execute adequate testing activities, and report on the progress of these activitiesthe progress of these activitiesProvide the necessary evidence and data to support evaluations and findingsImplement the necessary tools and techniques to achieve the d fi d l

ydefined goals

Advanced Test Analyst exams (and courses) focus on these main concepts



Let’s look at sample course content and an exam question


D i i T blDecision TablesConcept: test the rules that govern handling Concept: test the rules that govern handling of transactional situationsModel: table (or Boolean graph) connecting ( g p ) gconditions with actionsTest derivation: fulfill conditions, check

tiactionsCoverage criteria: at least one test per combination of conditions (DT column)combination of conditions (DT column)Bug hypothesis: improper action or missing action




E pl D i i T bl (F ll)Example: Decision Table (Full)Conditions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Conditions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Real account? Y Y Y Y Y Y Y Y N N N N N N N N

Active account? Y Y Y Y N N N N Y Y Y Y N N N NActive account? Y Y Y Y N N N N Y Y Y Y N N N N

Within limit? Y Y N N Y Y N N Y Y N N Y Y N N

L ti k ? Y N Y N Y N Y N Y N Y N Y N Y NLocation okay? Y N Y N Y N Y N Y N Y N Y N Y N

Actions

A ?Approve? Y N N N N N N N N N N N N N N N

Call cardholder? N Y Y Y N Y Y Y N N N N N N N N



Call vendor? N N N N Y Y Y Y Y Y Y Y Y Y Y Y


E pl D i i T tExample: Deriving TestsIn the example just shown, each column of the table In the example just shown, each column of the table is a test case

We will create the conditions (which are the test’s inputs)We will verify the actions (which are the test’s expected We will verify the actions (which are the test s expected results)

In some cases, we might generate more than one test case per column (more later)case per column (more later)In this case, some of the test cases don’t make much sense; e.g.:

A l b i ?Account not real but account active?Account not real but account within limit?

Maybe we don’t need all the columns in our decision



table?


C ll p i D i i T blCollapsing a Decision TableIf the value of one or more particular conditions can’t If the value of one or more particular conditions can t affect the actions for two or more combinations of conditions, we can collapse the decision tableThis involves combining two or more columnsThis involves combining two or more columns

Combinable columns often but not always next to each otherLook for two or more columns that result in the same Look for two or more columns that result in the same combination of actions (for all the actions in the table)Replace the conditions that are different in those columns with “-” (for don’t care/doesn’t matter/can’t happen)( / / pp )

Repeat this process until no further columns share the same combination of actions or where collapse would erase an important distinction



would erase an important distinctionBe careful with tables that have non-exclusive rules


E pl D i i T bl (C ll p d)Example: Decision Table (Collapsed)Conditions 1 2 3 5 6 7 9 Column numbers retained for Conditions 1 2 3 5 6 7 9

Real account? Y Y Y Y Y Y N

Active account? Y Y Y N N N

Column numbers retained for ease of reference to full table

Study carefully to understand h l 4 ld ll i Active account? Y Y Y N N N -

Within limit? Y Y N Y Y N -

L ti k ? Y N Y N

why rule 4 could collapse into rule 3, but not rule 3 into rule 2

The same logic also applies to Location okay? Y N - Y N - -

Actions

A ?

g pprule 8 collapsing into rule 7, but not rule 7 into rule 6

Formula for number of columns Approve? Y N N N N N N

Call cardholder? N Y Y N Y Y N

Formula for number of columns (2conditions) no longer applies

Regular pattern of conditions no l li



Call vendor? N N N Y Y Y Y longer applies


E i H E it L IExercise: Home Equity Loan InsuranceA new HELLOCARMS feature will allow selling a A new HELLOCARMS feature will allow selling a life insurance policy for the amount of a home equity loan to the borrower (no applicants for lines of credit

t )or reverse mortgages)The premium is calculated annually, at the beginning of each policy period, based on the loan balance at p y p ,that timeThe base premium is $1 per $10,000 loan

Premium increases by 50% based for each “yes” answer to the health questions on the next pagePremium increases based on age and body mass index (BMI)



table shown on following page


S pl E Q tiSample Exam QuestionAn on-line shoe-selling e-commerce Web site stocks the following

f ’ l fg g

options for men’s loafers:• Tassel: Tassel (T) or non-tassel (~T)• Color: Black (B), cordovan (C), or white (W)• Size: all full and half sizes from 8 to 14 (S=n)• Size: all full and half sizes from 8 to 14 (S=n)The store is overstocked with tasseled loafers of all sizes and colors, along with white loafers in all sizes, and cordovan loafers in sizes 13, 13 ½, and 14. As a result, they are offering a 10% discount (10%) and free shipping (FS) on these items free shipping (FS) on these items. Design a full decision table that shows all combinations of conditions, then collapse that table by using don’t care (“-“) notation where one or two conditions cannot influence the action. Which of the following statements is true about these two tables?statements is true about these two tables?A. The full table has 8 rules; the collapsed table has 5.B. The full table has 12 rules; the collapsed table has 7.C. The full table has 12 rules; the collapsed table has 5.pD. Both tables have 12 rules, as no combinations can collapse.





Ad d T h i l T t A l tAdvanced Technical Test Analyst


B i Ad d T h i l T t A l tBeing an Advanced Technical Test AnalystYou should be able to:

Structure the tasks defined in the test strategy in terms of technical requirementsAnalyze the internal structure of the system in sufficient detail t t th t d lit l lto meet the expected quality levelEvaluate the system in terms of technical quality attributes such as performance, security, etc.Prepare and execute adequate testing and report on progressPrepare and execute adequate testing and report on progressConduct technical testing activitiesProvide the necessary evidence to support evaluationsImplement the necessary tools and techniquesImplement the necessary tools and techniques

Advanced Technical Test Analyst exams (and courses) focus on these main conceptsLet’s look at sample course content and an exam question



Let s look at sample course content and an exam question


T h i l S it T tiTechnical Security TestingSecurity is a key risk for many applicationsSecurity is a key risk for many applicationsSecurity tests and failures are different from functional tests and failures, often arising from unintended side-effectsVulnerabilities (and thus tests) include data access, function malicious code insertion denial of service function, malicious code insertion, denial of service, sniffing, encryption, and virus/ wormsVulnerabilities can arise from user interface, file system, operating system, and external softwareIncreased quality in security can decrease quality in usability perform and functionality



usability, perform, and functionality


D i i T h i l S it T tDesigning Technical Security TestsThe following approaches can be used The following approaches can be used to develop security tests

Information retrievalInformation retrievalVulnerability scanAttack plansttack p a sSecurity attacks

The last is very similar to the functional The last is very similar to the functional attacks described in Chapter 4Let’s take a closer look…



Let s take a closer look…


Att ki D p d i d th UIAttacking Dependencies and the UIDependenciesDependencies

Block access to libraries

lManipulate registry (or similar information)Force use of corrupt filesManipulate and

User interfaceOverflow inputsManipulate and

replace filesForce low-resource operation

Overflow inputsSwitches and optionsCharacters,



operation commands


Att ki D i d I pl t tiAttacking Design and ImplementationDesignDesign

Common accounts and

dpasswordsUnprotected APIsConnect to all ports Implementation

M i l t tip

Create loops (e.g., using scripts)Use unusual

Manipulate timeDuplicate high-privilege filesUse unusual

workflowsForce resets

p gForce error messagesSniff temporary files




E pl S it S tti Att kExample: Security Setting Attack




S pl E Q tiSample Exam QuestionWhich of the following is an example of a Which of the following is an example of a defect we would expect to find during technical security testing?technical security testing?A. Slow response timeB. Resource over-utilizationC. Invalid privilege elevationp gD. Frequent system crashes





Bibli phBibliography

St d dStandardsBritish Computer Society, BS 7925-2 (1998), “Software p y, ( ),Component Testing” Institute of Electrical and Electronics Engineers, IEEE Std 829(1998/2007), “IEEE Standard for Software Test Documentation”Institute of Electrical and Electronics Engineers, IEEE Std 1028 (1997), “IEEE Standard for Software Reviews”Institute of Electrical and Electronics Engineers, IEEE Std 1044 (1993) “IEEE Standard Classification for Software Anomalies”(1993), IEEE Standard Classification for Software AnomaliesInternational Standards Organization, ISO/IEC 9126-1:2001, “Software Engineering – Software Product Quality”International Software Testing Qualifications Board ISTQB International Software Testing Qualifications Board, ISTQB Glossary (2007), “ISTQB Glossary of terms used in Software Testing, Version 2.0”US Federal Aviation Administration, DO-178B/ED-12B,



, / ,“Software Considerations in Airborne Systems and Equipment Certification”


B kBooksBoris Beizer, Black-Box Testing, Wiley, 1995o s e e , lack ox esting, W ey, 995Rex Black, Managing the Testing Process (2nd edition), Wiley, 2002 Rex Black, Critical Testing Processes, Addison-Wesley, 2003Rex Black Pragmatic Software Testing Wiley 2007 Rex Black, Pragmatic Software Testing, Wiley, 2007 Ilene Burnstein, Practical Software Testing, Springer, 2003Lee Copeland, A Practitioner’s Guide to Software Test Design, A h H 2003Artech House, 2003Rick Craig and Stefan Jaskiel, Systematic Software Testing, Artech House, 2002 Paul Gerrard and Neil Thompson, Risk-based e-Business Testing, Artech House, 2002Tom Gilb and Dorothy Graham, Software Inspection, Addison-



Wesley, 1993


B kBooksDorothy Graham, Erik van Veenendaal, Isabel Evans, Rex Black, Dorothy Graham, Erik van Veenendaal, Isabel Evans, Rex Black, Foundations of Software Testing, Thomson Learning, 2007M. Grochmann, “Test case design using Classification Trees”, Conference Proceedings of STAR 1994g 99Paul Jorgensen, Software Testing: A Craftsman’s Approach (Second Edition), CRC Press, 2002Cem Kaner James Bach Bret Pettichord Lessons Learned in Cem Kaner, James Bach, Bret Pettichord, Lessons Learned in Software Testing; Wiley, 2002Tim Koomen, Martin Pol, Test Process Improvement, Addison-Wesley 1999 Wesley, 1999 Glenford Myers, The Art of Software Testing, Wiley, 1979 Martin Pol, Ruud Teunissen, Erik van Veenendaal, Software Testing: A Guide to the T map Approach Addison Wesley 2002



Testing: A Guide to the T-map Approach, Addison-Wesley, 2002


B kBooksSteven Splaine and Stefan Jaskiel, The Web-Testing Handbook, Steve Sp a e a d Ste a Jask e , he Web esting andbook, STQE Publishing, 2001D. H. Stamatis, Failure Mode and Effect Analysis, ASQ Press, 1995 Erik van Veenendaal editor The Testing Practitioner UTN Erik van Veenendaal, editor, The Testing Practitioner, UTN Publishing, 2002 James Whittaker, How to Break Software, Addison-Wesley, 2003 James Whittaker and Herbert Thompson How to Break Software James Whittaker and Herbert Thompson, How to Break Software Security, Addison-Wesley, 2004





F M I f tiFor More Information…

C t t RBCSFor over a dozen years, RBCS has delivered services in consulting, outsourcing and training for software and hardware testing Employing the industry’s most

…Contact RBCStraining for software and hardware testing. Employing the industry s most experienced and recognized consultants, RBCS conducts product testing, builds and improves testing groups and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save g g p ptime and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.Add RBCS IAddress: RBCS, Inc.

31520 Beck RoadBulverde, TX 78163-3911USA

Phone: +1 (830) 438-4830Fax: +1 (830) 438-4831E-mail: [email protected] b b



Web: www.rbcs-us.com



Questions, Comments, Q , ,and Discussion?

guiding the way to better software testing - rbcs, inc · guiding the way to better software...

Documents