Jaak Tepandi

Software processes, quality, and standardsVerification, testing, and validation arrangement

Jaak Tepandi, Jekaterina Tšukrejeva, Stanislav Vassiljev, Pille HaugTallinn University of TechnologyDepartment of Software Science

Moodle: „Software Quality (Tarkvara kvaliteet)” Alternate download: tepandi.ee

Version 15.11.2017

Jaak Tepandi

Context and content

Software quality and standards

Quality management

V&V

Basic concepts

Proofs &Co

Testing

Arrangement

Static analysis

Basic arrangementsNeeds and costsVTV in SDLCTest automationFrameworks, standards

Jaak Tepandi

From simple to advanced VTV arrangement

- Simple

- Need for / costs of more complicated management and methods

- VTV in SDLC: Module, integration, validation, system testing

Jaak Tepandi

Simple arrangement (1)

• user = developer • develop yourself, test yourself (use yourself)

Jaak Tepandi

Simple arrangement (2)

• few requirements• low criticality

customer gives an order programmer delivers the required

software customer tests the software and

returns description of errors found• programmer corrects the errors

and delivers the next version

Jaak Tepandi

Simple arrangement (3)

• more critical software• many users

testing by the developer testing by the application and testing

experts (for example from a support group)

testing by the users• possibility to go from each level to

previous level if an error was found

Jaak Tepandi

Need for more complicated management

when software is built into a system, the system test is usually expensive – there is a need for separate testing of software and the system

the product is complicated - start the control in the beginning of development and divide it into manageable parts

product with critical reliability – divide into stages, use special methods organizationally separated development stages – cooperation of analysts,

designers and programmers organizationally complicated user - cooperation of different user groups the product is used in different ICT, language, legal environments, with many

users - need for stress / portability / interoperability testing, frequent releases - development and operations need to be integrated etc

Jaak Tepandi

Price effectiveness of some testing approaches• developer testing• smoke testing• risk based testing• walkthroughs, reviews• testing with user data• exploratory testing• expert based error search• testing automation• testing non-functional requirements (methods of different difficulty)• boundary situations• equivalence partitioning• program based testing• random testing• reliability oriented methods: fault trees, formal verification, N-version programming,...

Jaak Tepandi

Application => Method

?

• developer testing• smoke testing• risk based testing• walkthroughs, reviews• testing with user data• exploratory testing• expert based error search• testing automation• testing non-functional requirements• boundary situations• equivalence partitioning• program based testing• random testing• reliability oriented methods

Jaak Tepandi

VTV in software development

System

Software

Module

Jaak Tepandi

Testing on software / system levels

• Module: white box, formal verification, tools; consider cost-effectiveness

• Integration: „Integrate and test”, Bottom up, Top-down, Mixed, Mock objects (simulated objects that mimic the behavior of real objects in controlled ways - also on module level)

• Validation testing: requirements, software, user/project documentation

• System testing: black box; usability, performance, load, security, reliability, recovery etc

Jaak Tepandi

Syst. requirem. analysis System testing

Validation testingSoftw. requirem. analysis

Software design

Softw. programming Module testing

Tests

Changes

Error here = 1

30...

9..100

3…10

When? When not? Testing levelsRisk based

ExpertFuzz

Random

User

SecurityLoad

JMeter Monkey

Non-functional

Black boxCanooSelenium

Smoke Data driven MockStatic

White box xUnit

Costar

Verification

Fault seeding Usability tests

Integration testing

Tool examples

Method examples

Jaak Tepandi

Integration testing Testing performed to expose defects in the interfaces and in the

interactions between integrated components or systems [ISTQB] „Integrate and test” for simpler systems Bottom up Top-down Mixed Mock objects

Jaak Tepandi

Bottom-up

Jaak Tepandi

Draiver (mock object) examples

A B C D

Control Data

Jaak Tepandi

Top-down

Jaak Tepandi

Shortcut (stub) examples

A B C D

Jaak Tepandi

Validation testing

Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled [ISO 9000, ISTQB]

Activity trying to demonstrate that what was required has been achieved

Did we build the right software?

May involve requirements, software, user/project documentation

May involve Alpha (customer tests, software on developer's site) / Beta (customer tests, software on customer's site) testing

Jaak Tepandi

System testing

The process of testing an integrated system to verify that it meets specified requirements [ISTQB].

May involve: black box; usability, performance, load, security, reliability, recovery etc – see requirements

Jaak Tepandi

Test automation

• What for?• Possibilities and examples• The cost of automation• Recommendations• Advantages and disadvantages

Jaak Tepandi

A naive start with mock objects (drivers)

Program

Test environment

Tests

Test results

Jaak Tepandi

xUnit: for many languagesExample: code to be tested in gradef_er.py51...60 => "1", ...,91 and more => "5"

#simple grading with possible errors#lab4 - GP for Lab 4#sem - GP for semester activities#exam - GP for examimport mathdef gradef(lab4, sem, exam): total=sem+exam gradef=math.trunc((total)/10)-4

if ((lab4 < 1) or (sem<15)): print ("Sorry, not qualified") gradef=0 elif total < 51: print ("Sorry, not passed with the total of ", total, " grade points") gradef=0 else: print ("Congratulations, passed with grade ", gradef)

return gradef

Jaak Tepandi

xUnit example: testsWhere is the test environment?Test for a boundary value?

#simple grading with possible errorsimport unittestimport mathfrom gradef_er import gradef

class TestGrade(unittest.TestCase):

def test_t1(self): self.assertEqual(gradef(3,3,3), 0) def test_t2(self): self.assertEqual(gradef(25,25,25), 0) def test_t3(self): self.assertEqual(gradef(30,30,30), 1) def test_t4(self): self.assertEqual(gradef(30,30,31), 2)

suite = unittest.TestLoader().loadTestsFromTestCase(TestGrade)unittest.TextTestRunner(verbosity=2).run(suite)

Jaak Tepandi

xUnit example: results#test_t1 (__main__.TestGrade) ... Sorry, not qualifiedoktest_t2 (__main__.TestGrade) ... Sorry, not passed with the total of 50 grade pointsoktest_t3 (__main__.TestGrade) ... Congratulations, passed with grade 2FAILtest_t4 (__main__.TestGrade) ... Congratulations, passed with grade 2ok======================================================================FAIL: test_t3 (__main__.TestGrade)----------------------------------------------------------------------Traceback (most recent call last): File "C:\Users\JT\AppData\Local\Programs\Python\Python36\gradef_er_test.py", line 13, in test_t3 self.assertEqual(gradef(30,30,30), 1)AssertionError: 2 != 1----------------------------------------------------------------------Ran 4 tests in 0.109sFAILED (failures=1)

Jaak Tepandi

Why automation?

• Improve quality (both actual and perceived!)

• Faster integration / feedback (depends)

• Agile methods impossible without automation

• Simplify routine work

• Regression testing

• Seems simple

Jaak Tepandi

Automation (1)

Saving and executing tests (eg Selenium, xUnit)

Tests scripts - generating test scripts automatically / manually

GUI test drivers

Automation of test generation – automatically generate statement or branch coverage tests, tests based on functional description, discover branches not covered etc

Jaak Tepandi

Automation (2)

load testing tools

tools for simplifying the testing process: for example software that simplifies the generation of drivers during integration testing, random fault generators and so on

Performance analysis

Coverage analysis

Jaak Tepandi

Automation (3)

tools for estimating the testing quality, for example to find program components not covered during testing

static analysis tools, for example software for evaluating program metrics

bug management tools

testing process and resources management tools

website testing tools

Jaak Tepandi

Website testing tools (examples)

W3C Markup Validation ServiceW3C Link CheckerCSS Validation Service WDG HTML ValidatorA Real ValidatorCynthiaSaysCanoo WebTestRational Robot Parasoft WebKingSeleniumAppache Jmeter...

Jaak Tepandi

Automation costsCost-effective and simple?

Resources needed: Tools Training Development procedures Creating automated tests Test execution Updating tests Porting to new environments Other resources

Jaak Tepandi

Recommendations (1)

• Understand the difference between the test process and the automation process (tools). In case of automation testing must still remain transparent

Automated testing is a supplement to the whole testing process – not as a replacement

Choose the tools carefully, get acquainted with them first, collect the information

Choose carefully the structure of test sets, let it be understandable and supportive to the testing process

Jaak Tepandi

Recommendations (2)

Each automated testing session must deliver a report describing tests performed and errors found

Ensure that the product is sufficiently developed – then there is hope that benefits from the test automation exceed the expenses of test transformation

Automation pays off if the object(s) of testing are (1) large enough, (2) sufficiently variable (regression testing is needed), (3) sufficiently stable (tests are not redesigned too often)

Jaak Tepandi

Advantages and disadvantages of testing tools

• The testing tools enable to get free from a great deal of routine handwork

• Many agile development methods are possible only with testing tools (why?)

• The testing tools require a lot of extra work, especially in case of fast changing requirements, software, and environment

Jaak Tepandi

Testing frameworks

- RUP - TPI - Standards

Jaak Tepandi

The RUP Testing Philosophy

• Iterative development + mission

• Low up-front documentation. Detailed Test planning is defined iteration by iteration, based on a governing master Test plan

• Holistic approach - tests from the requirements and from other sources.

• Automation. Testing continues throughout, the RUP lifecycle. Tools help generate test data conditions, run tests, and analyze results

Jaak Tepandi

RUP iteration mission examples

Iterative development + mission• Find as many defects as possible• Find important problems fast• Assess perceived quality risks• Advise about perceived project risks• Advise about perceived quality• Certify to a given standard• Assess conformance to a specification (requirements, design, or product claims)

Jaak Tepandi

Artifacts

• Test Evaluation Summary

• Test Plan (and sometimes a Master Test Plan)

• Test Ideas, and Test-Idea List

• Test Suites and Test Cases

• Defect and Defect List

• Workload Model

Cf IEEE 829

Jaak Tepandi

Roles and artifacts

• Test Manager

• Test Analyst

• Test Designer

• Tester

• Test Evaluation Summary

• Test Plan (and sometimes a Master Test Plan)

• Test Ideas, and Test-Idea List

• Test Suites and Test Cases

• Defect and Defect List

• Workload Model

?

Jaak Tepandi

Test Process Improvement, TPI / Tmap NEXT/ TPI NEXT

• Model, key areas, methodology

• Test Maturity Matrix

• TPI - Test Process Improvement (the earliest)

• TMAP NEXT - methodology for structured testing of software products

• TPI NEXT - improving test processes

....A sound product risk analysis is the basis for all testing (No Risk No Test)

TPI® Next

16 Key areas4 Maturity levels: Initial, Controlled, Efficient and OptimizingRelated: CMMI® or SPICESource: http://www.slideshare.net/InfinITnetvaerk/tpi-en-metode-til-forbedring-af-testproces-af-elisabeth-andresen-sogeti

Jaak Tepandi

Enterprise

TQM

SPI, CMMM,...

TPI and others

TPIClients

ProductsResources...

Jaak Tepandi

ISO/IEC 12207 …. + IEEE 829-2008 => ISO/IEC/IEEE 29119-3Software and System Engineering - Software testing - Part 3:

Test documentation (selection)… cf Labs

Organisational test process doc-n

• Test Policy• Organisational test

strategy

Test management processes doc-n

• Test Plan• Test Status Report• Test Completion

Report

Dynamic Test Processes doc-n

• Test Design Spec-n• Test Case Spec-n• Test Procedure Spec-n• Test Data Req-s• Actual Results• Test Result• Incident Report

Jaak Tepandi

IEEE Std 1012-2012

1012-2012 - IEEE Standard for System and Software Verification and Validation

The scope of V&V processes encompasses systems, software, and hardware, and it includes their interfaces.

V&V processes include the analysis, evaluation, review, inspection, assessment, and testing of products.

Control has been integrated into life cycle – faults found at early step are more cheaper to repair

Tests at each development level

https://standards.ieee.org/findstds/standard/1012-2016.html

Jaak Tepandi

Takeaway

Basic testing management schemes Need for more complicated arrangement Module, integration, validation, system testing Testing automation - types, advantages, disadvantages Testing with XP and TDD (other lessons), RUP, V-model Test Process Improvement (TPI), ISO/IEC/IEEE 29119

Jaak Tepandi

Additional reading (examples)

Ian Sommerville. Software Engineering. Ninth Edition. Addison-Wesley, Ch 8.

Daniel Galin, Software Quality assurance from theory to implementation, Pearson - Addison-Wesley. Chapter 10.

Guide to the Software Engineering Body of Knowledge (SWEBOK), IEEE. Chapter 4.

Certified Tester Foundation Level Syllabus, ISTQB. Chapter 5,6.

Test Process Improvement - http://www.tmap.net/test-process-improvement

Moodle: „Software Quality (Tarkvara kvaliteet)”. Alternate download: tepandi.ee