vipul kocher - software testing, a framework based approach

Software Testing - A Framework based approach

Vipul Kocher www.puretesting.com

Dhanasekaren R

http://www.puretesting.com/

(C) Vipul Kocher www.PureTesting.com2

Roadmap

Premise of frameworks Introduction to Frameworks Applying Frameworks Caveats


Deja Vu

Have you ever looked at a new system to be tested and thought “I have done this before”?

What are the things that you had “done before?”


How do I reuse my testing experience Can I leverage this

For similar applications For different applications with similar features For the same or different organization

How do we capture knowledge, information acquired during the course of a project

Can we propagate knowledge, to enable faster, better testing

Can one take advantage to guide comprehensively any project through SDLC for variety of product/project contexts across

Testers Test Managers Test Process

Introduction to Framework


Introduction to Framework The free dictionary[1] defines a framework as:

1. A structure for supporting or enclosing something else, especially a skeletal support used as the basis for something being constructed.

2. An external work platform; a scaffold.

3. A fundamental structure, as for a written work.

4. A set of assumptions, concepts, values, and practices that constitutes a way of viewing reality.

Wikipedia[2] defines a framework as “a real or conceptual structure intended to serve as a support or guide for the

building of something that expands the structure into something useful.”

Thus a framework is an “external” structure that supports some activity and consists of various

things such as assumtions, practices, concepts, tools and various other things which can be used to create models and thus be useful for whatever activity for which these are being applied.

[1] http://www.thefreedictionary.com/framework [2] htttp://en.wikipedia.org/wiki/Framework

http://www.thefreedictionary.com/framework


Introduction to Framework Tester’s questions

How do I take advantage of my learning in the next project Can I compare metrics, and manage/improve for next

phases/regress cycles How can I reuse what I added as a method/adapted a tool/type

of testing, to reveal a class of defects How can I sharpen my skills <Problem solving, frees up time to

think of the problem and strategize, track my Knowledge Gap> How can I synthesize my learning, and choose to track what is

relevant How do I Learn from others? Explain my practice as a Transfer of Testing Technology, Mentor

some one


Introduction to Framework Tester’s outcomes

My diary of events – weblog, new tests, transient knowledge are made explicit

Helps me reflect on a new method/a tool that I adapted, that helped me reveal a class of defects

Enables to compile a tool box, which I have relationship with and will have inclination to pick up and use in a given situation

At each phase of a SDLC – Agile/Iterative/Spiral/Waterfall one can have an application of the FW for stages/parts

I can start identifying patterns, categorize - Taxonomy Leads to creating my database, becomes powerful when I start

sharing with another tester <helps validating use of my method to another context, can enable adding new features like a open source feedback>

Promotes to develop systematic problem solving, innovation


Introduction to Framework Lead/Manager’s point of view

How can I help my testing team view testing as a cohesive set of activities

Can I give a set of flexible tools, processes Guidelines as a starting point, team experiments and improves

as they do How do I bring in points of view, leverage team members

strengths, collect data on risks within and across projects How to evolve processes, metrics within the team Sharing of Lessons learnt How to manage transitions in leadership of test leads managers

with minimal impact to team and culture How can my team learn from other parts of the organization


Introduction to Framework Lead/Manager’s outcomes

Motivates teams to learn from each other’s experiences and answers emerge

Improve the effectiveness through Communication and Collaboration, process improvements, learning

Enables collective thinking on context specific Risks Models Metrics Dashboard

Empowerment, Trust and Open feedback


Introduction to Framework Process point of view

How can we evolve a right Test strategy Can we tailor depth of testing Can we bring out reporting of metrics based on goals

Process point of view outcomes Higher test quality With depth of testing, allows teams to choose the mix on testing Aligns to goals, real data as the project unfolds


Simple


Expanded

Applying Framework: Requirements, Design


Applying Framework-Requirements

Tool box Story boards Context Free questions (reference: Exploring requirements, Weinberg and Gause)

Who is the client What is a highly successful solution worth to this client What is the real reason for wanting to solve this problem How much time do we have for this project What problems does this product solve/create What environment is this product likely to encounter

Review requirements Stressing words o “Mary had a little lamb”, <It was Mary’s lamb and not some one else>o “Mary had a little lamb” <She no longer has the lamb>

Interpretation against various contexts by substituting synonyms <examine dictionary meanings>

Noun, Verb techniqueo Look for properties of this noun, ask What, Why, When, Where, Who, Which, How,o How much/manyo Look for properties of this verb, ask What, Why, When, Where, Who, Which, How,o How much/many

Modeling State transitions, equivalence partitions, Requirement formalism that can enable test case generation from UML, Specification and

Description Language, Entity Relation ship Diagrams Categorize, Risk Prioritization Q-Patterns: User centric views


Understand underlying and impacting technology Complexities due to shifts such as: External failures leads to increase in Data

protection/replication, Vulnerability and faults leads to increase in coverage for Security testing

Understanding Technology trends and adapt to those shifts ahead, or in parallel. Some examples are

SaaS, SoA Rich embedded devices Wireless/Security

Processes Development models

RUP – Use cases to derive tests Agile – Story boards, WiKi V model – Acceptance tests first

Metrics Trace ability – Every requirement is mapped to one or more test cases IEEE standard on coverage - The degree to which a given test or set of tests addresses all specified requirements for a given system or

component. Your own Requirement Coverage Index – Rt/SR where Rt = Number of requirements or Use Cases for which Test cases has been

written and SR = Number of specified requirements or use cases within scope

Improvements Defect escape in Requirement phase Requirements change that lead to a risk, in spite of change management

Applying Framework Requirements


Skills/knowledge Skills inventory

o That will enable me for Planning, Analyzing, Implementing, Designing, Executing

Skills requiredo Elicitation, Interviewing, Visual methods. Ability to reviewo Understand constrains, Tradeoffs

Sources for acquiring skillso SPIN groupso IEEE Requirements Engineering conferences

Contextual– New idea or product

– Tester who comes in when most of the requirements are laid out will need to o Clarify the solution, even though the designers would have gone through a certain level of exploring the solution, scope

definition. o Begin again, since the tester would have missed the beginning process of thoughtso Understand who are the users and the perceive needs

– Existing solution– Tester, may not have the benefit of an updated document and could have the following approaches

o Hands on of the working product or an equivalent solution, studying every possible action that can be exercised and see which of them map to the new solution

o Traverse through every functiono Understand existing design documents, to understand various interfaces between the system modules

– Enhancement versionso Impact of new requirementso How to manage continuous new requirements o Bugs as a source of enhancements/new requirements

Applying Framework Requirements


Tool box Resources

Bookmarkso http://www.testingfaqs.org/t-design.htmlo http://www.testingeducation.org/BBST/ (BBST course from Cem Kaner)o http://www.satisfice.com/tools/satisfice-tsm-4p.pdfo Grochtmann, M., and Wegener, J. "Test Case Design Using Classification-Trees and the Classification-

Tree Editor CTE"

Articles: Practitioners sharing on stickyminds.com, Testing experts blogs Bookso Lee Copeland: A practitioner's guide to software test designo Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyen (1999) o Boris Beizer: Test Design techniques o Robert Binder. Testing Object-oriented Systems

Documents Checklists

Test target checklist Pradeep Soundarajan’s screen saver

Templates IEE829 Test design, test case and test procedure templates Rex Black’s excel template Tabular template

Q-Patterns, Many Q-Patterns exists for various domains

Applying Framework Design


Tool box Techniques

o Boundary Value Analysis, Equivalence Partitioningo Cause-effect diagram, Decision tables, Orthogonal arrays and all-pairso State-Transition tables, Finite State machines do Node/Edge coverageo Extension to Noun-Verb Techniqueo Heuristic or exploratory testso Domain based tests, Syntax testingo Bug taxonomy based test designo Fault/attack models

Lessons learnto "Lessons Learnt", examples, storieso Using bugs to find gaps in written testso Uselessness/Usefulness of: Detailed test scripts, Conversion of bugs to test cases, Group review of test

caseso Gray-box test designo Stories: Reduction in bug count for Mobile Notes

Tools Open-Source/Free-ware/COTSo All-Pairs, Jenny, Multi, Dadada o COTS: Case maker BenderRBT o Test data generatorso Visualization tools



Processes Testing design processes

o FSM models can give test cases early

Modelso Agile/Test Driven – focus on automated unit testingo V model – Early test designo RUP – Use case driven test design

Metricso Test case design productivityo Test case per functiono Test case to bug ratioo Missed test cases percentage

Process improvementso Bug taxonomy based test designo Improving coverage by bug analysiso Improving coverage by reliability analysis

Skills/knowledge Skills inventory

o That will enable me for Planning, Analyzing, Implementing, Designing, Executing

Knowledge mapo Domain, Product, Technology

Skills required o Estimation, Test design techniques

Sources for acquiring skillso Certification – ISTQB, CSTE, CSTPo Education – BBST courseo Practice – Volunteer for Test design for projects, for open source projects, reading test cases for various open-source

projects



Framework: Caveats Possibility of

unwieldy nodes, contradictions

It is not just a simple application to help in testing nor is it an automation tool based framework. It is also not a new method to do testing or a new process model!

Framework is not available as an IDE or a tool but that is something that authors consider as work in progress.


More information

http://www.whatistesting.com/qpatterns.htm

Mail: vipul at PureTesting.com

http://www.whatistesting.com/qpatterns.htm

Thanks

vipul kocher - software testing, a framework based approach

Software

orgwikiframeworkc vipul

innovationc vipul kocher

onec vipul kocher

rc vipul kocher

approachvipul kocher

testing experiencecan

software testing

better testing