Upload File

decreasing false positives in automated testing

30

Click here to load reader

Upload: sauce-labs

Post on 15-Jul-2015

1.783 views

Category:

Engineering


2 download

Report

TRANSCRIPT

Page 1: Decreasing false positives in automated testing

Reducing False

Positives

&

Presents

Page 2: Decreasing false positives in automated testing

QASource Confidential

QASource & automation

100,000+ automated test cases

100+ Selenium engineers

500+ years collective automation experience

100+ automation frameworks

Page 3: Decreasing false positives in automated testing

QASource Confidential

Question #1

How many years of experience do you have with Selenium?

< 2 years 2-4 years 5+ years

What is the size of your automation suite?

< 100

test cases

100-250

test cases

250-500

test cases

500+

test cases

What is the most important reason why you invest in automation?

Save MoneySave Time/

Release Faster

More QA

Coverage

Question #2

Question #3

Attendees poll

Page 4: Decreasing false positives in automated testing

QASource Confidential

Anand Ramakrishnan

QA Director,

QASource

• 15+ years of automation experience

• Directing a team of 150 engineers

• M.S. in Computer Applications

• M.S. in Cyber Law & Security

Page 5: Decreasing false positives in automated testing

QASource Confidential

Key learnings & takeaways

What are false positives and why do they occur

Common causes and the challenges

How to implement key strategies to reduce them

Case studies and real world examples

FREE step by step guide to creating CSS locators

Page 6: Decreasing false positives in automated testing

QASource Confidential

What are false positives?

Tests that are marked as

failed, when in reality they

should have passed or should

functionally work.

Page 7: Decreasing false positives in automated testing

QASource Confidential

Why do false positives occur?

Automation

ApproachImplemented

FrameworkWritten Test

Page 8: Decreasing false positives in automated testing

QASource Confidential

Relying on UI

Relying on sequence of execution

Manual intervention

Slow performance

Reference data change

Shared environment

Change in UI element properties

Potential causes for false positives

Page 9: Decreasing false positives in automated testing

QASource Confidential

Impacts on automation

Ignoring failures

Wasting time investigating

failuresDecrease in productivity

Frustration for engineering team

Risk of missing potential bug

Increasing cost of automation

Uncertainty of application health

Page 10: Decreasing false positives in automated testing

QASource Confidential

Attendees poll

What is the rate of false positives

in your automation?

5% or less 5% to 15% 15% to 20% 20% or more

Page 11: Decreasing false positives in automated testing

QASource Confidential

Complex workflow application

Rich interactive UI

2500+ automated test cases

Page object pattern approach

Multiple OS, browsers and environments

Case study

Page 12: Decreasing false positives in automated testing

QASource Confidential

Frequent UI Changes

Synchronization issues

Memory leak issue

Challenges

Elements identified using

Xpath

Hardcoded waits

No tear down

ROOT CAUSE

CH

ALLEN

GES

Page 13: Decreasing false positives in automated testing

QASource Confidential

Keys to reducing false positives

Deploy application on optimal

configurations

Page 14: Decreasing false positives in automated testing

QASource Confidential

Keys to reducing false positives

Controlled environment for automation

Page 15: Decreasing false positives in automated testing

QASource Confidential

Keys to reducing false positives

Keeping tests short

Page 16: Decreasing false positives in automated testing

QASource Confidential

Keys to reducing false positives

Keeping tests independent

TEST SCRIPT 1

TEST SCRIPT 2

TEST SCRIPT 3

TEST SCRIPT 4

Create User

Create Group

Add User to Group

Search User

Page 17: Decreasing false positives in automated testing

QASource Confidential

Using right locators for objects identification

Keys to reducing false positives

IDs

Name

CSS locators

Page 18: Decreasing false positives in automated testing

QASource Confidential

Bad Practice

Object identification strategy

Good Practice

Page 19: Decreasing false positives in automated testing

QASource Confidential

Keys to reducing false positives

Tear-down approach

Page 20: Decreasing false positives in automated testing

QASource Confidential

Bad Practice

Tear down approach

Page 21: Decreasing false positives in automated testing

QASource Confidential

Good Practice

Tear down approach

Page 22: Decreasing false positives in automated testing

QASource Confidential

Keys to reducing false positives

Dynamic object synchronization

Page 23: Decreasing false positives in automated testing

QASource Confidential

Bad Practice

Object synchronization

Page 24: Decreasing false positives in automated testing

QASource Confidential

Good Practice

Object synchronization

Page 25: Decreasing false positives in automated testing

QASource Confidential

Keys to reducing false positives

Re-execution capability

RE-EXECUTION

LOGIC FLAG

TEST SUITE

TEST CASE 1

TEST CASE 2

—————

—————

—————

TEST CASE N

Pass

Failed

RE-EXECUTION OF

FAILED TESTS

Test case

name

Status Re-

executed

TEST CASE 1PASS NO

TEST CASE 2 PASSYES

TEST CASE 3 FAIL YES

_____ _____ _____

TEST CASE N PASS YES

EXECUTION

EXECUTION REPORT

Page 26: Decreasing false positives in automated testing

QASource Confidential

Conclusion

Using right locators for objects identification

Dynamic object synchronization

Re-execution capability

Tear-down approach

Keeping tests short Keeping tests independent

Controlled environment for automation

Deploy application on optimal configuration

Page 27: Decreasing false positives in automated testing

QASource Confidential

Benefits of eliminating false positives

Certainty of application health

Increase in productivity

Save time not investigating false

positives

Will not miss potential bugs

Decrease cost of automation

Page 28: Decreasing false positives in automated testing

QASource Confidential

Get your FREE Step by Step

Guide to Create CSS

Locators

qasource.com/[email protected]

Page 29: Decreasing false positives in automated testing

QASource Confidential

Q&A

Page 30: Decreasing false positives in automated testing

QASource Confidential


False Negatives and False Positives in High Throughput Screening … · 2005-06-17 · False Negatives and False Positives in High ... 1 project –Additional validated hits:

PrivacEye: Privacy-Preserving Head-Mounted Eye …...4 Julian Steil, Marion Koelle, Wilko Heuten, Susanne Boll, and Andreas Bulling (a) True positives (b) False positives (c) False

Reducing False Positives When Monitoring for Low ... · Reduction of False Positives • Improved operator training—on causes of false positives • Proper equipment set up—to

When Hackers Err The Impacts of False Positives on

Reducing False Positives In Automated Testing

Correcting false discovery rates for their bias toward ... · Correcting false discovery rates for their bias toward false positives 2017 Annual Meeting of the Statistical Society

CCEEU and FIDH - False Positives, The War is Measured in Litres of Blood

LNCS 4338 - Reducing False Positives in Video Shot ...mnitya/Icvgip06.pdf · Reducing False Positives in Video Shot Detection Using Learning Techniques 423 1. Illumination changes:

Reducing False Positives with Automated NMR Verification · Reducing False Positives with Automated NMR Verification Author: Ryan Sasaki Subject: SMASH 2011 Created Date: 10/25/2011

Reducing False Positives of a Bloom Filter using Cross ... · Reducing False Positives of a Bloom Filter using Cross-Checking Bloom Filters Hyesook Lim1, Nara Lee1, Jungwon Lee1 and

Methods The difficulty of avoiding false positives in ......Methods The difficulty of avoiding false positives in genome scans for natural selection Swapan Mallick, 1,2,3 Sante Gnerre,

State-of-the-Art Blocking of False Positives

Reducing False Positives without Increasing Regulatory Risk · 2 | REDUCING FALSE POSITIVES WITHOUT INCREASING REGULATORY RISK Achieving a Balance Financial institutions are instructed

Reducing IDS False Positives by Clustering Related Alerts

Reduction of False Positives in Lung Nodule Detection

Efficiently Deducing Ids False Positives System Profiling 33223

Reducing False-Positives and False-Negatives in Security Event Data Using Context

Handling False Positives in PVS-Studio and CppCat

CTScanninginSuspectedStrokeorHeadTrauma:IsitWorth Going ... · Chest CT results were 6 true-positives, 15 true-negatives, 5 false-positives, and 1 false-negative. Diagnostic performance

Reducing False-Positives and False-Negatives in Security Event Data Using Context Derek G. Shaw August 2011

FEV’s Greatest Bloopers: False Positives in Formal Equivalence

Dealing with False Positives in Intrusion Detection · Presentation Overview Introduction & Problem Statement Characteristics of IDS alarms Filtering False Positives Problems with

ROC Curves. True positives and False positives True positive rate is TP = P correctly classified / P False positive rate is FP = N incorrectly classified

Understanding Type Changes in Java - Danny Dig...Table1shows the number of true positives (TP), false positives (FP), and false nega- tives (FN) detected/missed by RefactoringMiner

Modern Diagnostic Tests for Chronic Lyme Disease and Co ...determine false-positive results, the C6 ELISA generated 73.6% false positives with 5% prevalence and 4.7% false positives

Blame it on you for the false positives

Central Statistical False Positives –Predicting True

Adaptive Alert Classification to Reduce False Positives in ...tadek.pietraszek.org/publications/pietraszek06_alert-thesis-slides.pdf · Alert classification to reduce false positives

Tentacle probes: eliminating false positives without ...€¦ · Tentacle probes: eliminating false positives without sacrificing sensitivity Brent C. Satterfield1,2, Jay A.A. West2,*

FEV’s Greatest Bloopers: False Positives in Formal Equivalence · 1 February 21-23, 2007 FEV’s Greatest Bloopers: False Positives in Formal Equivalence Erik Seligman, Joonyoung

Tail estimation for false positives in high-throughput testingrootzen/papers/high-throughput... · 2012-03-16 · Tail estimation for false positives in high-throughput testing

How Not to Build a State: Evidence from Colombia's False Positives · 2015-03-24 · How Not to Build a State: Evidence from Colombia’s False Positives D. Acemoglu MIT L. Fergusson

Reduction of False Positives in Structure-Based Virtual

State-of-the-Art Blocking of False Positives - Scantibodies

REDUCING FALSE POSITIVES BY MARKING AND OVERCLASSIFYING · Reducing false positives by marking and overclassifying 19 Stage I. Detecting skin prototypes regions in the image obtained