test automation – solving common challenges through the optical layer

Test Automation:

Solving Common Challenges Through

“Black Box” Testing

A Primer on Using Image-Based Testing and

Device Control

July 8th, 2014

2 ©Alliance Global Services 2014

Today’s Presenter

As Director of Solution Engineering, Joshua Berry has 20 years

of industry experience, with more than a decade in leadership

and innovation. He is a leader within Alliance’s Solution

Engineering organization in developing cutting edge engineered

solutions. His focus includes test automation and tooling,

problem solving, product testing, and agile techniques in

software development.

Today’s Host

As Vice President, Sales, Bob Donaldson is responsible for

driving and overseeing Alliance’s sales and business

development activities while focusing on client value and client

service that translates into long-term partnerships and revenue

growth. Bob has over 20 years of experience in sales

management and operations. Bob holds a B.A. from

Elizabethtown College, Pa.


Overview

• Background

• Tool Agnostic Presentation

• Assumes Automation Exposure

• Agenda

• Problem Statement

• Black Box Testing Control and Feedback

• Image Analysis Best Practices & Tricks

• Solutioning Black Box Automation

• Real World Solutions

• Integrating IBT and Object-Based Testing


The “Textbook” Definition

What is Software Test Automation?

In other words,

Test Automation is using software to test software. One part test engineering + one part software engineering.

Software Test Automation is the application of computer software and electro-mechanical equipment to:

• Dynamically Control the execution of tests, • Compare actual outcomes to predicted outcomes, • Set up test preconditions and cleanup test modifications, • Perform other test control and test reporting functions.


Problem Statement

• Traditional Automation uses application Objects for feedback and

control.

• There are many cases where this does not apply:

– Device Automation

– The test is of video or image content

– Control’s object cannot be accessed


• All applications have physical input and output

• Primary User Interface

• “Black box” Automation interacts as a user would

– Image-Based Automation

• Reference Bitmaps and Optical Character Recognition (OCR)

• Real-time analysis

– A software or physical input harness

• Touch the application

• Real command format

Potential Solution: Black Box Automation

Input HarnessVideo Analysis Card


Object vs Image Based Testing

Object Based Testing (OBT)

• Relies on “under the hood” access to application controls

• Can quickly interact with an object

• Object can be hidden

• Can get highly detailed information about the object

• Can be highly dynamic using descriptive programming

• Some technologies, e.g. Flash, Silverlight difficult to automate

• Poor at testing “look and feel”

• Relies on good programming

• Tool or agent must be loaded on the System Under Test.

Image Based Testing (IBT)

• Relies on testing the graphical output as displayed on the screen

• Based on X,Y Coordinate system, Origin usually at upper left corner

• OCR analysis for text analysis

• Can enable cross-platform, touch-screen application testing

• Can be remotely installed

• Bitmap comparison takes time

• Fragile to slight UI changes

• Must capture images ahead of time

• Not recommended for enterprise or when object available


First Generation Image Analysis

• Analyze every pixel until failure

• Configurable tolerances for RGB variance

• Set the % match for pass / fail

Primitive Pixel Comparison

• Have to teach tool the font

• Unreliable / expensive OCR engines Primitive OCR

• Highly sensitive to changes in images

• Sensitive to video cards and screen settings

• Complex logic to build in basic robustness, such as location variance

Problems with First gen IBT:


Second Generation Image Analysis

• Treats image holistically

• Resolves images into primitives

• Can accommodate Size / Zoom, Rotation, Tilt / Skew, Partially Covered Images

• Facial Recognition and Pattern Template Analysis

Fuzzy Logic Image Analysis

• Reliable OCR Engines

• Open source near as reliable as commercial Advanced OCR

• Image still has to be visible

• Still sensitive to changes in base OS

• Greater chance of a false positive

Problems with Second Gen:


IMAGE ANALYSIS BEST PRACTICES

AND TRICKS


Back to Basics A Good Framework and Engineering Design

• All Automation Best Practices still apply

– Error and Exception Handling

– Logging and Reporting

– Scripting

– Test Data Management

• Temptation of Capture-Replay and Object-Action scripting

– Robust framework of reusable, modular test verbs to handle complex actions

– Don’t hard code navigation

– IBT is just another way of feedback

• Can still have common framework features

– Scriptless Automation

– Continuous Integration

– TCM Management Tools


Setup and Other General Guidelines

• Rule of Thumb: If it is hard for you to see it, it is hard for

the tool to see it.

• Set application desktop backgrounds to a standardized,

light color to provide contrast

• Standardize automation PC’s and device under test to a

standard video setting

– Best Resolution available

– Color Palette

• Avoid regions of the screen that change:

– Clocks

– Video

– Weather plug-ins

– Ads

• Be careful of “bleed through” of images near dynamic

sections of screen


Design a Framework that “Knows” Your Application

• Create a virtual map of the screen in your framework.

• “Know” where objects might be

• Limit verifications and actions to proper region

Logo Quick Navigate Live Video

Listings

Channel List

Time


Reference Images • Smallest Image that still has enough information

– Don’t always need the full image

– Example initial image: “Hello Kitty”

– Optimized reference image might look like:

– Is this reference image good enough to ensure Hello Kitty and only Hello Kitty is found?

– “Hello Stachie!!”

• Make use of Masking

– Most tools allow you to tell it to ignore regions

– A way of optimizing the bitmap

– Also a way of reusing a bitmap

• Pick images with high contrast

• Choose meaningful bitmap names

• Take variations of same image per OS

Masked Region


• Minimize the amount of bitmap / OCR verifications

• Avoid verifying images in exact X,Y Coordinates

• Search in a reasonable sized region

• Use OCR for text validation

• Avoid OCR over opaque / variable backgrounds

– OCR Engines need strong contrast between letter and background

– Text over video

Search Region Buffer

Exact X,Y Position

Searching for Bitmaps and Text


Use of Highlights

• Many Applications make use of a highlight, as shown below:

• For validation and navigation routines, your framework needs

to know where the highlight is.

– Map out where the buttons could be

– Create a bitmap of just the highlight with the inside masked.

– For variable width highlights, mask the ends.

– Use the tool to locate bitmap in icon region.

– To validate the highlight:

• Compare X,Y coordinates to the map,

• Dynamically determine what is at that position.


On-Screen Navigation in an IBT World

1. Determine X,Y Coordinate of Target Button.

2. Determine X,Y Coordinate of Highlight

3. Calculate Delta X,Y Navigation Vector from Highlight to Target

4. Move Up / Down / Left / Right appropriately

5. Recheck X,Y Coordinate of highlight

6. If within tolerance, Press <Enter>, else Step 3


Cross-Screen Navigation in an IBT world

• Develop Cross-Screen Navigation Routes

– Use Datums and Waypoints

– Map how to navigate to all screens from a common datum

– Find common screen between current and target screen

– <Back> to common screen, then follow Navigation to target screen

• Screen Identification

– Identify properties that optimally identify each screen

– Create an IdentifyScreen() function

• Example Navigation Map

– Language Setup: Menu, Setup, Language

– Video Setup: Menu, Setup, Video.

– Search: Menu, Search


SOLUTIONING BLACK-BOX AUTOMATION


• Direct Connection

• IP Stream

• VNC / RDP

• Webcam

• OBT / Feedback Agent

• Debug Port / Logs

• Open Loop Record

Image Capturing / Feedback Architectures

Test Automation PC

Automation Software

Device Under Test

App Under Test

App Under Test

Video Output

VNC Client

VNC Server

Video Output

OBT Feedback

Agent

Debug Data


• OBT

• Coordinate Click / Keys

• VNC / RDP

• Data / Diagnostic Port

• Device Agent / Instrumented Code

• Physical Test Harness – IR / RF Signals

– Robotics / Servos

– Interrupting Native I/O

– Voice Control

Application / Device Control Architectures

Test Automation PC

Automation Software

Device Under Test

App Under Test

App Under Test

VNC Client

VNC Server

Diagnostic PortControl Agent

Physical Test Harness

Clicks and Keys


• The means of control affects the test validity and accuracy

• It is up to the engineer to determine which type of architecture to use

• Real-world defects

– Key Queue

– Touch Screen

The Impact of the Solution

Input Harness

VNC / RDP

OBT

Debug Control

Physical Interrupt


Embedded Devices

• Software running in devices that can’t run a standard test tool Description

• Mobile, Cable STB, Game Consoles

• Video Conference, POS Terminals, Delivery Manifests

• Automotive Telematics / Sat-Nav Systems Examples

• Video Output

• VNC / RDP

• Webcam

• OBT Harness

Feedback

• Physical Hardware

• VNC / RDP

• Diagnostic Port

• OBT Harness

Control


Mobile and Touchscreen

• Specialized subset of Embedded Devices Description

• Jail breaking and instrumenting code often not allowed

• Getting a standard tool across Android / iOS / Win / Blackberry

• iOS5, iOS6 allows some VNC apps, iOS7 does not.

• iOS8 Health Check breaking “sandbox” paradigm.

• Standardizing tooling and testing with Web versions of the same application.

• Automating phone Interrupts and Interoperability

• A text or call interrupt

• Loss / Gain of WiFi

• Screen orientation change

Challenge(s)

• OBT Harnesses

• Some Tests manually, e.g. orientation, iOS7

• Harness on WiFi Router

• 3PA Text and Fake Text / Call Apps

Solutions


Video / Image Validation

• Testing the content of a picture or video independent of the platform application resides. Description

• Correct Image: User Authentication, UI Validation

• Streaming Video: eLearning, VOD, Video Ads

• Video / Image Quality: Streaming, Video Conference Examples

• Canned Pictures and Video Assets

• Short Loop for Video compare to partial still

• Canned High Bandwidth Videos

• “Moving Icon”

• Checksum validation

Successful Techniques


Custom Controls

• 3PA / non-standard controls OBT tools cannot recognize. Description:

• Not all parts of control visible

• Have to hunt for specific control

• Mapping can be difficult Challenges:

• Brute force click on parts of control and hope the hidden component appears

• Complex algorithm to determine region to search Solutions:


Some Current Tools Tool License Architecture Notes

QTP / UFT Commercial OBT tool with limited IBT added in 11.50 version, VNC to remote

Native mix of OBT and IBT

EggPlant Commercial VNC connection to remote machines

Pure IBT tool.

Ranorex Commercial Direct Connect IBT and OBT

Test Quest Commercial Direct Connect IBT

Sikuli Open-Source Work locally or through VNC IBT

Open.CV / EMGU Libraries

Open-Source C++ / C# image processing libraries

Powerful library to integrate into a tool

Robotium Open-Source Android OBT, Instrument

Frank Open-Source iOS OBT, Instrument

JAMO Commercial Mobile OBT

SeeTest Commercial Mobile OBT

Perfeco Mobile Commercial Cloud-hosted, Physical device input interrupt.

Integrates with UFT


SOLUTIONS FROM REAL-WORLD PROJECTS


Cable Set Top Boxes

• Control:

– PC-controlled IR Emitter taped to

front

– Key commands through diagnostics

– Power through PC control

• Feedback

– Diagnostic Port

– Real-Time Video Analysis

– Connected to TV’s for human

monitoring of test

• Both direct connection and IP-based

• Single STB and racks of 32


Robotic Tablet Automation

• Low cost high school robotics kit

• Webcam for feedback, Bitmap / OCR

• Custom control software: C#, EMGU

• CHALLENGES:

– Calibrating Webcam to tablet

– Virtual Coordinates of Webcam to Real Coordinates of Robot.

– Pressure / speed of stylus

• APPLICATIONS: Automotive Telematics, Point of Sale, Laptops


Visual Authentication

• User Security Image Problem Statement:

– Incorporate “Security Image” check into existing OBT automation test

bed

• Solution

– Create security images for each user in test environment

– Create meta data linking reference bitmaps to specific users

– Integrated open source comparison tool to framework

– VerifySecurityImage(UserName) to lookup reference image and

compare to screen.

• Video Asset Validation Problem Statement

– Ensure the correct video asset was streaming

• Solution

– Reduce assets to 15 – 20 second loops.

– Capture still at time = 5 seconds

– Crop reference bitmap and validate


Native Applications on Mobile and POS

• UFT 11.5x + SeeTest for OBT within mobile device

• Selenium + Siluki + VNC for IBT

• UFT 11.5x + VNC for IBT


Concluding Thoughts: Mixing IBT and OBT

• Image-Based Testing and Black Box Control are powerful and can solve many automation challenges.

– Object testers will have to adjust to changes such as Navigation

– Good Engineering still applies

– Use judgment with how you control the application

• A good automation engineer uses every means of feedback possible to suit to application and the test.

– There may be cases where one can mix IBT and OBT

– Image based feedback might only be 5% or 95%.

– Use IBT to augment OBT

– Look into tools like JAMO and SeeTest for mobile and devices

• Consider tools like UFT and Ranorex that can do IBT and OBT or integrate Image Analysis tools / libraries (e.g. Sikuli, EMGU) into OBT only tools (e.g. Selenium).


Questions?

Thank You

Joshua Berry

Director of Solution Engineering

[email protected]

www.allianceglobalservices.com

mailto:[email protected]

test automation – solving common challenges through the optical layer

Technology