from manual to automated tests - stac 2015

MOVE FAST AND FIX TESTS

Shauvik Roy Choudhary @shauvik http://shauvik.com

GOING FROM MANUAL TO AUTOMATED:

MOVE FAST AND FIX TESTS

Shauvik Roy Choudhary @shauvik http://shauvik.com

GOING FROM MANUAL TO AUTOMATED:

MOVE FAST AND FIX TESTS

Shauvik Roy Choudhary @shauvik http://shauvik.com

GOING FROM MANUAL TO AUTOMATED:

Checks

ABOUT ME• PhD Thesis:

Cross-platform Testing & Maintenance of Web & Mobile Apps

• Industry: Google, Yahoo!, FullStory, Fujitsu Labs, IBM Research, HSBC, Goldman Sachs

• Entrepreneurship: Georgia Tech TI:GER program, VentureLab, ATDC, TechSquare Labs

Shauvik Roy Choudhary PhD, Georgia Tech

Founder, CheckDroid

OUTLINE• MOTIVATION: Why Automate ?

• MOVE FAST: How to Automate Quickly?

• FIX TESTS: How to Resolve Automation Issues?

• SCENARIO: Android UI Testing

• DISCUSSION

WHY AUTOMATE TESTS?

Software

Manual Testing

WHY AUTOMATE TESTS?

Software

Manual Testing

WHY AUTOMATE TESTS?

Software

Manual Testing

WHY AUTOMATE TESTS?

Software

Manual Testing

WHY AUTOMATE TESTS?

Software

Manual Testing

PLATFORMS

Web Tester

PLATFORMS

Web Tester

PLATFORMS

Web Tester

PLATFORMS

Web Tester

PLATFORMS

Web Tester

PLATFORMS

Web Tester

PLATFORMS

Web Tester

PLATFORMS

Web Tester

Mobile Tester

LollipopAndroid 5.0

9

HOW TO AUTOMATE?

HOW TO AUTOMATE?

WHAT

Parts of your App

HOW TO AUTOMATE?

HOW

Infrastructure

Tool Support

WHAT

Parts of your App

WHAT: TO TEST

• Functional Tests Positive and negative scenarios

• Non-Functional Tests Performance, Security, Compatibility…

• Previous Issues (identified Manually) Never to occur in Regressions

HOW: INFRASTRUCTURE

• Unit TestsxUnit for different programming languages

• UI TestsSelenium/WebDriver for Web Apps Appium for Mobile (iOS, Android)Espresso for Android

IDEAL TESTING PYRAMIDby Mike Cohn

ICE-CREAM CONE

HOUR GLASS

source: just-about.net

TESTING DIAMOND

Source: toddlittleweb.com

CHRISTMAS TREE

source: just-about.net

UI TESTS CLOSEST TO

END-USER TESTING

MOVE FASTHow to automate quickly?

RELIABLE INFRASTRUCTURE

VISUAL TEST RECORDING

FIX TESTSHow to fix broken tests?

UI TESTING ISSUES

UI TESTING ISSUES

• FRAGILITY

UI TESTING ISSUES

• FRAGILITY

• SPEED

UI TESTING ISSUES

• FRAGILITY

• SPEED

• FLAKINESS

1. FRAGILE TESTS

REDUCING FRAGILITY

• Use durable selectors

• IDs over XPath

• Relative XPaths over Absolute XPaths

• Other durable labels (class, content-desc..)

2. SLOW TESTS

MAKING TESTS FAST

• Don’t use Sleeps

• If absolutely necessary, sleep while polling

• Replace slow resources with stubs

doSomething()Thread.sleep(1000) checkSomething()

doSomething()do{ Thread.sleep(1000) }while(!loaded) checkSomething()

FLAKINESSUI Thread

Test Thread

Work in Queue

FLAKINESSUI Thread

Test Thread

Work in Queue

click

FLAKINESSUI Thread

Test Thread

Work in Queue

click

Motion down

Motion up

FLAKINESSUI Thread

Test Thread

Work in Queue

click

Motion down

Motion up

assert

FLAKINESSUI Thread

Test Thread

Work in Queue

click

Motion down

Motion up

assert

FLAKINESSUI Thread

Test Thread

Work in Queue

click

Motion down

Motion up

assertsleep

FLAKINESSUI Thread

Test Thread

Work in Queue

click

Motion down

Motion up

assertsleep

TESTS

3. FLAKY TESTS

Image credits: Google, GTAC talk

REDUCING FLAKINESS• Have a hermetic test environment

Source: googletesting.blogspot.com

REDUCING FLAKINESS• Have a hermetic test environment

Source: googletesting.blogspot.com

FIX TESTS

1. FRAGILE TESTS

FIX TESTS

1. FRAGILE TESTS

2. SLOW TESTS

FIX TESTS

1. FRAGILE TESTS

2. SLOW TESTS

3. FLAKY TESTS

Image credits: Google, GTAC talk

FIX TESTS

ANDROID UI TESTING

MOBILE TESTING PYRAMID

Source: Daniel Knott, AdventuresInQAAuthor: Hand-On Mobile App Testing

ANDROID TESTING TOOLS

Image credits: BitBar

ESPRESSO: HOW IT WORKSUI Thread

Test Thread

Work in Queue

Test Case

Motion down

Motion upEspresso

action assert

Blocked

ESPRESSO METHODS

onView(Matcher) .perform(ViewAction) .check(ViewAssertion)

EXAMPLES

onView(withId(R.id.greeting)) .perform(click());

onView(withText("Hello Steve!")) .check(matches(isDisplayed()));

EXAMPLES

onView(withId(R.id.greeting)) .perform(click());

onView(withText("Hello Steve!")) .check(matches(isDisplayed()));

Find

EXAMPLES

onView(withId(R.id.greeting)) .perform(click());

onView(withText("Hello Steve!")) .check(matches(isDisplayed()));

Find

Do Stuff

EXAMPLES

onView(withId(R.id.greeting)) .perform(click());

onView(withText("Hello Steve!")) .check(matches(isDisplayed()));

Find

Do Stuff

Check

onView(Matcher<View>)

●  withId ●  withText ●  withContentDescription ●  isDisplayed ●  hasFocus ●  hasSibling ●  ... ●  mySpecialMatcher

perform(ViewAction)

●  click ●  longClick ●  doubleClick ●  typeText ●  scrollTo ●  ... ●  myCustomAction

check(ViewAssertion)

●  matches(...) ●  doesNotExist ●  myCustomAssertion

HIERARCHYwithParent(Matcher)withChild(Matcher)hasDescendant(Matcher)isDescendantOfA(Matcher)hasSibling(Matcher)isRoot()

UI PROPERTIESisDisplayed()isCompletelyDisplayed()isEnabled()hasFocus()isClickable()isChecked()isNotChecked()withEffectiveVisibility(…)isSelected()

ROOT MATCHERSisFocusable()isTouchable()isDialog()withDecorView(…)isPlatformPopup()

COMMON HAMCRESTMATCHERSallOf(Matchers)anyOf(Matchers)is(...)not(...)endsWith(String)startsWith(String)

SEE ALSOPreference matchersCursor matchers

USER PROPERTIESwithId(…)withText(…)withTagKey(…)withTagValue(…)hasContentDescription(…)withContentDescription(…)withHint(…)withSpinnerText(…)hasLinks()hasEllipsizedText()hasMultilineText()

INPUTsupportsInputMethods(…)hasImeAction(…)

CLASSisAssignableFrom(…)withClassName(…)

MatchersCLICK/PRESSclick()doubleClick()longClick()pressBack()pressImeActionButton()pressKey([int/EspressoKey])pressMenuKey()closeSoftKeyboard()openLink(…)

GESTURESscrollTo()swipeLeft()swipeRight()swipeUp()swipeDown()

TEXTclearText()typeText(String)typeTextIntoFocusedView(String)replaceText(String)

POSITION ASSERTIONSisLeftOf(Matcher)isRightOf(Matcher)isLeftAlignedWith(Matcher)isRightAlignedWith(Matcher)isAbove(Matcher)isBelow(Matcher)isBottomAlignedWith(Matcher)isTopAlignedWith(Matcher)

LAYOUT ASSERTIONSnoEllipsizedText(Matcher)noMultilineButtons()noOverlaps([Matcher])

View Actions

matches(Matcher)doesNotExist()selectedDescendantsMatch(…)

View Assertions

onView(Matcher) .perform(ViewAction) .check(ViewAssertion)

CHEAT SHEET

2.0

BARISTAEspresso Test Generation

http://checkdroid.com/barista

DEMO

Tests

TEST CLOUD

WHY AUTOMATE TESTS?

Software

Manual Testing

WHY AUTOMATE TESTS?

Software

Manual Testing

MOVE FASTHow to automate quickly?

WHY AUTOMATE TESTS?

Software

Manual Testing

MOVE FASTHow to automate quickly?

1. FRAGILE TESTS

2. SLOW TESTS

3. FLAKY TESTS

Image credits: Google, GTAC talk

FIX TESTS

WHY AUTOMATE TESTS?

Software

Manual Testing

MOVE FASTHow to automate quickly?

1. FRAGILE TESTS

2. SLOW TESTS

3. FLAKY TESTS

Image credits: Google, GTAC talk

FIX TESTSANDROID UI TESTING

DISCUSSIONAre you currently automating tests?

What are some challenges that you see?

THANKS!Shauvik Roy Choudhary

@shauvik http://shauvik.com