Tests for Every Branch Using CircleCI & Sauce Labs to

Continuously Test CS Curriculum at Code.org

@bcjordanBrian Jordan, software engineer at Code.org

Code.org

non-profit

expanding participation in CS

Hour of Code

full curricula, district partnerships,

professional development, policy change

K-12 CS Curriculum

43,000 K-12 teachers trained

courses: 400k teachers 12mm students

how did Code.org start automated testing?

development in 2013-2014

why test? what is Code.org's testing context?

the context

the context

the context

the context

the context

RTL LTR

the context

the context

so how do we test all that?

how we started Selenium testing

how our tests are organized and run

selenium-webdriver cucumber

takeaways

Build a library of steps

Build a library of test levels

Build a library of annotations

@no_mobile@no_ie9

@skip

@db_access@as_student

@eyes

who writes tests?

who/what runs tests?

2015: one pipeline

2016: every commit!

what challenges did we end up facing?

challenge:multiple browsers

solution:selenium-webdriver

Sauce Labs

challenge:

testing new changes

locally:chromedriver Sauce Connect

challenge:

big buckets of changes

pull request tests in

❤

pull request tests incircle.yml

github.com/code-dot-org/code-dot-org

tips and tricks

github.com/code-dot-org/code-dot-org

⚡

tips and tricks

github.com/code-dot-org/code-dot-org

⏰

tipsadding more [commit tags]

tips

use Sauce Connect for Circle <=> Sauce Labs

tips

use Sauce Connect for Circle <=> Sauce Labs

tips

use Debug via SSH and Manual Sessions for debugging

tipsdebugging failures—APIs to the rescue!

tipsdebugging failures—APIs to the rescue!

challenge:

visual, responsive changes

hackathon solution?

not easy...

actual thing!

using eyes

what do tests look like when running?

Speed Run!

(not) wasting time

speeding up test runs

2014: 60-90 min of tests

parallelizationat browser level

2015: 60-90 min of browser tests

to 20-30 min

parallelizationby containers

parallelizationby containers

justifying tests

the bug collection

missing button

button height change

extra margin

overflow: hidden added

duplicate button

stray pixel

Firefox getBBox()

Chrome 50 (deprecated SVGElement.offsetWidth)

my favorite bug ever

who investigates failures?

dev of the day

dev of the day😭

dev of the day+

what's next?

further flakiness investigations

(less flakiness: more parallelization with less pain)

💪

Applitools Eyes test results as GitHub PR comments

takeaways

start small

everybody tests

periodically invest in speed-ups

In memory of Laurel Fan

Thanks to the Code.org team who spread the burden of f leshing out, monitoring and improving our test infrastructure, especially Laurel Fan, Brendan Reville, and Brad Buchanan.

Thanks to the kind folks at Sauce Labs, CircleCI and Applitools who helped us get set up, and to Dave Haeffner @tourdedave for sharing his advice on testing.

code.org/help

github.com/code-dot-org

Thanks!

@bcjordanBrian Jordan, software engineer at Code.org

Q&A