sauce labs beta program overview

Beta Program Overview

Al [email protected]

+1 415 730 1430

April 15, 2023

DimDim tip

04/15/23 Sauce Labs Confidential 2

Choose Fit Screen

The Problem

Many, many functional tests…


Hundreds of test flows- Happy path- Corner cases - Negative tests- Before check-in, after check-in

Dozens of test platforms

Hundreds of test flows- Happy path- Corner cases - Negative tests- Before check-in, after check-in

… and test platforms


6, 7 (and soon 8)

9.5 (and soon 9.6)

1.0

3.0, 3.1 (and soon 4)

2, 3 (and soon 3.1)

Add-ons

Browser configurations

Languages/locales

… and test data


Lots of test data Correct data, Incorrect data, Missing data, Data overflow

Dozens of test platforms

6, 7 (and soon 8)

9.5 (and soon 9.6)

1.0

3.0, 3.1 (and soon 4)

2, 3 (and soon 3.1)

Add-ons

Browser configs

Languages/locales

…run on minimal test hardware.


Test jobs In-house test lab

Minutes

10 20 30 40 50 60 70 80 90

Test results

Can’t run all scripts + browsers

Can’t run all scripts + browsers

Test lab ratholeTest lab rathole Too long to find and fix bugs

Too long to find and fix bugs

How many tests?

Which browsers/OS combos?

How much data?


Concurrent Test Execution


Minutes

0 5

Test jobs

2000 nodes Test results

Run all scripts + browsers

Run all scripts + browsers

Find and fix bugs fast

Find and fix bugs fast

No configuration

needed

No configuration

needed

How many test machines?

How long to run today?

How long, ideally?

How often run today?

How often, ideally?


User Experience


Agile developer / tester

Source control

1 Check in code & test scripts

5 Run scripts

HTTP(s)

7 View results

Web, RSS

REST

8

Deletescripts & results

6Download results

REST

4 Upload scripts

Sauce LabsSauce Labs

FTP

App Under Test

App Under Test

3 Promote new build to cloud-accessible location

(Your infrastructure)

Start new build

Polling or trigger

Build System

2

cronrake

How does this map to your workflow?

Which CI or build system?

Build script language?

Test script language?


What’s in our cloud?



ScriptsSelenium test scriptsWritten in Python, Ruby (now)Selenese, PHP, Java, C#, Perl (future)

ConfigsWhich browser, addons, & OS thata job should run on

JobsSimilar to a print jobwaiting, running, done…

ResultsOutput of ScriptStdout, stderr, and more…

UsersCan access certain Scripts, Configs, Jobs, and Results

What’s in our cloud?



Waiters (small number)

Listen to API requests

Dispatch Job execution to Chefs

Handle other requests directly

Chefs (many, up to 2000)

Selenium RC + browser

Run Job and return Results

What’s the app under test?

Internet-accessible?

Max # of Selenium RCs?


require 'rubygems'require 'couchrest' # http://github.com/jchris/couchrest

# Link to user's databasedb = CouchRest.database("http://jdoe.saucelabs.com")

# Upload a script# -----------------------------------------------------------------------# A SauceLabs script is simply a regular ruby script wrapped in JSONresponse = db.save( { "Type" => "script", "Name" => "Google Search", "Language" => "ruby", "Content" => "open ('google.com')\ntype ('q', 'selenium')\

nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')" })

script_id = response['id’]


4 Upload script

# Run a job

# ------------------------------------------------------------------

# You run a job by uploading a new "job" document, linking to the

# script id.

# A job is instantly queued for execution upon upload.

response = db.save(

{ "Type" => "job",

"ScriptId" => script_id,

})

job_id = response['id’]


5 Run job

# Check job status# ----------------------------------------------------------------------------status = db.get(job_id)['Status']

# 'status' will be one of:# "new"# "queued"# "in progress"# "complete"# "error”

# Download result# ----------------------------------------------------------------------------# Once the job is complete, a "result" document is created. The "job" document# is updated with a reference to the result.

# First, retrieve the ResultId from the jobresult_id = db.get(job_id)['ResultId']

# Second, retrieve the complete result document:result = db.get(result_id)

# You can now introspect the results# View standard outputstandard_output = result['Stdout']

# View standard errorstandard_error = result['Stderr’]


6 Download result

# Delete result

# ------------------------------------------------------------------

result = db.get(result_id)

db.delete(result)

# Delete script

# ------------------------------------------------------------------

script = db.get(script_id)

db.delete(script)


8 Delete result and script

What we’ll do for you

Provide free access to product

Provide coaching on:

How to integrate our service into your build scripts

How to parallelize your test scripts

Demonstrate prototypes for your feedback

Notify you of downtime and updates (planned or not)

Take precautions to ensure that your information remains secure


What we need from you

Information about your existing testing environment

Perform actual integration of Sauce Labs into your test environment

Use Sauce Labs' product in your testing process

Report any bugs or feature shortcomings you find

Assign a primary point of contact

Participate in weekly check-in calls

Provide a product review (good or bad) that we may publish

Speak with potential Sauce Labs investors about your experiences


Next steps


xUnit, TestNGFit(nesse), RSpec

HudsonContinuumCruiseControlTeamCityBambooGit(hub)SubversionCronEclipse

Internet ExplorerFirefox SafariChromeOperaFlash/FlexiPhoneAndroid

Selenium outputScreencastsScreenshots

DOMshotsJavaScript state

YslowHTML errorsAccessibilityHTTP traffic

Dtrace… all tracked over time

PythonRubyJavaPerl

SeleneseC#

PHPSelenium on Rails

Where do we go from here?


Platforms Languages

Build Systems Telemetry

Frameworks

# CouchDB library from http://code.google.com/p/couchdb-python/

from couchdb import Server

# Link to user's database

server = Server('http://sauceapp.com/')

db = server['jsmith’]

# Upload a script

# ----------------------------------------------------------------------

# A Sauce Labs script is simply a regular python script wrapped in JSON

script_id = db.create({

"Type" : "script",

"Name" : "Google Search",

"Language" : "python",

"Content" : "open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')"

})


4 Upload script

# Run a job

# ----------------------------------------------------------

# You run a job by uploading a new "job" document, linking to the script id.


job_id = db.create(

{ "Type" :"job",

"ScriptId" :script_id,

})


5 Run job

# Check job status# --------------------------------------------------------------------------status = db[job_id]["Status"]

# 'status' will be one of:# "new"# "queued"# "in progress"# "complete"# "error”

# Download result# --------------------------------------------------------------------------# Once the job is complete, a "result" document is created. The "job" document# is updated with a reference to the result.

# First, retrieve the ResultId from the jobresult_id = db[job_id]['ResultId']

# Second, retrieve the complete result documentresult = db[result_id]

# You can now introspect the results.# View standard outputstandard_output = result['Stdout']

# View standard errorstandard_error = result['Stderr']


6 Download result

# Delete result

# -----------------------------------------------------------------

del db[result_id]

# Delete script

# -----------------------------------------------------------------

del db[script_id]



# Using CouchDB library by Michael Hendricks (not in CPAN, yet!)# http://git.ndrix.com/?p=Net-CouchDB use Net::CouchDB; # Link to user's databasemy $server = Net::CouchDB->new('http://localhost:5984');my $db = $server->db('perltest'); # Upload a script# --------------------------------------------------------------------------# A Sauce Labs script is simply a regular perl script wrapped in JSONmy $script = $db->insert( { Type => "script", Name => "Google Search", Langage => "perl", Content => "open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\

nwaitForPageToLoad()\nverifyTextPresent('openqa.org')" }); my $script_id = $script->id;


4 Upload script

# Run a job

# ----------------------------------------------------------

# You run a job by uploading a new "job" document, linking to

# the script id.


my $job = $db->insert(

{ Type => "job",

ScriptId => $script_id

});

my $job_id = $job->id;


5 Run job

# Check job status# --------------------------------------------------------------------------my $status = $db->document($job_id)->{'Status'}; # 'status' will be one of:# "new"# "queued"# "in progress"# "complete"# "error” # Download result# --------------------------------------------------------------------------# Once the job is complete, a "result" document is created. The "job" document# is updated with a reference to the result. # First, retrieve the ResultId from the jobmy $result_id = $db->document($job_id)->{'ResultId'}; # Second, retrieve the complete result document:my $result = $db->document($result_id); # You can now introspect the results.# View standard outputmy $standard_output = result->{'Stdout'}; # View standard errormy $standard_error = result->{'Stderr'};


6 Download result

# Delete result

# ------------------------------------------------------------------

my $result = $db->document($result_id);

$result->delete;

# Delete script

# ------------------------------------------------------------------

my $script = $db->document($script_id);

$script->delete;



Curl command:

curl -X POST -u jdoe:password -d '

{

"Type":"script",

"Name":"Google Search",

"Language":"python",

"Content":"open ('google.com')\ntype ('q', 'selenium')\nclick'btnG')\nwaitForPageToLoad()\nverifyTextPresent('openqa.org')",

"CreatedBy":"jdoe",

"CreatedDate":"Mon, 18 Aug 2008 14:26:10 +0000",

"LastModifiedBy":"jdoe",

"LastModifiedDate":"Mon, 18 Aug 2008 14:26:10 +0000"

}' http://saucedb.com/jdoe/ 04/15/23 Sauce Labs Confidential 32

4 Upload script

Curl returns:

{"ok":true,"id":"4749f6c4159ae5a097e586f3e5040cbe","rev":"668367075"}

To view in browser: (You'll probably need to provide name/password)

http://jdoe.sauceapp.com/4749f6c4159ae5a097e586f3e5040cbe


4 Upload script

Curl command:

curl -X POST -u jdoe:password -d '

{

"Type":"job",

"Status":"new",

"ScriptId":"3ece08113906d84325f9341e794caefa",

"ResultId":null,

"CreatedBy":"jdoe",

"CreatedDate":"Mon, 18 Aug 2008 14:54:55 +0000"

}' http://saucedb.com/jdoe/


5 Run job

Curl returns:

{"ok":true,"id":"3c4c1f2f21ea4e346aebc244f21dde76","rev":"1359430379"}

To view in browser: (You'll probably need to provide name/password)

http://jdoe.sauceapp.com/3c4c1f2f21ea4e346aebc244f21dde76


5 Run job

Partnering with Sauce Labs

• Sauce focuses on hosted testing

• Looking for independent consultants, and consulting firms, who use our service during their engagements

• Guru pricing, training, etc. available


Agile is now mainstream

Source: Dr. Dobb’s Journal. 5/7/08. http://bit.ly/4EjcDH

04/15/23 37Sauce Labs Confidential

http://bit.ly/4EjcDH

Continuous Integration is now mainstream


Source:VersionOne. 8/22/08. http://www.versionone.com/pdf/3rdAnnualStateOfAgile_FullDataReport.pdf

http://www.versionone.com/pdf/3rdAnnualStateOfAgile_FullDataReport.pdf

Functional tests tend to run at end of day


Source: Viewtier Systems. http://bit.ly/2qQ5LX

http://bit.ly/2qQ5LX

Our current thinking about pricing(Subject to change)


$ per test minute

Tests queued no later than…

Five minutes

Express: Relatively high per test minute

8 hours

Economy: Low amount per test minute

Economy

Chef #

minute

Express entails lower chef utilization(We’ll need beta usage to quantify)


Expressscript 1 script 6 script 10

script 2 script 7 script 11 script 12

script 3 script 8 script 13

script 4 script 9 script 14

script 5

10 20 30 40 50 60

1

2

3

4

5

Chef #

10 20 30 40 50 60

script 1 script 2 script 3 script 4 script 5 script 61

minute

Thinking about pricing the right way?

How much Express would you use?

How much Economy?


sauce labs beta program overview

Software