how we test at ebay europe

How we test at eBay

Michael Palotas Head of Quality Engineering Europe Zürich, 17 August 2011

Produced by: Michael Palotas

The Company

2


eBay Inc.

3


Setup

4

EU

Quality Engineering

(Zürich, London,

Berlin, Paris)

Global Product

San Jose

USA

Global Product

Shanghai

China

Global Product

Chennai

India

Local Product

Zürich, London

Mobile

Portland USA

Adjacencies (i.e. Brands for Friends)


Not only the Web

5

Web

Windows Desktop

Mobile


Landscape

eBay Inc 6

90.000 development days / year

Continuous Integration not consistent

1000+ new bugs found every week

Teams all around the globe in all possible time zones

Major pieces outsourced to India + China

300 test engineers around the globe

Unstable test environment


Waterfall...SCRUM…and everything in between

7


Testing at eBay in the past

8

Very manual

Manual regression testing for 22 sites, every 2 weeks

Teams did not rely on automation

Skill set of QA Engineers mainly non-technical

Sources: www.linesofflight.net www.generallyAwesome.com


Testing Today – Ownership of Quality

9

Manual Test

Automation

Consulting +

Coaching


Manual Test

Still very important, mainly exploratory type

All new features are tested manually first

Edge cases

Not automatable tests

Manual testing still yields the highest number of bugs

10


Automation

Automation is not about eliminating jobs !!!

Save time and reinvest into manual testing

Regression Testing (fully) automated

Programmatic approach (JAVA)

Open source only (WebDriver/Selenium 2 GRID)

3500 end to end testcases running in less than 1 hour 11


GRID

12

Para

llel

exec

utio

ns

Test 1

Test 2

Test 3

Test …

Test …

Test3500

Execution Time

Sequential Execution

Execution time 7000 minutes (~116 hours = ~5 days)

eBay Europe: -  3500 e2e regression testcases -  2-3 minutes execution time (each)

Test

Test

Test

Test

Test

Test

Execution Time

Para

llel e

xecu

tions

Parallel execution in Grid

Test

Test

Test

Test

Test

Test

Test

Test

Test 150 Virtual Machines / Browsers in

the WebDriver GRID Execution time: 60 minutes


Coaching / Consulting

EU team conducts project audits

Improvement initiatives

Consulting for dev and test teams

13

Produced by: Michael Palotas 14

Software Measurement - SONAR


Project Audits

15


EU Quality Engineering Team – Automation Impact

16

.00% 2.00% 4.00% 6.00% 8.00%

10.00% 12.00% 14.00% 16.00% 18.00% 20.00%

1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13

Week

Team % of global Bugs 2009 Start of automation efforts in early 2009

"   Team makes up for only 2% of global test engineer headcount

"   Clear correlation between global bug contribution and automation investment / coverage

headcount share

headcount share

Contribution Trendline

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

18.0%

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

Week

Team % of global Bugs 2010

headcount share


Bug Quality

17

Bug Quality = real bugs / submitted bugs

75

80

85

90

95

100

Q1 08 Q2 08 Q3 08 Q4 08 Q1 09 Q2 09 Q3 09 Q4 09 Q1 10 Q2 10 Q3 10 Q4 10

Bug Quality in % (2008-2010)

Bug Quality 2008-2010

Start of automation efforts


eBay checkbox – vs. HTML checkbox

18

<div align="center”> <input type="checkbox" name=”myName" value=”eBay"> eBay<br> </div>


Summary

Testing is changing

Great career opportunities in testing

More technical skill sets required

Testautomation can have a big impact

….

or make your life hell

Essential to work with developers

Measure and improve

19


C o n t a c t : m p a l o t a s @ e b a y . c o m

THANK YOU! QUESTIONS?

20

how we test at ebay europe

Business