testing after you’ve finished testing
TRANSCRIPT
Testing… (After You’ve Finished Testing)
Jon BachQE Director, Live Site [email protected]
STAR Canada 2013
2
These are slides for the Conference CD
This deck is written to meet a deadline for the conference CD on January 25th.
If you’re reading this slide, please know that the content is likely out-of-date because I change content before, after (and even DURING) my presentations.
If you want the latest-and-greatest slides, go to my public Dropbox share:
https://www.dropbox.com/sh/bg1r6qz3wqhcil3/-R_hgcinfR
3
What I do
4
10 Testing-related activities after release
• Experimentation• Monitoring• Investigative• Surveys• Scenario• Reputation• Follow-up• Regression• Events• Competitive
5
Experimentation (A/B testing)
A
B
6
As
Monitoring
7
As
Monitoring
8
This is a dip in bids we saw in Production.
What questions do you have to troubleshoot this?
Investigative Testing
9
Topics for investigation (FORCOSTTTIP)
• Feature – Do we know what happens when people bid?• Origin -- When did this start happening?• Recent – Did we just push code to Production?• Consistency / Correlation -- What happened yesterday,
tomorrow? Are there other graphs and metrics?• Overlap – Is there any other process happening?• Security – Is it malicious? Intentional?• Terminology -- What kinds of “bids”? US, UK, International?• Third-Party / Silent Partner – What’s not in the room? What sort
of information have we NOT heard from?• Truth is out there – does someone already know about this?• Instrumentation – is the graph wrong? Can I get more data?• Prediction – can we test a conjecture with a prediction
10
When site visitors click this…
Surveys
11
…we see this…
We see this…
Surveys
12
Scenario
13
Reputation
14
I go on eBay Radio (a weekly podcast for customers) as “The Bug Hunter” to elicit stories about technical problems in Production.
-------------------------------------------------------------------------------
“I have attached the screen shot for you. Try as I might, I can't get
the selling reminders to go to the top. The arrows don't work, but I
can move sold, unsold and active selling. I even set the default to summary, but no luck. FYI- I am using Safari on my trusty Mac.
Also, the default font size is too big. When I zoom it out, it's too small. It was a better size the way it was before. I sent a message
to the team through the 'what do you think' for that page.”
Follow-up
15
Video of an automated test:
Regression
16
A large seller on eBay wants to have a “Flash Sale”.
Are we ready?
Events
17
Competitive Testing
18
10 Testing-related activities after release
• Experimentation• Monitoring• Investigative• Surveys• Scenario• Reputation• Follow-up• Regression• Events• Competitive
Questions?