ntc 2016 step right up testing presentation 16x9

Step Right Up! Everyone’s a Winner!

Steve Kerhli, PETA & Steve Daigneault, M+R

Ask String on Main Donation Form

Test Control

Vote: Control Test Tie

Winner: Test

Renewal Language to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Higher Ask String for Prospects

Test Control

Vote: Control Test Tie

Winner: Tie

SMS Deadline Reminder

Test Control

Vote: Control Test Tie

Winner: Test

Nudge to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Font Type and Size

Test Control

Vote: Control Test Tie

Winner: Test

$5 Ask to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Make your tax-deductible gift

of $50 or whatever you can

afford now. Help us meet our

urgent Year-End Challenge goal

of $125,000.

Make your tax-deductible gift

now. Pitch in $5 or more to help

us meet our urgent Year-End

Challenge goal of $125,000.

Adding PayPal as Payment Option

Test Control

Vote: Control Test Tie

Winner: Test

“Renew” Language to Prospects

Test Control

Vote: Control Test Tie

Winner: Test

Another Nudge

Test Control

Vote: Control Test Tie

Winner: Tie

Cents Per Day

Test Control

Vote: Control Test Tie

Winner: Control

Yes, Another Nudge Test

Test Control

Vote: Control Test Tie

Winner: Test

Multi-Button vs. Single Button

Test Control

Vote: Control Test Tie

Winner: Tie

Quick Donate Based on HPC

Test Control

Vote: Control Test Tie

Winner: Tie

EOY Lightbox

Test Control

Vote: Control Test Tie

Winner: Tie

$10 Ask to Lapsed and Deeply Lapsed Donors

Test Control

Vote: Control Test Tie

Winner: Test

“Read More” Button

Test Control

Vote: Control Test Tie

Winner: Tie

Nudge to Prospects + HPC<$125 Donors

Test

Control

Vote: Control Test Tie

Winner: Test

Video on Donation Form

Test Control

Vote: Control Test Tie

Winner: Tie

Reverse Ask String

Test Control

Vote: Control Test Tie

Winner: Test

Button Colour

Test Control

Vote: Control Test Tie

Winner: Test

Email Recruitment Lightbox

Version 2 Version 1

Vote: Version 1 Version 2 Version 3

Winner: Version 2

Version 3

Donate vs. Membership Lightbox at EOY

Test Control

Vote: Control Test Tie

Winner: Test

Donate vs. Membership Lightbox at EOY

Test Control

Vote: Control Test Tie

Winner: Tie

Streamlining Donation Form

Test Control

Vote: Control Test Tie

Winner: Control

OMG, Another Nudge Test

Test Control

Vote: Control Test Tie

Winner: Test

Recommendations for Nudge Tests

• Tested variations of nudge tests to 10 organizations.

• 50% saw improvement in at least one key metric. No organization saw nudges depress response or revenue.

• Recommended language: • “Most people are donating at least $X right

now.”

Upper Case vs. Sentence Case Lightbox

Test Control

Vote: Control Test Tie

Winner: Test

Multistep vs. Single Column Form

Test Control

Vote: Control Test Tie

Winner: Tie

Another PayPal Payment Option

Test Control

Vote: Control Test Tie

Winner: Test

Ask String vs. Open Field

Test

Control

Vote: Control Test Tie

Winner: Test

Right Call-Out vs. Centered

Test Control

Vote: Control Test Tie

Winner: Test

Review vs. Donate Button

Test Control

Vote: Control Test Tie

Winner: Tie

Paragraph Spacing

Test Control

Vote: Control Test Tie

Winner: Control

One Column vs. Two Column

Test Control

Vote: Control Test Tie

Winner: Tie

PETA Tests

$5 Ask to Non-Donors

Test

Control

Vote: Control Test Tie

Winner: Test

$5 Ask vs $10 Ask to Non-Donors

Test Control

Vote: Control Test Tie

Winner: Control

Copy Length on Donation Form

Test Control

Vote: Control Test Tie

Winner: Test

Single Image vs. Multi-Image

Test Control

Vote: Control Test Tie

Winner: Test

Larger Link Size

Test Control

Vote: Control Test Tie

Winner: Test

Animal Specific Donation Forms

Test

Control

Vote: Control Test Tie

Winner: Test

Renewal Language to Current + Lapsed Donors

Test Control

Vote: Control Test Tie

Winner: Test

Membership Buttons on Homepage

Test Control

Vote: Control Test Tie

Winner: Test

Tools for Running a Strong Testing Program

✳ The purpose of hypotheses is to make sure we're thinking critically about what we want to test and why and to make sure we learn from the test.

✳ To help create a hypothesis, fill in the blanks:

Changing (control) to (variation) will improve (test objective) because (reason you think the change will help) .

✳ If you craft a good hypothesis, you will learn something from every test, regardless of the outcome.

Crafting the hypothesis

✳ Before running a test, figure out the sample size needed for the test.

✳ To do that, you’ll need to understand how much of a change you’ll be able to detect.

✳ If you’re evaluating specific audiences or pages (e.g. mobile traffic), do this for each page or segment that you will evaluate

Determine Sample Size needed

Baseline Conversion Rate - the current average for the metric you are trying to improve (e.g. response rate, action rate)

Minimum Detectable Effect (MDE) - the amount of change in the conversion rate that you want to be able to detect. We recommend 25% or lower.

Elements of the sample size calculator

Minimum Audience Size - the total number of people you need to include in the experiment, per variation.

Sample Testing Roadmap

Sample Test Plan

Sample Results Tracker

All In One Place