basic research shapiro 10_2011
TRANSCRIPT
![Page 1: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/1.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 1/39
Founder Institute: Basic Research
Dan Shapiro
Ontela
![Page 2: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/2.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 2/39
Three projects
• Researching your market with Mturk and
SurveyMonkey
• Testing your company with Google AdWords
• Picking your pitch with Excel
![Page 3: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/3.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 3/39
Mturk & Surveymonkey
![Page 4: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/4.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 4/39
Surveymonkey
• Cheap tool for building surveys
• You’ll probably need the “pro” subscription
• Zoomerang is even better (but more
expensive)
The problem: you need people to actually take
the survey
![Page 5: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/5.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 5/39
mTurk
• Amazon Web Services offering
– Makes it easy to recruit people to do small tasksfor very little money
– Designed to replace AI
– Fund with a regular old Amazon account
– Try it in bits and pieces
•
Perhaps the worst named product, ever – OK, maybe “expertsexchange.com” is worse
![Page 6: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/6.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 6/39
What you need to do
• Pay turkers to take surveymonkey surveys
![Page 7: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/7.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 7/39
Example surveys
• Demographics, “state of the industry”
• Sales literature fodder
• Usability test
• Naming
• Competitive/landscape research
• Pricing
• Survey your customers (if you’re big)
![Page 8: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/8.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 8/39
But is it “valid”?
• Pretty much
• Turkers are reasonably representative of the internet-using public
• Studied pretty thoroughly in academia
•
Empirically our results match internal quantitativeresearch of our customers
• Do you know how shoddy most of the research outthere is? It’s great by comparison
•Recommended reading:
http://bit.ly/7ZhCBs
![Page 9: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/9.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 9/39
ENTICING PEOPLE TO DO IT
Step 1
![Page 10: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/10.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 10/39
Price it and sell it
• Sell it in the HIT: “Just take a minute!” “Fun survey!” “Getpaid to help a startup!” – Be sure to read the T&Cs about what you can & can’t do
• Price it right – 11 cents is a good starting point
– 21 cents got us 786 responses to a 10 page survey in two weeks – $1.01 really gets people’s attention
– The price controls the rate at which people respond• Start low/slow and increase as needed
• NB: A new HIT means people become re-eligible –
So you can’t combine results originating from different HITs – You can add people to an existing HIT, though, if you want more
data
![Page 11: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/11.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 11/39
GET THE RIGHT PEOPLE
Step 2
![Page 12: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/12.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 12/39
The secret: throw what you don’t want
• Do not ask, “Are you an asian-american medical doctor licensed topractice in Ohio?” – Ummm... Yes?
– Many turkers will answer in a way that gets them paid
• Collect data about the respondents, then ignore (and pay anyway)those you don’t want – Seems wasteful, but they’re cheap
– Does make this expensive for gathering data on niches
• Some work arounds exist – Gather real contact info in survey one; real compensation comes when
you re-contact them for survey two• Violates TOS
– Use mTurk built-in filtering capaibilities• I have no idea how these work
![Page 13: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/13.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 13/39
CONNECTING MTURK AND
SURVEYMONKEY
Step 2
![Page 14: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/14.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 14/39
Not too hard
• Build some sort of API fancitude
– I don’t know how to do this
• Put a “code” at the end of the survey
• The HIT is: “Go here, take the survey. It willgive you a password. Enter the password
here:______”
•
Use very explicit instructions – Open a tab, enter this URL... etc
![Page 15: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/15.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 15/39
BUILD A GOOD SURVEY
Step 4
![Page 16: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/16.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 16/39
“Ask” versus “Tell”
• In “Ask” surveys, the goal is to learn
• In “Tell” surveys, the goal is to create good
marketing material
• You can ask first and tell second in the samesurvey, but not vice versa
Don’t confuse them!!!(i.e. don’t believe your own hype)
![Page 17: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/17.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 17/39
Basics
• Ask anything you might ever possibly want
• Get demographic data
– Suggest mirroring the US Census questions
– Lets you slice by demographics
– Lets you compare your results to US averages
• Make almost every question required
–
Or else people can just skip through to the end
![Page 18: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/18.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 18/39
Be careful
Things to avoid in “ask” surveys:
• Leading questions
• Order of questions
• Identifying what you’re interested in
•
Telling them who you are – Consider this when creating your account names!
• Creating unrealistic situations – E.g. making them read a page of info when a real customer will
have only a one-line description
Ref: “How to lie with statistics”(Amazon: http://bit.ly/7aot27)
![Page 19: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/19.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 19/39
Advanced techniques you should use
• Skip logic
– Annoying but not hard
• Randomized surveys
– Hack combines skip logic with “randomize order of responses”
• Sneaky validity double checks
– Ask the same question a different way
– Leading questions that lead in the wrong direction
![Page 20: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/20.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 20/39
ITERATE
Step 5
![Page 21: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/21.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 21/39
Do, retry, do again
• Start out with 10 tasks
• Review the results
• Fix the survey
• Repeat• Try again with 100
• Etc.
You will discover and fix all sorts of weird stuff
![Page 22: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/22.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 22/39
RUN THE FULL SURVEY
Step 6
![Page 23: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/23.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 23/39
Launch the big one
• 1000 respondents gives
you 3% MOE
• Consider subsample
size – if you want to
report on women only,
for example
• Do-overs are not
unaffordable
1,000 * $0.21 = $210
![Page 24: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/24.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 24/39
Mind your feedback
• Pay daily
• Watch for user comments; they often find
broken things
– But also are often just confused
• If you get banned, follow the directions to fix
it
– And read the terms more carefully next time!
![Page 25: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/25.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 25/39
ANALYZE THE RESULTS
Step 7
![Page 26: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/26.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 26/39
Play with the filters
• Exclude unqualified people
• Look at interesting subpopulations
• Check out the boolean combinations
especially
![Page 27: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/27.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 27/39
Write up the results
• Parrot the style of analysts
• Use lots of pretty graphs
• Combine related stuff (like multiple questions
in one graph)
• Use filters to expose information
![Page 28: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/28.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 28/39
Dan’s todo
• Import in to a database
• Normalize for demographics
• API integration
• Zoomerang
• mTurk qualifications
![Page 29: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/29.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 29/39
Google adwords
![Page 30: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/30.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 30/39
What’s cool about adwords
• You only pay for success
• You can use it a la carte
• It tests in a crucial, real-world environment
– Assuming you’re going to actually use adwords or
similar to promote your service
• Doing well on adwords lowers your cost of
acquisition
![Page 31: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/31.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 31/39
What to use adwords for
• Measuring zeitgeist – Use the tools to see what people are searching for
• Your pitch – Use a placeholder name
– Vary the ad – your “pitch”• Your name/URL
– Once you’ve fixed a pitch, try different names and/orURLs
• Driving traffic to your site to measure stuff – Landing page effectiveness etc.
![Page 32: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/32.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 32/39
Considerations
• Your eyeballs are biased
– Your keywords determine who you’re surveying
• Expensive for real customer acquisition
– If a customer is worth $1, forget it
• Slow
• They’re prone to turning your ads off
![Page 33: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/33.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 33/39
PICKING YOUR PITCH WITH EXCEL
![Page 34: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/34.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 34/39
Step 1
• Brainstorm all the characteristics that are
important
– How pitch performs on Google Adwords
– Maximum revenue earnable
– Time to first revenue
– ...
![Page 35: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/35.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 35/39
Step 2
• Weight each factor from Step 1 on a scale of
1-10
![Page 36: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/36.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 36/39
Step 3
• List every business concept you can think of
![Page 37: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/37.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 37/39
Step 4
• Assign a score for each characteristic
![Page 38: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/38.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 38/39
Step 5
• Math time! Multiply each score times weight,
then sort.
Tada! You have your new company.
(not really, but it’s a good start)
![Page 39: Basic Research Shapiro 10_2011](https://reader033.vdocuments.site/reader033/viewer/2022052515/577d38481a28ab3a6b977a13/html5/thumbnails/39.jpg)
8/9/2019 Basic Research Shapiro 10_2011
http://slidepdf.com/reader/full/basic-research-shapiro-102011 39/39
email me – any time