draft: opinion space pilot study

DRAFT:Opinion Space Pilot Study

Topic: Reputation of the American Automotive Industry

Participants: Automotive Industry Online Community

Research Period: Fall 2010 – Spring 2011

May 2011Rick Wagner, Joyce Salisbury, Ken Goldberg, David WongUC Berkeley Logo

Pilot Study Goals1) Sample Existing Perceptions of the

Image of the American Automotive Industry

2) Collect Ideas about How to Improve the Image of the American Automotive Industry and Identify the Most Insightful Ideas

3) Study how Opinion Space compares with Existing Survey Methods in Terms of Participant Engagement

4) Consider Demographic Variations among Participants

5) Consider Applications for gm.com

Executive Summary1) Sample Size (# Participants):

1198 2) Avg Number of Minutes Spent on Site:

363) Avg Number of Peer-to-Peer Evals: 904) % Who are “Extremely Likely”

to Participate Again: 95

5) Average Age of Least Active Participants: 44

6) Average Age of Most Active Participants: 50 (!)

7) Key Issues:8) 20 Most Insightful Ideas

Executive SummaryDiscussion Question: Improving Industry

Image1) Integrity: Honesty, Responsiveness to

Flaws2) Value: Quality, Price, Safety, Warranty3) Only 2 of top 20 mention “bailout”4) Listen more to consumers5) Don’t over-power vehicles6) Fewer “Gee-Whiz” features7) Become leaders in innovation8) Don’t charge too much for GPS9) LED Headlights10)Develop Autos that run on Natural Gas!

Executive Summary• Opinion Space was very effective in engaging AIOC participants

– Average total time spent on the system: 36 minutes.– Average number of ratings per user: 90 insight and agreement ratings.– Much excitement expressed to participate again (see figure).

• Top Insights– AIOC members suggested many things GMis already working on.

• Next steps– Host another Opinion Space on GM.com toengage the public in generating ideas.– Reengage the AIOC to better communicate what GM is working on and solicit ideas on how toimprove.

Q150. How willing would you be to participate in other versions of this

system (on related topics)?

Not likely

Extremely likely

Opinion Space

Our Approach

1. Visualization 2. Level Playing Field

3. Wisdom of the Crowds 4. Game Structure

"Opinion Space will harness the power of connection technologies to provide a unique forum for international dialogue. This is...an opportunity to extend our engagement beyond the halls of government directly to the people of the world.”

- U.S. Secretary of State Hillary Clinton

“The world spends over 110 billion minutes per month on social networks and blog sites.”

- NielsenWire, June 2010

“We’re moving from an Information Age to an Opinion Age.” - Warren Sack, UCSC

20 sec. per comment X 35,387 comments

= 8 days

goalsFor Organizations• Understand the diversity of their

community• Engage their community

– Solicit feedback and creative suggestions

– Rapidly identify patterns, insightful ideas

For Community Members• Understand relationships with other community members• Engage with a diversity of viewpoints

and ideas• Express ideas, and be heard hybrid vigor

opinion.berkeley.edu

Step 1: Enter your opinions and response

Step 2: Visualize your position

Step 3: Read and rate others users

System Design (Aug-Dec 2010)

Baseline Questions:

• When deciding on which major product to buy, I often consider where it was made in making my decision.

• Reducing oil consumption should be a higher priority in the U.S.

• I am willing to pay more for a product that comes from a socially responsible corporation.

• The U.S. government should not give loans to corporations.

• New technologies are causing more problems than they are solving.

Discussion Question:What is the Most Important Step or Steps that US

Automotive Manufacturers could take to Improve Their Image?

System Stats• Duration 20 Dec 2010 – 21 Jan 2011• Announcement, 2 Reminders

• 1198 participants (57% response rate)• 1,148 Responses • 96,000 Insight Ratings• 96,000 Agreement Ratings

Opinion Space

Users and sessions per day

Unique visits per

day

Active users per

day

Session duration

Zoomed out

Zoomed in

Ratings per day

Distribution of ratingsAll responses are rated on two scales: 1) How much do

you agree with the response?

2) How insightful is this response?

Ratings and views per comment

Insight ratings per user

Zoomed out

Zoomed in

Agreement ratings per user

Zoomed out

Zoomed in

Statement Statistics1) When deciding on which major

product to buy, I often consider where it was made in making my decision.

2) Reducing oil consumption should be a higher priority in the U.S.

3) I am willing to pay more for a product that comes from a socially responsible corporation.

4) The U.S. government should not give loans to corporations.

5) New technologies are causing more problems than they are solving.

Statement 1: When deciding on which major product to buy, I often consider where it was made in making my

decision.

Statement 2: Reducing oil consumption should be a higher priority in the U.S.

Statement 3: I am willing to pay more for a product that comes from a socially responsible corporation.

Statement 4: The U.S. government should not give loans to corporations.

Statement 5: New technologies are causing more problems than they are solving.

Statements in relation to position in Opinion Space

For each statement, a participant’s point will move in the direction and angle indicated by the arrow as they drag the slider.

Participant Responses• Participant responses were ranked using the

insightfulness ratings from other participants. • Two main algorithms were used to rank responses.

– Spatial ranking: ratings of a response are weighted by the spatial distance between the author of the response and the rater.

– Confidence interval ranking: responses are ranked by the lower

bound of a 95% confidence interval of the mean.

• See Appendix for a verbatim list of the top 10 responses

Topics and Insights from the Top 18 Participant Responses

Topic: US Automotive Corporations

- Achieve more integrity- Streamline production to reduce costs- Renegotiate union contracts to keep

prices low- Find out what the customer really

wants- Engage in product development with

the goal of long term use- Market value to consumers using

measurable and verifiable methods- Pay back government loans- Lessen environmental impact

Topic: Price

- Remove extraneous, meaningless features that increase the price and don’t add significant value.- GPS cited as an example*

- Keep prices low and competitive with foreign companies

- Affordability is the bottom line for most people

*participant1329’s response (rank #1): “When offering new technology, they must keep the price unchanged or slightly higher. Newer technology is often exponentially more than its value. For example, remember the GPS? Automakers were charging $1500-2500 for an installed GPS when you could buy one for $300. This makes the automaker look greedy…”

Topics and Insights from the Top 18 Participant Responses

Topic: What participants want in quality and value

- Durable, reliable, dependable, well-built, no manufacturing defects, built with stronger and lighter materials- Safety - Fuel-saving, fuel-efficient, and good

fuel-economy and mileage- Increased warranties that reflect

higher quality cars- Performance- Comfortable for larger families

Topic: Electric cars

- Stop building electric cars and focus on increasing gas mileage

- Make electric cars with higher performance that can go longer distances

- Make cars large enough to fit larger families*

*participant2053’s response (rank #5): “…Make vehicles that contain fuel efficient technologies in more then just the small car genre. We are a family of 5 and need to transport a wheelchair for our middle son. Most super economical cars wont carry us as a family. I like the idea of electric vehicles but would need to see them able to go greater distances before needing to be charged…”

Top terms from participant responses

Top co-occurring terms from participant responses

Post Survey Analysis

Q105. What was your overall opinion of the Opinion Space site?

Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the

Opinion Space activity?

Q130_1. I was motivated to consider responses from people far from my own response on the opinion map

Q130_2. The size of each point was a reasonable measure of how insightful the corresponding response was

Q135. How do you feel about using two sliders (agreement, insightfulness) to rate responses?

Q140_1. I would have liked to comment on other participants' responses

Q140_2. I would have liked to have been notified when my response was rated by other participants

Q150. How willing would you be to participate in other versions of this system (on related topics)?

Q110. What aspects of the Opinion Space interface did you like?

Main Ideas: - Ability to explore others’ ideas and see

how similar or different you are from others

- Learning a lot from reading the opinions of others; opened up from previous way of thinking

- The ability to rate others opinions- Allowed ongoing participation over

time so participants could come back- Sense of being heard - Interesting change of pace; fun, fast,

different, unique, and easy to use

- Enjoyed the combination of statements and free form answer box

- Enjoyed the anonymity- The UI is very visually pleasing,

uncluttered, and intuitive - The ability to see who has rated you- Topics were of high interest - The star metaphor with larger points

meaning more insightful idea- Engaging and made excited to rate

the opinions of others

“I really enjoyed being able to read and rate other participants' responses. I found that there were many with which I didn't necessarialy agree with but were well thought out and concisely argued. That gave me the chance to rate highly an opinion other than my own and also opened my eyes to other sides of "the argument" so to speak.”

Q110. What aspects of the Opinion Space interface did you like?

“I liked the way we could change screens to get to the ones we didn't rate yet, instead of it all being on one page in some kind of a continuous boring list…”

“I loved reading other members ideas and/or opinions - some were very interesting and I even learned something from some! Other ideas/opinions were unique and things that I never thought about.”

“…the idea of an incentive and/or prize (not to mention just the morale-boost of having others treat my own opinions with respect and interest) was itself an incentive to read as many entries as I could. It was a contest of sorts, but a most supportive and kind one, not the cutthroat, nasty kind that I try to steer clear of. I was so sorry when it ended!...”

Q115. What aspects of the Opinion Space interface could be improved (and how)?

Main Ideas: - Include a tutorial, more instructions,

and/or a demo video on how to use the system and interface

- See more details on each participant- Allow commenting on your response- Ability to see who has rated you and

get feedback on your ideas- Ability to create categories and

clusters- Want a dedicated page to visit as

opposed to finding link through email- Keep the system online longer- Add a spell checker- Include background info on

discussion topics- More real time information and

interaction with other participants- Want a way to view the higher level

trends in the system

- Use a mouse-over as opposed to clicking to view a participant’s response

- Organization of who you’ve rated and who rated you; a list here makes sense

- Have system give suggestions of whose ideas are similar/different than your own

- Add additional questions to make conversation more in-depth

- Have more reminder emails with notifications of the status of the system (e.g. new responses to be rated).

- Increase size of the stars- Increased features for navigation

(see all stars, zoom in and out).- Rewards for everyone who

participates- Larger viewing box for long

responses

Q145. What new features would you like to see in Opinion Space?

Subgroup Analysis of Participation

100 Least Active vs. 100 Most Active Reviewers

Age of 100 least active reviewers

Mean Median Stdev

43.48 40.5 13.29772913


Age

Age of 100 most active reviewers

Mean Median Stdev

49.76 51 11.17149945


Gender

Gender of 100 least active reviewers

Male Female Ratio (M:F)

49 51 0.96:1

Gender of 100 most active reviewers

Male Female Ratio

56 44 1.27:1

100 Least Active vs. 100 Most Active ReviewersEducation

Education of 100 least active reviewers Education of 100 most active reviewers

Median MaxCollege Graduate (4

years)College Graduate (4

years)



years)


Income

Income of 100 least active reviewers

Median

$75,000 to $99,999

Income of 100 most active reviewers

Median

$75,000 to $99,999


Region

Region of 100 least active reviewers Region of 100 most active reviewers

100 Most Active vs. 50 Most Active Reviewers


100 Most Active vs. 50 Most Active ReviewersAge


Mean Median Stdev

49.76 51 11.17149945


Mean Median Stdev

50.88 53 10.69885975

100 Most Active vs. 50 Most Active ReviewersGender


Male Female Ratio

56 44 1.27:1


Male Female Ratio

30 20 1.5:1

100 Most Active vs. 50 Most Active ReviewersEducation

Education of 100 most active reviewers




years)



years)

100 Most Active vs. 50 Most Active ReviewersIncome


Median

$75,000 to $99,999


Median

$75,000 to $99,999


Region

Region of 100 most active reviewers Region of 50 most active reviewers

50 Most Active Reviewers vs. 50 Top Authors


50 Most Active Reviewers vs. 50 Top AuthorsAge


Mean Median Stdev

50.88 53 10.69885975

Age of 50 top authors

Mean Median Stdev

47.7 49 9.428149341


Male Female Ratio

30 20 1.5:1

50 Most Active Reviewers vs. 50 Top AuthorsGender

Gender of 50 top authors

Male Female Ratio

28 22 1.27:1


50 Most Active Reviewers vs. 50 Top AuthorsEducation

Education of 50 top authors



years)


years) Postgraduate College


Median

$75,000 to $99,999

50 Most Active Reviewers vs. 50 Top AuthorsIncome

Income of 50 top authors

Median

$75,000 to $99,999

Region of 50 most active reviewers

50 Most Active Reviewers vs. 50 Top AuthorsRegion

Region of 50 top authors

Subgroup Analysis of ParticipationPost Survey Results

*red text indicates statistically significant differences

100 Least Active Reviewers100 Most Active Reviewers50 Most Active Reviewers

50 Top Authors

Aggregate Results

Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space

activity?


50 Top Authors

Aggregate Results


Q130(1). I was motivated to consider responses from people far from my own response on the opinion map


50 Top Authors

Aggregate Results

Q130 (2) The size of each point was a reasonable measure of how insightful the corresponding response was


50 Top Authors

Aggregate Results





50 Top Authors

Aggregate Results



Of all statistically significant differences (95% confidence interval)

Subgroup Analysis of Statement Ratings

Subgroup Analysis of Statement Ratings(statistically significant differences)

Statement 1: When deciding on which major product to buy, I often consider where it was made in making my decision.

Education = PostgraduateIncome = $125k - $150k

Aggregate Results

Statement 2: Reducing oil consumption should be a higher priority in the US

Income = $15k - $25kRegion = Northeast

Aggregate Results

Region = West

Statement 3: I am willing to pay more for a product that comes from a socially responsible corporation.

Age = 30 - 39Age = 40 - 49

Aggregate Results

Education = College GraduateGender = MaleRegion = West

YAMYEF


Statement 4: The U.S. government should not give loans to corporations.

Education = High SchoolEducation = Some College

Aggregate Results

YAM

Income = $25k - $35kIncome = $35k - $50k


Statement 5: New technologies are causing more problems than they are solving.

Aggregate ResultsAge = 18 - 29Age = 50 - 59

Education = High SchoolGender = Female

Gender = Male




Subgroup Analysis of Post Survey Ratings

Of all statistically significant differences (95% confidence interval)

Subgroup Analysis of Post Survey Ratings(statistically significant differences)


Aggregate ResultsEducation = Postgraduate

Education = Trade/Vocational SchoolIncome = $100k - $125k

Region = NortheastYAM


Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space

activity?Aggregate Results

Age = 50 - 59Education = Postgraduate

Region = West

Q130(1). I was motivated to consider responses from people far from my own response on the opinion mapAggregate Results

Age = 60 - 70Education = College Graduate


Q130 (2) The size of each point was a reasonable measure of how insightful the corresponding response was

Aggregate ResultsAge = 30 - 39Age = 60 – 70

YAM

Q135. How do you feel about using two sliders (agreement, insightfulness) to rate responses?

Aggregate Results

Age = 60 - 70Age = 30 - 39



Aggregate ResultsGender = Female

Gender = Male

Income = $100k - $125k


Summary of Subgroup Findings

Young Affluent Males (YAM)

Young Affluent Males (YAM)

Aggregate rating statistics

Subgroup rating statistics Statements:

1) When deciding on which major product to buy, I often consider where it was made in making my decision.





Young Educated Females (YEF)

Aggregate rating statistics

Subgroup rating statistics Statements:

1) When deciding on which major product to buy, I often consider where it was made in making my decision.





Young Educated Females (YEF)

Incentive Analysis

Subgroup Response to Q120:“How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space activity?”

Subgroup Q120 Mean Q120 StdevTop 100 Reviewers 3.450549451 1.206791553

Top 50 Reviewers 3.369565217 1.340971349

income-$125,000 to $149,999 3.038461538 1.181506076

age-18-29 3.012658228 1.237531955

Top 50 Authors 3 1.268761639

YEF 2.948905109 1.234073238

YAM 2.939393939 1.179095543

age-30-39 2.933014354 1.21604544

income-$50,000 to $74,999 2.909090909 1.231660693

region-Northeast 2.894736842 1.263556556

100 least active reviewers 2.626666667 1.230429013

…

Measurements of Participant Activity

Mean Stdev

Insight ratings per user 45.56 142.52

Agreement ratings per user 46.03 142.57

Session Length (min) 12.69 11.25

Average Session Count 3.00 5.89

User Events 46.46

Total comments rated 105,861.00

Total comments viewed 118,737.00

Total Sessions 3,147.00Comments viewed per

session 37.73

Insight ratings per viewed comment 0.81

Agreement ratings per viewed comment 0.82

AIOC Opinion Space – All Users

Mean Stdev





User Events 44.68




session 35.73



AIOC Opinion Space – Rogue Raters Removed


Mean Stdev





User Events 15.21




session 11.58



Unilever Opinion Space – 1939 Total users

Mean Stdev





User Events 43.99



Total Sessions 390.00Comments viewed per

session 25.34



Opinion Space 3.0 (state.gov/opinionspace) – 370 Total users


Mean Stdev





Opinion Space 2.0 previously at state.gov/opinionspace – 5711 Total users

Rogue Rater Analysis

Summary

Appendix

Google Moderator

draft: opinion space pilot study

Documents