iir october 19, 2009 - mktg, .iir october 19, 2009 “unusual survey results have caused ... m102

Download IIR October 19, 2009 - Mktg, .IIR October 19, 2009 “Unusual Survey Results Have Caused ... M102

Post on 18-Aug-2018

212 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • TheStateofOnlineResearchPanels

    IIROctober19,2009

  • UnusualSurveyResultsHaveCausedAlarm

    RonGailey,workingforWAMU,presented(attheIIR,November2008)theresultsof29researchstudies,atotalof40,000interviews,conductedfrom2006to2007.

    Problem:demandforfinancialproductsdecreased overtime;aphenomenonnotsupportedbyexperienceinthemarket.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • InconsistencyinhissampleleftRonstrugglingtounderstandtheresultsoftwoyearsoftrackingwork.

    Businessdecisionswereclearlyatrisk.

    TherewasnoexternalmeasureofconsistencyforRontoreferto.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Ineverystudyexamined

    peoplewithmorepaneltenuregavelowerdemand.

    RonGailey,2008

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Paneltenureisprogressive;itchangesthroughtime.

    Itrepresentsaninconsistency.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • CopyrightMktg,Inc.2009.Allrightsreserved.

    0%10%20%30%40%50%60%70%80%90%

    100%M25

    Russia

    M79

    Russia

    M10

    0Ru

    ssia

    M10

    9Ru

    ssia

    M50

    China

    M99

    China

    M12

    3Ch

    ina

    M24

    Japan

    M10

    4Japan

    M12

    5Japan

    M44

    S.Korea

    M11

    1S.Ko

    rea

    M12

    6S.Korea

    M43

    Australia

    M12

    2Au

    stralia

    M47

    Hon

    gKo

    ng

    M12

    4Ho

    ngKon

    g

    M48

    Taiwan

    M52

    Thailand

    M12

    1New

    Zealand

    M67

    US

    M76

    US

    M95

    US

    M96

    US

    M10

    1U

    S

    M12

    7U

    S

    M12

    9US

    AGING

    Aging0Months Aging6months Aging12months Aging18months Aging24months

  • AGlobalEffort

    Webeganwith17AmericanPanelsasapreliminarysample.Mktg,Inc.releasedtheanalysisinJanuary2009,atCASRO.

    Wearenowcollectingdatain35countriesand140panels.

    WecalltheprojecttheGrandMean

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • TheGrandMeanProject

    TheGrandMeanistheaveragevalueofavailablepaneldataforacountryorregion.

    Forexample,aGrandMeancanbemaintainedforrespondenttenuremeasuredacrosspanelswithinacountry.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Sowebegan Selecteddemographicquotas(age,income,gender,ethnicity)wereusedtosimulatecensus.

    Medianlengthwas15minutes.

    Questionscovered:Technologyandthemedia,Participationinmarketresearch,Buyer Behavior,ValuesandLifestyle,Demographics,QuestionnaireSatisfaction.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • BUYINGBEHAVIOR..

    .itsatthecoreofMarketResearch.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • CopyrightMktg,Inc.2009.Allrightsreserved.

    0%

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%

    BuyerBehavior

    BuyerBehaviorSegmentsDomestic/Coupons BuyerBehaviorSegmentsCredit/Environment

    BuyerBehaviorSegmentsPriceSensitiveShoppers BuyerBehaviorSegmentsBroadRange/Credit

  • CopyrightMktg,Inc.2009.Allrightsreserved.

    0%

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%M83

    UK

    M85

    UK

    M68

    Spain

    M73

    Spain

    M69

    Germany

    M74

    Germany

    M86

    Germany

    M75

    France

    M80

    France

    M87

    France

    M98

    Brazil

    M10

    5Brazil

    M10

    7Swed

    en

    M11

    4Swed

    en

    M99

    China

    M12

    3Ch

    ina

    M10

    4Japan

    M12

    5Japan

    M12

    8Japan

    M11

    1S.korea

    M12

    6S.korea

    M12

    2Au

    stralia

    M12

    4Ho

    ngKon

    g

    M79

    Russia

    M10

    0Ru

    ssia

    M13

    3Ru

    ssia

    M10

    9Ru

    ssia

    M67

    US

    M76

    US

    M95

    US

    M96

    US

    M10

    1U

    S

    M12

    7U

    S

    M12

    9U

    S

    Automotive BuyerBehavior

    AutomotiveReplacement AutomotiveBrandoverPrice AutomotiveQuality/Shopping

  • CopyrightMktg,Inc.2009.Allrightsreserved.

    0%

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%M83

    UK

    M85

    UK

    M68

    Spain

    M73

    Spain

    M69

    Germ

    M74

    Germ

    M86

    Germ

    M75

    France

    M80

    France

    M87

    France

    M71

    Ita

    lyM84

    Italy

    M89

    Ita

    lyM99

    China

    M12

    3Ch

    ina

    M10

    4Japan

    M12

    5Japan

    M12

    8Japan

    M11

    1Skorea

    M12

    6Skorea

    M12

    2Au

    stralia

    M12

    4Ho

    ngKon

    gM79

    Russia

    M13

    3Ru

    ssia

    M10

    0Ru

    ssia

    M10

    9Ru

    ssia

    M10

    7Swed

    enM11

    4Swed

    enM67

    US

    M76

    US

    M95

    US

    M96

    US

    M10

    1U

    SM12

    7U

    SM12

    9U

    S

    ClothingBuyingBehavior

    ClothingNewClothes/Online ClothingCredit ClothingBrands/DecisionMaker

  • 0%

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%M83

    UK

    M85

    UK

    M68

    Spain

    M73

    Spain

    M69

    Germany

    M74

    Germany

    M86

    Germany

    M75

    France

    M80

    France

    M87

    France

    M72

    Canada

    M97

    Canada

    M10

    2Canada

    M11

    0Canada

    M10

    7Sw

    eden

    M11

    4Sw

    eden

    M99

    China

    M12

    3Ch

    ina

    M10

    4Japan

    M12

    5Japan

    M11

    1Skorea

    M12

    6Skorea

    M12

    2Au

    stralia

    M12

    4Ho

    ngKon

    gM79

    Russia

    M10

    0Ru

    ssia

    M10

    9Ru

    ssia

    M67

    US

    M76

    US

    M95

    US

    M96

    US

    M10

    1U

    SM12

    7U

    SM12

    9U

    S

    EntertainmentandTravelBuyingbehavior

    Entertainment&TravelPersonalTravel

    Entertainment&TravelBuyTickets

    Entertainment&TravelFrequentFliers

    Entertainment&TravelNonTravel/NonEntertainment

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • 0%

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%

    Sociographic

    SociographicSegmentsOpinionated/NotComputer

    SociographicSegmentsHappywithLife/NotComputer

    SociographicSegmentsHighComputer/StaysInformed

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • 0%

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%

    Media

    MediaSegmentsInternet MediaSegmentsStayInformed

    MediaSegmentsEnjoysPolitics MediaSegmentsConcerned

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Whatisaresearchertodo?

    Uncontrolledvariablescandriveinconsistency: Respondenttenure Professionalrespondents Speeders Consistencyerrors Satisficing Shiftingsamplesources

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Complexweightingschemes masktheproblemandarenotlikelytheanswerthereisnocensustofallbackon.Especiallywhentheproblemisinpurchasingintent.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Optimization

    Wecanminimizetheriskofinconsistencybyusingoptimizationmodels.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Optimization Profile The third panel is determinate.

    This is a three panel solution. We want to try to minimize the number of panels and minimize the distance from the Grand Mean.

    0.00

    0.01

    0.02

    0.03

    0.04

    0.050.06

    0.07

    RM

    S Er

    ror

    0%20%

    40%60%

    80%100%

    0% 20% 40% 60% 80% 100%

    Percent of M8

    Percent of M17

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • WecanoptimizetotheGrandMean.Inthisexampleweshowtheexpectedstandarderrorfromthe

    GrandMeanbasedontheaverageofallrandomchoices(8.31%).

    BasedonequalweightingofthreepanelsselectedbyoptimizationtotheGrandMean(2.36%)

    andthesamethreepanelsblendedinproportionstooptimizetotheGrandMean(0.40%).

    Panels Optimum AverageExpected

    (1SE)Inherent (1SE)

    M8 24% 33%M17 26% 33%M12 50% 34%

    Root Mean Square Error 0.40% 2.36% 8.31% 2.45%

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • CONSISTENCY

    Wemustknowifthedatashiftsweseearerealorchangesinthesample.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • ConsistentTrackAnAuditProgram

    Theobjectiveistocapturethevariabilityinonlinesamplesourcesthroughtime.

    ItisanaturaloffshootoftheGrandMeanproject. Thefirstwaveofdatacollectionforeachresearchpartnerispotentially

    thefirstwaveofaconsistencyanalysis. Aseriesofrepeatstudies,whenanalyzedaccordingtostandardquality

    controlmetrics,providesameasureofpanelvariabilitythroughtime. Thecostisminimalaseachpanelneedcompleteonlyonesequenceof

    wavesatregulartimeintervals. Interimreportsareprovidedateverywaveandcanbeusedasareference

    foralltrackingstudiesbeingcompletedatthattime. Thecombinationofmultipleconsistencyanalysiswithinamarketprovides

    datasimilartotheGrandMeanProjectonlyingreatervolumeandsequentiallythroughtime.

    CopyrightMktg,Inc.2009.Allrightsreserved.

  • Theresultsoftheaverageconsistencyofthesamplesetscomparedtotheoverallerrorboundforthevariousmetrics.

    Fifteenmetricswereanalyzed.Thelighterline/circledefinestheexpectederror.Thedarkerline/circleisthe

Recommended

View more >