stascal thinking at the introductory level stascs and ... › ~nolan › talks › beijing ›...

7/21/12

1

Sta)s)calThinkingattheIntroductoryLevel

DeborahNolanUniversityofCalifornia,Berkeley

Sta)s)csandMathema)cs

AdaptedfromCobbandMoore,Mathema)cs,Sta)s)cs,andTeaching

Sta)s)csisa

•  Mathema)calscience•  Datascience•  Computa)onalscience

•  Sta)s)csisnotasubfieldofmathema)cs

•  Sta)s)csmakesessen)aluseofmathema)cs

Aphorisms:

GeorgeBox:•  Allmodelsarewrong,butsomeareuseful

GeorgeCobb:

•  Inmathema)cs,contextobscuresstructure.Indataanalysis,contextprovidesmeaning

7/21/12

2

Aphorisms:

DavidMoore:•  Mathema)caltheoremsaretrue:sta)s)calmethodsaresome)meseffec)vewhenusedwithskill

Variability

•  Needofsta)s)csarisesfromtheomnipresenceofvariability

•  Repeatedmeasurementsonthesameindividualvary.

•  Some)meswewanttofindunusualindividuals

•  Other)meswefocusonthevaria)onofmeasurements.

•  Other)meswewanttodetectsystema)ceffectsagainstthebackgroundnoiseofindividualvaria)on.

Theroleofcontextinsta)s)cs

Context

•  Sta)s)csrequiresadifferentkindofthinking•  Dataarenotjustnumbers

•  Dataarenumberswithacontext

7/21/12

3

Mathema)csandContext

•  Contextisusedformo)va)on•  Contextisasourceofproblems

•  Ul)mately:–  Contextistheirrelevantdetailthatweignoretorevealthehiddenpurestructure.

–  Contextobscuresstructure.–  FocusisonabstractpaVerns

DataAnalysisandContext

•  FocusisalsoonstructureandpaVerns•  Ul)mately,– ThepaVernshavemeaning,– Thestructurehasvalue,–  IfthepaVernsmakesenseinthecomplementarythreadsofthestoryline

•  Contextprovidesmeaning

Implica)onsforteaching

Implica)onsforteaching

•  Needmorethanmathema)caltheory•  Needtounderstandthenon‐mathema)caltheoryofsta)s)cs

AND

•  Needrealillustra)ons•  Needtouseillustra)onstodevelopcri)caljudgment

7/21/12

4

Essen)alpiecesofsta)s)calanalysis

•  Designfordataproduc)on•  Explora)onforpaVernsandstructure•  Formula)onofmodels

•  Applica)onofmethods

•  Summarizeresults

•  Interpreta)onofresults

ASA/MAARecommenda)ons

•  Almostanysta)s)cscoursecanbeimprovedonby– Moreemphasisondataanalysis

– Moreemphasisonconcepts– Fewerrecipes– Lesstheory

ASA/MAARecommenda)ons

•  Mainfocusofanintroductorycourseshouldbeonsta)s)calthinking.

•  Sta)s)calThinkingincludes:– Theneedfordata– Theimportanceofdataproduc)on,– Theomnipresenceofvariability,– Thequan)fica)onandexplana)onofvariability.

Sta)s)cstaughtasmagic

•  Studentisthesorcerer’sappren)ce•  Incanta)onhasautoma)ceffec)veness,e.g.rendersastudypublishable

•  Appren)ceisnotmeanttounderstandhowtheincanta)onworks

•  Followtherecipeexactly,beVeryet–useso]ware

7/21/12

5

CounterSta)s)csasmagic

•  Retrea)ngtomathema)csdoesnotsolvetheproblem

CounterSta)s)csasmagic

•  Presentanintellectualframeworkthat– Makessenseofthecollec)onoftoolsthatsta)s)ciansuse

– Encouragesflexibleapplica)onoftoolstosolveproblems.

– Reasonsfromuncertainempiricaldata

TopicsTodayandTomorrow

Workshoptopics

1)  Examplesofrealillustra)onstodevelopastudent’scri)caljudgment

2)  Compu)ngtechnologyhascompletelychangedtheprac)ceofsta)s)csandisanecessarytool

3)  BoxModel–Ateachingstrategyforlearninghowrandomnessindataproduc)onleadstoinference

7/21/12

6

Workshoptopics

4)  Engagingingraphicsearlycanestablishgoodhabits,preparefordesignandinference,provideexperiencewithdatadistribu)ons,introduceconcepts

5)  Physicalexamplescangiveapictoralgraspofasta)s)calconceptandbeeffec)veinconveyingideas

Sta)s)csCoursesatBerkeley

IntroductoryCourses

•  Quan)ta)veReasoningrequirementforArtsandHumani)esmajors

•  Sta)s)csforBusinessmajors

•  Sta)s)csforstudentswithcalculusbackground

IntroductorySta)s)csCourse

•  120students•  Firstandsecondyearstudents•  Undeclaredmajors,interestedineconomicsandbiologicalandphysicalscience

•  Calculusprerequisiteforthecourse•  3hoursof“lecture”aweek•  2hoursoflab–25studentstothelab

7/21/12

7

•  Threeversionsoftheintroductorycourse•  Allfocusonsta%s%calthinking•  PhilosophyofSta%s%cs,Freedman,Pisani,andPurves

•  Thisphilosophyappearsthroughoutthecurriculum––  inthetheore)calcoursesforthesta)s)csmajorand

–  inthePhDlevelcourses

Threestories:Realillustra)onstodevelopastudent’scri)caljudgment

RandomizedControlledExperiments

TheHIPTrialadaptedfromSta)s)calModels:TheoryandPrac)ce,Freedman

Background

•  Breastcancercommonmalignancyamongwomen

•  IfDetectedearly,thenchanceofsuccessfultreatmentbeVer

•  Mammography–screeningbyX‐ray

7/21/12

8

Doesmammographyspeedupdetec)onenoughtomaVer?

HealthInsurancePlaninNewYork

•  HIP–groupmedicalprac)cewith700,000membersin1960s

•  Subjects:62,000women– Aged40‐64– MembersofHIP

•  Splitatrandomintotwogroups

Treatments

•  “Treatment”:invita)onto4roundsofannualscreening– Clinicalexam

– Mammography

•  Control:receivedusualhealthcare

Results

GroupSize

BreastCancerNumber

Rate

Treatment 31,000 39 1.3

Control 31,000 63 2.0

Arethesetherightnumberstocompare?Notallwomeninthetreatmentgroupacceptedtreatment

7/21/12

9

Results

GroupSize

BreastCancerNumber

Rate

Treatment

Screened 20,200 23 1.1

Refused 10,800 16 1.5

Total 31,000 39 1.3

Control 31,000 63 2.0

Adifferentcomparison:thosewhoacceptedscreeningtothosewhorefused.

Results

GroupSize

BreastCancerNumber

Rate

Treatment

Screened 20,200 23 1.1

Refused 10,800 16 1.5

Total 31,000 39 1.3

Control 31,000 63 2.0

Anothercomparison:thosewhoacceptedscreeningtothoseincontrolgroup.

Whichcomparisonmakessense?

Considera)ons

•  Inves)gatorschose(atrandom)thosetoreceivetreatment

•  Subjectsthemselvesdecidedwhetherornottoaccepttreatment

•  ComparingthosewhoaccepttothosewhorefuseisanObserva)onalComparison

7/21/12

10

Differencesbetweengroups

•  RicherandbeVer‐educatedsubjectsmorelikelytoacceptinvita)onthanthosewhowerepoorerandlesswell‐educated

•  Richerwomenarelessvulnerabletomostdiseases,butbreastcancerhitsrichharder

•  Socialstatusisaconfoundingfactor:afactorassociatedwiththeoutcomeandwiththedecisiontoacceptscreening

Whichcomparison?

•  Thecomparisonofthosewhoaccepttreatmenttothosewhorefuseisbiasedagainstscreening

•  Thecomparisonofthosewhoaccepttreatmenttothoseinthecontrolgroupisalsoproblema)cbecausethecontrolgroupincludeswomenwhowouldhaverefusedscreening

Results

GroupSize

BreastCancerNumber

Rate

AllOtherNumber

Rate

Treatment

Screened 20,200 23 1.1 428 21

Refused 10,800 16 1.5 409 38

Total 31,000 39 1.3 837 27

Control 31,000 63 2.0 879 28

•  Experimentalcomparison:betweenwholetreatmentgroupandwholecontrolgroup

•  Inten)on‐to‐treatanalysis•  Effectoftheinvita)on:– 63–39=24livessaved–  Inrela)veterms39/63=62%

7/21/12

11

SurveySample

TheHiteReport,adaptedfromSampling:Designand

Analysis,byLohr

Publishedin1987

Surveyof4,500women

Hite’ssample&USpopula)on

Income Hite’sSampleU.S.popula)on

under$2,000 19.0% 18.3%$2,000‐$4,000 12.0% 13.2% $4,000‐$6,000 12.5% 12.2%$6,000‐$8,000 10.0% 9.7%$8,000‐$10,000 7.0% 7.4%$10,000‐$12,500 8.0% 8.8%$12,500‐$15,000 5.0% 6.2%$15,000‐$20,000 10.0% 9.8%$20,000‐$25,000 8.0% 6.4%$25,000andover8.5% 8.2%

Hite’ssampleandUSpopula)onTypeofarea Hite’sSample U.S.popula)on

Largecity/urban 60% 62%Rural 27% 26%Smalltown 13% 12%Race Hite’sSample U.S.popula)on

White 82.5% 83.0%Black 13.0% 12.0%Hispanic 1.8% 1.5%Asian 1.8% 2.0%

7/21/12

12

AccordingtotheWomenSurveyed:

•  84%ofwomenare“notsa)sfiedemo)onallywiththeirrela)onships”.

•  70%ofallwomen“marriedfiveormoreyearsarehavingsexoutsideoftheirmarriages”.

•  98%wanttomakebasicchangesintheirrela)onships.

•  84%ofwomenreportformsofcondescensionfromthemenintheirloverela)onships.

Thissurveyappearedtobeground‐breaking?

Or,didsomethinggowrong?

OtherStudiesandExperts:

•  Harrispoll(1987)89%saytheirrela)onshipwiththeirpartnerissa)sfying.

•  "Anyques)onyouaskedthatgot98%iseitherawrongques)onorwronglyphrased"saysTomSmithoftheNa)onalOpinionResearchCenter.

•  Severalotherpollsfound25‐30%whoaremarriedhavehadorarehavinganextramaritalaffair.

Ques)onnaire

•  Vaguewordingofques)ons,e.g.“inlove”•  Leadingques)ons:Doesyourhusband/loverseeyouasanequal?Orarethere%meswhenheseemstotreatyouasaninferior?Leaveyououtofthedecisions?Actsuperior?

•  Longsurveywith127essayques)ons,manywithseveralparts

7/21/12

13

Selec)onBias

•  Ques)onnairesmailedtowomen’sgroupsincludingprofessionalwomen’sorganiza)ons,counselingcenters,churchsocie)es,…Non‐responsehigh:100,000ques)onnairesmailed,4.5%returned

•  Self‐selectedsample

=USWomen

Women’sGroups

Notincluded=WomennotbelongingtoaWomen’sGroup

Belongtogroupbut:Didn’treceiveques)onnaireChosenottofillout

Womenwhobelongtoagroupandtookthe)metocompletethesurvey

=Girls&Meninthegroups

HypothesisTes)ng

UnitedStatesv.KristenGilbert

(adaptedfromCobbandGehlbach,“Sta)s)csintheCourtroom”,inSta%s%cs:AGuidetotheUnknown)

KristenGilbert

•  Bornin1967inFallRiver,MA•  Graduatedhighschoolat16•  GraduatedfromGreenfieldCC,andreceivedcer)fica)onasaregisterednursein1988.

•  In1989,shejoinedtheVAMedicalClinicinNorthampton,MA.

7/21/12

14

VAMedicalClinic

Whenapa)entwentintocardiacarrest,shewould:– Soundthecodebluealarm

– Staycalm– Administerashotofepinephrinetorestarttheheart

Gilbertestablishedareputa)onofbeingpar)cularlygoodincrisis

Suspicions

•  Bythemid1990’s,othernurseshadbecomesuspiciousofGilbert.

•  Itseemedthereweretoomanycodebluecalls,toomanycriseswhenGilbertwasontheward.

•  Anini)alVAreportfoundthatthenumberofdeathswereconsistentwithpaVernsatotherVAhospitals.

•  Thesuspicionsofthestaffremained.

Suspicions

•  Thestaffbroughttheirconcernsagaintotheadministra)onoftheVAClinic.

•  Theyhiredasta)s)cianasaconsultanttolookintothesitua)on(Gehlbach)

•  Hisfindingsagreedwiththestaffconcerns.

AssistantU.S.AVorneyWelchconvenedagrandjuryin1998toheartheevidenceagainstGilbert.

7/21/12

15

GrandJury

•  Agrandjurydetermineswhetherthereisenoughevidenceforatrial.

•  Thegrandjuryexaminesevidenceandissuesanindictment,aformalaccusa)onthatapersonhascommiVedacrime.

EvidenceConsidered

•  MoJvaJon–Gilbertlikedthethrillofacrisis,neededtherecogni)on,andwantedtoimpressherboyfriendwhoalsoworkedattheVAClinic.

•  TesJmonyofco‐workersaboutaccessGilberthadtoepinephrine.

•  TesJmonyofaphysicianaboutthesymptomsofthemen(healthy,middle‐aged,nottypicalcandidatesforcardiacarrest).

Convincing?

•  NoonehadseenGilbertgivefatalinjec)ons.•  AmajorpartoftheevidencewasstaJsJcal.

QUESTION:WeretheresomanyexcessdeathswhenGilbertwaspresentastobesuspiciousintheeyesofscience?

7/21/12

16

Gelbach’sTes)mony

•  PaVernofdeaths,byshi]andbyyearonthemedicalwardwhereGilbertworked

•  Variabilityinachanceprocess

•  Sta)s)caltestforwhetherthepaVernlinkingtheexcessdeathstoGilbert’spresenceonthewardwastooextremetoberegardedasordinary,expectedvariability

PaVerninDeaths

1988 1990 1992 1994 1996

010

20

30

40

year

Deaths

Night

Day

Evening

PaVerninDeaths

•  ThereisaclearpaVernassocia)ngGilbert’spresencewithexcessdeaths

•  However,thepaVernmightbenothingmorethantheresultofordinary,expectablevariaJon.

Sta)s)calTest

•  Considerthe18monthsleadinguptoFeb1996,whenGilbertwentonmedicalleave.

•  Therewere547daysinthis18monthperiodand3shi]saday,foratotalof1641shi]s

•  Foreachshi],wehavewhetherornotGilbertworkedtheshi]andwhetherornottherewasadeathontheshi].

7/21/12

17

ASta)s)calTest

!"#$%&'(&)%*+$,&

-*./"0$&10")"($,& & 234& & (5& & $5678&

234& & & & & 9:& & &&&;<=& & &&&;>=&

(5& & & & & ?9& & <?>:& & <?@9&

$5678& & & & & =9& & <>A=& & <A9<&

&

• Thetablesummarizesrecords.• Inthe1641shi]stherewere74shi]sforwhichtherewasatleastonedeath.• On40ofthe74shi]s,Gilbertwasworking.Isthatmorethanyouwouldexpect?

P(atleast40)<1inatrillion

0 20 40 60

0.00

0.04

0.08

0.12

Shifts with at least one death

Chance

Howlikelyisittogetatleast40deathsinthe257shi]s?

GrandJury

•  Thegrandjuryfoundthesta)s)calevidencepersuasive

•  Gilbertwasindicted•  TheVAhospitalislegalpropertyofthefederalgovernmentsoitwasafederalindictment.

•  Trialwouldbeinfederaldistrictcourt,andthedeathpenaltywouldbeapossiblesentence,iffoundguilty.

TheTrial

•  ThepeJtjury(ortrialjury)hearstheevidenceinatrialaspresentedbyboththeplain)ffandthedefendant.

•  A]erhearingtheevidence,thegroupre)resfordelibera)on,toconsideraverdict.

•  Themajorityrequiredforaguiltyverdictwasasimplemajority(7outof12).Aunanimousverdictforthesentencewasneededforthedeathpenalty.

7/21/12

18

ExpertWitness

Thecourtsystemallowsexperttes)monywhentheevidenceinvolvesspecializedtechnicalorscien)ficissuesthatgobeyondwhatjurorswouldordinarilybefamiliarwith.

TheexpertshelpthejuryunderstandtheevidencebeVer.

DuelingExpertWitnesses

•  SupremeCourthassetguidelinesatmakingsureunscien)fictes)monyisnotadmiVed.

•  Ifthereisexperttes)monyononeside,aVorneysfortheothersidesome)meshireanotherexpertwhowilldisagreeand“cancelout”theotherexpert.

QUESTION:Shouldthetrialjurybeallowedtohearthesta)s)calevidence?

ReporttotheJudge

•  Wasthesta)s)calanalysisdonecorrectly?•  Whatdoestheprobabilitycalcula)onNOTtellyou?

• Associa)onvsCausa)on:ConclusionsdrawnfromanObserva)onalStudyvsaDesignedExperiment

• Prosecutor’sFallacy:Probabilitycomputedunderassump)onsofinnocence

7/21/12

19

Wasthesta)s)calanalysisdonecorrectly?

•  TheDefenseSta)s)cian(Cobb)agreedwiththeanalysisperformedbytheProsecu)onSta)s)cianforthegrandjury.

•  Thenumberofexcessdeathswasanextremelyunlikelyoutcomeduetochancevaria)on.

•  ThepaVernofdeathsjus)fiedtheindictment.

Associa)onvsCausa)on

•  Inawell‐designedexperiment,e.g.theSalkFieldtrials,theonlydifferenceisinthetreatment,allotherpossiblecausesofaneffecthavebeeneliminated.

•  The)nyprobabilityrulesoutchancevaria)on,andtheconclusionisthatthedifferenceis“real”.

•  Inadesignedexperiment,wecanconcludethattheexplana)onfortheobserveddifferenceisthetreatment.

Associa)onvsCausa)on

•  ThiswasNOTarandomizedcontrolledexperiment.(Gilbert’spresenceonthewardwouldhavehadtobeassignedusingachancedevice).

•  Wecanconcludethatthedifferenceisnotduetochancevaria)on,butthe)nyprobabilitydoesnotprovideanexplana)onforwhathappened.

•  Therecouldbeotherpossibleexplana)ons.

Prosecutor’sFallacy

•  Theprobabilityof1inatrillionwascomputedassumingthataresultisduetochancevaria)on.

•  Wecomputethechanceofgexngaresultasextremeastheoneobserved.

•  Ifitisveryrare()nyprobability),weconcludethatitisnotreasonabletothinkthatrandomvaria)onisthecause.

•  Thislogicsaysnothingaboutothercauses

7/21/12

20

Prosecutor’sFallacy–slipperylogic

•  SupposeGilbertisinnocentandthedeathsbehaveinachance‐likeway.Theprobabilityislessthan1inatrillionthatyouwouldseesomanyexcessdeathsonGilbert’sshi]s.

•  IfGilbertisinnocent,thenitwouldbealmostimpossibletogetsomanyexcessdeaths.

•  Withthismanyexcessdeaths,thechanceislessthanoneinatrillionthatGilbertisinnocent.FALSELOGIC

Conclusion

•  Itisveryeasytobetemptedbythefalselogic(thattheprobabilityisthechanceofinnocence).

•  Judgeruledthatsta)s)calevidenceshouldnotbeallowedattrial.

•  JuryfoundGilbertguiltyon3countsoffirst‐degreemurderand2countsofaVemptedmurder.

•  Juryvoted8‐4foradeathpenalty.•  Gilbertwasgivenlifeinprisonwithoutpossibilityofparole.

Exercisestotry

Considera)ons

•  Provideananswerin“plainEnglish”or“plainChinese”

•  Iden)fythecoresta)s)calthinkingconcept•  Howisthemathinsufficientforansweringtheques)on?

7/21/12

21

Discussion

•  Whatarethechallengestoteachingthisway?

•  Hardtoteachtheseconcepts•  Hardtogradestudentwork•  Contextcandependonculturalbackground•  O]entheexamplesareveryreduc)onist

•  Thiscanleadtoanover‐cri)calapproach

Discussion

•  Whatarethebenefitstoteachingthisway?

•  Seethatthereismoretosta)s)csthanmanipula)onofformula

•  Gainprac)ceinsta)s)calthinking•  Seeingmanyexampleswillhelpwhenconfrontnewproblems

Sta)s)csshouldbetaughtassta)s)cs

Resources

•  Sta%s%cs,Freedman,Pisani,Purves

•  Sta%s%csaGuidetotheUnknown,Mostelleretal,edi)on

•  Sta%s%csaGuidetotheUnknown,Pecketall,edi)on

•  StatLabs:TheorythoughApplica%ons,Speed&Nolan