descriptive statistics and exploratory data analysis...exploratory data analysis, page 1 exploring...

Descriptive Statistics andDescriptive Statistics and Exploratory Data AnalysisExploratory Data Analysis

DeanDean’’s Faculty and Residents Faculty and Resident Development SeriesDevelopment Series

UT College of Medicine ChattanoogaUT College of Medicine ChattanoogaProbasco Auditorium at ErlangerProbasco Auditorium at Erlanger

January 14, 2008January 14, 2008

Marc Loizeaux, PhDMarc Loizeaux, PhDDepartment of MathematicsDepartment of Mathematics

University of Tennessee at ChattanoogaUniversity of Tennessee at Chattanooga

What is descriptive statistics?What is descriptive statistics?

Descriptive statistics Descriptive statistics describesdescribes your your data.data.

Visual and NumericalVisual and Numerical

Inferential statistics Inferential statistics draws inferencesdraws inferences about a larger population.about a larger population.

Estimation and hypothesis testingEstimation and hypothesis testing

Statistics

Descriptive Inferential

Visual Numerical Estimation HypothesisTesting

The Big PictureThe Big Picture

Why descriptive statistics?Why descriptive statistics?

To summarize our dataTo summarize our dataTo help us get to know our dataTo help us get to know our dataTo help us describe our data to an To help us describe our data to an audienceaudienceTo help us explore our data.To help us explore our data.

What is Exploratory Data What is Exploratory Data Analysis?Analysis?

““Exploratory data analysis is detective workExploratory data analysis is detective work–– numerical detective work numerical detective work

–– or counting detective work or counting detective work –– or graphical detective workor graphical detective work””

-- John Wilder John Wilder TukeyTukey, , Exploratory Data AnalysisExploratory Data Analysis, page 1, page 1

Exploring our dataExploring our data

Gives us an overall viewGives us an overall viewHelps us consider basic assumptionsHelps us consider basic assumptionsHelps us spot oddball valuesHelps us spot oddball valuesHelps us avoid embarrassing oversightsHelps us avoid embarrassing oversightsMay help us decide on the next stepMay help us decide on the next step

Visual DescriptionsVisual Descriptions (Tools for exploring your data visually)(Tools for exploring your data visually)

Charts and GraphsCharts and Graphs–– HistogramHistogram–– DotplotDotplot–– Stem and leaf plotStem and leaf plot–– BoxplotBoxplot–– ScatterplotScatterplot–– And many moreAnd many more

A simple exampleA simple example Grades on the first examGrades on the first exam

84 75 83 48 70 31 39 51 57 68 5584 89 45 53 55 69 93 54 65 75 78

88 90 91 95 88 55 55 41 47 78

20 30 40 50 60 70 80 90 100

Numerical DescriptionsNumerical Descriptions((UnivariateUnivariate, interval data) , interval data) We want to describeWe want to describe……..

–– The The central tendencycentral tendency of the dataof the dataWhat is a center point for the data?What is a center point for the data?What is a typical score?What is a typical score?

–– The The variationvariation of the data?of the data?How much spread is there to the data?How much spread is there to the data?How far apart are the data values from each other?How far apart are the data values from each other?

Measures of Central TendencyMeasures of Central Tendency

The The meanmean is the arithmetic average.is the arithmetic average.–– Easy to calculate, easy to understandEasy to calculate, easy to understand–– The balance point of the dataThe balance point of the data

The The medianmedian is the score in the middle.is the score in the middle.–– Resistant to extreme scoresResistant to extreme scores

Measures of DispersionMeasures of Dispersion

The range.The range.–– Easy to calculate and quickEasy to calculate and quick

Range = high score Range = high score –– low scorelow score–– Limited Limited –– only considers two scoresonly considers two scores

The standard deviation.The standard deviation.–– More complicated, butMore complicated, but……–– Indicates a Indicates a ““typicaltypical”” deviation from the meandeviation from the mean

Childhood Respiratory DiseaseChildhood Respiratory Disease ((playing with the data)playing with the data)

Data available from Data available from OzDASLOzDASL, , StatSci.orgStatSci.org

FEV (forced expiratory volume) is an index of pulmonary functionFEV (forced expiratory volume) is an index of pulmonary function that that measures the volume of air expelled after one second of constantmeasures the volume of air expelled after one second of constant effort. effort.

The data: determinations of FEV on 654 children ages 6The data: determinations of FEV on 654 children ages 6--22 who were seen 22 who were seen in the Childhood Respiratory in the Childhood Respiratory DeseaseDesease Study in 1980 in East Boston, Study in 1980 in East Boston, Massachusetts. The data are part of a larger study to follow theMassachusetts. The data are part of a larger study to follow the change in change in pulmonary function over time in children.pulmonary function over time in children.

Source:Source:–– TagerTager, I. B., Weiss, S. T., , I. B., Weiss, S. T., RosnerRosner, B., and , B., and SpeizerSpeizer, F. E. (1979). Effect of , F. E. (1979). Effect of

parental cigarette smoking on pulmonary function in children. parental cigarette smoking on pulmonary function in children. American Journal American Journal of Epidemiologyof Epidemiology, , 110110, 15, 15--26. 26.

–– RosnerRosner, B. (1990). , B. (1990). Fundamentals of Biostatistics, 3rd EditionFundamentals of Biostatistics, 3rd Edition. PWS. PWS--Kent, Kent, Boston, Massachusetts. Boston, Massachusetts.

Some of the DataSome of the Data

ID Age FEV Height Sex Smoker46951 12 3.082 63.5 Female Non47051 13 3.297 65 Female Current47052 11 3.258 63 Female Non72901 12 2.935 65.5 Male Non73041 16 4.27 67 Male Current73042 15 3.727 68 Male Current73751 18 2.853 60 Female Non75852 16 2.795 63 Female Current77151 15 3.211 66.5 Female Non

Descriptive StatisticsDescriptive Statistics

Age FEV Height

Mean 9.93 2.64 61.14

Median 10.00 2.55 61.50

Mode 9 3.08 63

Standard Deviation 2.95 0.87 5.70

Range 16 5.00 28

Minimum 3 0.79 46

Maximum 19 5.79 74

Pictures may say morePictures may say more

The ages look like thisThe ages look like this

And againAnd again

One variable, then twoOne variable, then two……

A A univariateunivariate explorationexploration–– Explore each data column individuallyExplore each data column individually

A multivariate explorationA multivariate exploration–– Explore the relationships between two data Explore the relationships between two data

columnscolumns

Consider natural subgroupsConsider natural subgroups

Raising more questions?Raising more questions?

It starts to make senseIt starts to make sense

Something else to study?Something else to study?

Differentiating SubgroupsDifferentiating Subgroups

Preparing for an AudiencePreparing for an Audience

Some DoSome Do’’ss–– Pick and choose your graphsPick and choose your graphs–– Include appropriate numbers for your type of Include appropriate numbers for your type of

datadata–– Include narrativeInclude narrative

Does the histogram indicate asymmetry? Does the histogram indicate asymmetry? Are there unexpected values in the data set?Are there unexpected values in the data set?Are there special problems you had to deal with to Are there special problems you had to deal with to describe the data?describe the data?

Preparing for an Audience (2)Preparing for an Audience (2)

Some DonSome Don’’tsts–– DonDon’’t include everything t include everything –– that just confuses that just confuses

us.us.–– DonDon’’t be redundant t be redundant –– some graphs say the some graphs say the

same thing.same thing.–– DonDon’’t include descriptors you dont include descriptors you don’’t t

understand (kurtosis?) understand (kurtosis?) –– ask the chauffeurask the chauffeur

Points to RememberPoints to Remember (in no particular order)(in no particular order)

DonDon’’t skip the simple stuff!t skip the simple stuff!Spend time playing with your data.Spend time playing with your data.Pictures say a lot.Pictures say a lot.Describe the spread as well as the center.Describe the spread as well as the center.Consider the natural subgroups in your Consider the natural subgroups in your data.data.

Next TimeNext Time

Confidence Intervals,Confidence Intervals,Hypothesis Tests,Hypothesis Tests,

and Statistical Significanceand Statistical Significance2 x 2 tables2 x 2 tables

Monday, February 11Monday, February 11