amy g. froelich department of statistics iowa state...

Amy G. FroelichDepartment of Statistics

Iowa State University

Acknowledgments This is joint work with Dr. Dianne Cook (Co-PI),

Xiaoyue Cheng, Kathleen Rey and James Curro This material is based upon work supported by

the National Science Foundation under Grant No. 12-45504.

Outline of Seminar Introductory Statistics Online Assessments

Structure of Online Homework Assignment Database Generation and Examples of Assignment Reports

Information about Student Learning Bar Graphs, Histograms and Variability Using the Normal Table

Introductory Statistics First course in statistics at the college or university

level. Typically 1 – 2 years of high school level algebra is

only prerequisite. Focus on three main areas:

Descriptive Statistics Data Collection through Random Samples and

Randomized Experiments Inferential Statistics

Reform Movement in Intro Stat Cobb (1992)

Emphasize Statistical Thinking The need for data The importance of data production The omnipresence of variability The quantification and explanation of variability

More Data and Concepts, Less Theory and Fewer Recipes

Foster active learning

5

Reform Movement in Intro Stat Moore (1997)

More data analysis, less probability Fewer lectures, more active learning Use Technology (for data analysis and simulations)

6

Reform Movement in Intro Stat Guidelines for Assessment and Instruction in

Statistics Education (GAISE) Reports from American Statistical Association (2005) Pre-K – 12 Curriculum Framework

Companion to NCTM Principles and Standards for School Mathematics (2000)

College Report Six Recommendations for Teaching Introductory Statistics 23 Course Level Outcomes for a First Course in Statistics

8

GAISE College Report Six recommendations for teaching

Emphasize statistical literacy and develop statistical thinking

Use real data Stress conceptual understanding, rather than mere

knowledge of procedures Foster active learning in the classroom Use technology for developing conceptual

understanding and analyzing data Use assessments to improve and evaluate student

learning

Assessments CAOS – Comprehensive Assessment of Outcomes

in a First Course in Statistics 40-item multiple choice test over material in

introductory statistics course. Emphasizes conceptual understanding, not

procedures. Used to assess effectiveness of interventions on

student learning.

Assessments Interventions are usually focused more on a

particular topic or topics. In order to develop an intervention, you first have to

understand: Where are the problems in student learning? What are the possible causes of these problems in

student learning? In order to assess the effectiveness of an

intervention, you have to be able to assess student learning of the affected outcomes.

Goals of Current Project Develop an electronic assessment model for

introductory statistics courses. Develop report generation software to give

instructors and course supervisors timely information about student learning and performance in course.

Study student learning of course topics Relative difficulty of topics. Common patterns in student learning. Common problems in student learning.

Electronic Assessments Current Project (Formative Assessments)

Homework* Vocabulary Clicker

Future Project (Summative Assessments)

Structure of Assessment Model Topics (27) Student Learning Outcomes (218) Question Sets (377) Questions (1291)

Topics Broad range across many different general intro

courses. Curriculum from AP Statistics and popular textbooks

used to guide selection. Not structured around specific textbook. Self-contained – instructors can choose only topics

covered in their course in any order.

Using Electronic Assessments to Inform Student Learning and Instruction in Introductory Statistics* 

Topic List 

Topic Number  Topic Description 01  Data02  Summarizing a Categorical Variable 03  Summarizing a Quantitative Variable04  Summarizing a Contingency Table 05  Summarizing a Quantitative Variable between Groups06  Normal Distribution 07  Summarizing Scatterplots 08  Summarizing the Least Squares Regression Line 09  Samples and Surveys 10  Experiments 11  Randomness and Probability 12  Introduction to Probability and Events13  Introduction to Random Variables 14  Binomial and Poisson Distributions15  Sampling Distribution for the Sample Proportion 16  Confidence Interval for the Population Proportion 17  Hypothesis Test for the Population Proportion 18  Sampling Distribution for the Sample Mean 19  Confidence Interval for the Population Mean 20  Hypothesis Test for the Population Mean 21  Inference for the Difference between Two Population Proportions 22  Inference for the Difference between Two Population Means 23  Inference for the Population Mean Difference (Paired Samples)24  Goodness of Fit Tests 25  Inference for Contingency Tables26  Inference for Simple Linear Regression 27  Basic Ideas of Statistical Inference 

*NSF DUE 1245504 

Learning Outcomes A list of statements describing what we want

students to know or be able to do after learning the topic.

Form structure of electronic assessment model. Typical number per topic is 7, ranges from 4 to 15.

Topic 16: Confidence Intervals for the Population Proportion

After studying this topic, students should be able to:

A. Calculate a sample proportion using data from a sample in order to estimate a population

proportion.

B. Calculate a confidence interval for the population proportion.

C. Interpret the confidence interval for the population proportion in context.

D. Understand and describe what it means to be confident.

E. Verify the conditions for the confidence interval method for a population proportion are

met.

F. Describe the effect of violating the success/failure condition on the confidence interval

method.

G. Describe the relationship between the width of a confidence interval (margin of error)

and the confidence level.

H. Describe the relationship between the width of a confidence interval (margin of error)

and the sample size.

I. Determine the sample size necessary for estimating the population proportion with a

particular degree of confidence within a specified margin of error.

Question Sets Group of similar questions with same format

covering same component of learning outcome Provide students with different questions

Specify number of questions presented to each student

Questions chosen randomly from set At least 1 question set per learning outcome No connection between question sets.

Questions Majority include real data examples Covers many application areas (excluding business) Current versions have been edited multiple times. Majority include answer-specific feedback or

correct/incorrect answer feedback.

Questions Question Types

TF: True/False (1 pt) MC: Multiple Choice (1 pt) MU: Multiple Answer (1 pt) MA: Matching (points = number of matches) FB: Fill in the Blank (points = number of blanks) JS: Jumbled Sentence (points = number of blanks) CA: Calculation (1 pt)

Question Titles Coded to identify all question components Will allow search by:

Topic Learning Outcome Question Set Question Type

Example – Question Title T16.A.A.04-1.1.MC.1

Example – Question TitleT16.A.A.04-1.1.MC.1

Topic 16

Example – Question Title T16.A.A.04-1.1.MC.1

Topic 16 Learning Outcome A for Topic 16

Example – Question Title T16.A.A.04-1.1.MC.1

Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16

Example – Question Title T16.A.A.04-1.1.MC.1

Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set.

Example – Question Title T16.A.A.04-1.1.MC.1

Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set. All questions in set worth 1 point.

Example – Question Title T16.A.A.04-1.1.MC.1

Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set. All questions in set worth 1 point. Multiple Choice question format

Example – Question Title T16.A.A.04-1.1.MC.1

Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set. All questions in set worth 1 point. Multiple Choice question format Question Label

Make Assignments Upload Topic Database to Blackboard Assignments open on first day of topic in lecture. Assignments close two class days after end of topic

in lecture. One attempt at each assignment. Question answers and feedback available to

students after assignment closes.

Learn from Student Responses Database structure allows analysis of student

performance at different levels. Topic Learning Outcome Question Set Questions

Electronic format provides “easily” accessible data on student learning.

IT’S NOT THAT EASY!!!

Student Response Output from Blackboard

Inputs for Report Generation Answer Key from Respondus

Question Titles Question text with answers Graphics

Student Responses from Blackboard List of Learning Outcomes

Turning Input into Output Functions in R

Parse graphics from Respondus Answer Key Clean Respondus Answer Key

Match Question IDs to correct Question Title. Match Student Responses to possible choices (ex. a, b, c,

d). Clean Student Response File from Blackboard Data summaries, graphics, and analyses

Reports Written in knitr/LaTeX Different Report Levels

Single Topic By Section Across Multiple Sections

Multiple Topics By Section Across Multiple Sections

Single Topic by Section How did my students do on this homework

assignment? Are there any particular learning outcomes for this

topic that are more difficult than others? Is there a pattern in student performance on the

learning outcomes for this topic?

Single Topic by Section Summary of Student Performance on Topic

Overall By Learning Outcome By Question Set By Question

By Learning Outcome

By Student• Outcomes E, A are fairly

easy for everyone.• Outcomes B, C tend to be

difficult only for lower scoring students.

• Outcomes G, I are difficult for low to medium scoring students.

• Outcomes D, F, H are difficult for broad range of students.

Heat Map of Student Percentile Rank by Learning Outcome

Single Topic Across Multiple Sections Are there are any sections whose performance is

higher than others? Overall By Learning Outcome By Question Set

Are there any problems with questions in the question set?

What are student responses for some of the more difficult questions?

Overall

Question T16.F.J.01-1.MC.1 • 58% of students

answered question correctly.

• Lowest performance of any learning outcome.

Multiple Topics by Section Student performance by Topic.

Overall Individual

Missing Assignment Analysis.

Missing Assignments (Spring 14)

Multiple Topics Across Sections Are there certain sections that either perform better

or worse consistently across topics? What are the most difficult/easiest topics in the

course?

Overall Performance (Spring 14) Insert graph from poster.

Informing Student Learning and Instruction What do students do well on? What do students struggle on? What changes can we make to instruction to

promote learning? Did our intervention work?

Example #1 – Bar Graphs, Histograms and Variability Current research* indicates students:

Do not understand the difference between a bar graph and histogram.

Confuse what is being displayed in the horizontal and vertical axes of a histogram.

Believe flatter histograms have less variability than unimodal histograms.

*Kaplan, J.J., Gabrosek, J.G., Curtiss, P., Malone, C. (2014). Investigating student understanding of histograms. Journal of Statistics Education, vol. 22, no. 2.

Bar Graph vs. Histogram Which of the following is/are appropriate displays for

the distribution of the Hair Color of Stat 101 students?

Question 1A Question 1BI. Bar ChartII. Frequency/Relative

Frequency TableIII. Stem and Leaf Plot

I. Bar ChartII. Frequency/Relative

Frequency TableIII. Histogram

Bar Graph vs. Histogram (F14)Question 1A Question 1B

Question Responses

Number of Students Percentage

Number of Students Percentage

I only 22 7.61 20 7.30II only 9 3.11 11 4.01I and II only 221 76.47 154 54.20I, II and III 37 12.80 89 32.48Total 289 100 274 100

Summary Statistics and the Histogram Two classes of 100 students each took the same

quiz. The possible scores on the quiz were 0, 1, 2, 3 or 4 points. The distribution of the quiz scores for the two classes can be found below.

Question Select the correct statement about the (statistic) quiz

scores of the two classes.A. The (statistic) scores of the two classes are the

same.B. The (statistic) score of Class 1 is greater than the

(statistic) score of Class 2.C. The (statistic) score of Class 1 is less than the

(statistic) score of Class 2.D. The relationship between the (statistics) cannot be

determined.

Student Responses (Fall 14)

Statistic Correct Answer

Percentage Correct out

of 534 studentsMedian A 69.66Mean A 70.22Range A 65.17IQR A 49.25Std. Dev. C 26.97

Current Work Further Investigation of Vocabulary Specific Questions on role of axes for bar graphs

and histograms. Does data representation (histogram, stem and leaf

plot, raw data) affect student responses to quiz question?

Compare different distributions.

Example #2 – Normal distribution and the z-table Outcome F – Find percentile or area values for a

given observation from a normal distribution. Outcome G – Find the value of an observation given

a percentile or area value from a normal distribution.

Example #2 (cont.) Spring 2014

Indications student performance related to: Probability – left-tailed or right-tailed Observation – less than mean or greater than mean

Database not structured for further testing.

Example #2 (cont.) Fall 2014

Restructure Database - Each student received one question of each type for each outcome: Left Tailed – less than mean Left-Tailed – greater than mean Right-Tailed – less than mean Right-Tailed – greater than mean

Build model to analyze results. Main effect plus interaction. Include additional variables – question scenario, section,

student.

Example #2 (cont.) Spring 2015

Students were shown applets for the normal distribution.

Reminded to use them on the online homework assignment.

Preliminary results indicate improved student performance.

Study effect on student understanding of statistical inference.

Future Work Develop set of equivalent summative electronic

assessments for each topic. Identify learning outcomes and questions for further

study. Develop and assess teaching interventions to

address problems with student learning.