amy g. froelich department of statistics iowa state...
TRANSCRIPT
Amy G. FroelichDepartment of Statistics
Iowa State University
Acknowledgments This is joint work with Dr. Dianne Cook (Co-PI),
Xiaoyue Cheng, Kathleen Rey and James Curro This material is based upon work supported by
the National Science Foundation under Grant No. 12-45504.
Outline of Seminar Introductory Statistics Online Assessments
Structure of Online Homework Assignment Database Generation and Examples of Assignment Reports
Information about Student Learning Bar Graphs, Histograms and Variability Using the Normal Table
Introductory Statistics First course in statistics at the college or university
level. Typically 1 – 2 years of high school level algebra is
only prerequisite. Focus on three main areas:
Descriptive Statistics Data Collection through Random Samples and
Randomized Experiments Inferential Statistics
Reform Movement in Intro Stat Cobb (1992)
Emphasize Statistical Thinking The need for data The importance of data production The omnipresence of variability The quantification and explanation of variability
More Data and Concepts, Less Theory and Fewer Recipes
Foster active learning
5
Reform Movement in Intro Stat Moore (1997)
More data analysis, less probability Fewer lectures, more active learning Use Technology (for data analysis and simulations)
6
Reform Movement in Intro Stat Guidelines for Assessment and Instruction in
Statistics Education (GAISE) Reports from American Statistical Association (2005) Pre-K – 12 Curriculum Framework
Companion to NCTM Principles and Standards for School Mathematics (2000)
College Report Six Recommendations for Teaching Introductory Statistics 23 Course Level Outcomes for a First Course in Statistics
8
GAISE College Report Six recommendations for teaching
Emphasize statistical literacy and develop statistical thinking
Use real data Stress conceptual understanding, rather than mere
knowledge of procedures Foster active learning in the classroom Use technology for developing conceptual
understanding and analyzing data Use assessments to improve and evaluate student
learning
Assessments CAOS – Comprehensive Assessment of Outcomes
in a First Course in Statistics 40-item multiple choice test over material in
introductory statistics course. Emphasizes conceptual understanding, not
procedures. Used to assess effectiveness of interventions on
student learning.
Assessments Interventions are usually focused more on a
particular topic or topics. In order to develop an intervention, you first have to
understand: Where are the problems in student learning? What are the possible causes of these problems in
student learning? In order to assess the effectiveness of an
intervention, you have to be able to assess student learning of the affected outcomes.
Goals of Current Project Develop an electronic assessment model for
introductory statistics courses. Develop report generation software to give
instructors and course supervisors timely information about student learning and performance in course.
Study student learning of course topics Relative difficulty of topics. Common patterns in student learning. Common problems in student learning.
Electronic Assessments Current Project (Formative Assessments)
Homework* Vocabulary Clicker
Future Project (Summative Assessments)
Structure of Assessment Model Topics (27) Student Learning Outcomes (218) Question Sets (377) Questions (1291)
Topics Broad range across many different general intro
courses. Curriculum from AP Statistics and popular textbooks
used to guide selection. Not structured around specific textbook. Self-contained – instructors can choose only topics
covered in their course in any order.
Using Electronic Assessments to Inform Student Learning and Instruction in Introductory Statistics*
Topic List
Topic Number Topic Description 01 Data02 Summarizing a Categorical Variable 03 Summarizing a Quantitative Variable04 Summarizing a Contingency Table 05 Summarizing a Quantitative Variable between Groups06 Normal Distribution 07 Summarizing Scatterplots 08 Summarizing the Least Squares Regression Line 09 Samples and Surveys 10 Experiments 11 Randomness and Probability 12 Introduction to Probability and Events13 Introduction to Random Variables 14 Binomial and Poisson Distributions15 Sampling Distribution for the Sample Proportion 16 Confidence Interval for the Population Proportion 17 Hypothesis Test for the Population Proportion 18 Sampling Distribution for the Sample Mean 19 Confidence Interval for the Population Mean 20 Hypothesis Test for the Population Mean 21 Inference for the Difference between Two Population Proportions 22 Inference for the Difference between Two Population Means 23 Inference for the Population Mean Difference (Paired Samples)24 Goodness of Fit Tests 25 Inference for Contingency Tables26 Inference for Simple Linear Regression 27 Basic Ideas of Statistical Inference
*NSF DUE 1245504
Learning Outcomes A list of statements describing what we want
students to know or be able to do after learning the topic.
Form structure of electronic assessment model. Typical number per topic is 7, ranges from 4 to 15.
Topic 16: Confidence Intervals for the Population Proportion
After studying this topic, students should be able to:
A. Calculate a sample proportion using data from a sample in order to estimate a population
proportion.
B. Calculate a confidence interval for the population proportion.
C. Interpret the confidence interval for the population proportion in context.
D. Understand and describe what it means to be confident.
E. Verify the conditions for the confidence interval method for a population proportion are
met.
F. Describe the effect of violating the success/failure condition on the confidence interval
method.
G. Describe the relationship between the width of a confidence interval (margin of error)
and the confidence level.
H. Describe the relationship between the width of a confidence interval (margin of error)
and the sample size.
I. Determine the sample size necessary for estimating the population proportion with a
particular degree of confidence within a specified margin of error.
Question Sets Group of similar questions with same format
covering same component of learning outcome Provide students with different questions
Specify number of questions presented to each student
Questions chosen randomly from set At least 1 question set per learning outcome No connection between question sets.
Questions Majority include real data examples Covers many application areas (excluding business) Current versions have been edited multiple times. Majority include answer-specific feedback or
correct/incorrect answer feedback.
Questions Question Types
TF: True/False (1 pt) MC: Multiple Choice (1 pt) MU: Multiple Answer (1 pt) MA: Matching (points = number of matches) FB: Fill in the Blank (points = number of blanks) JS: Jumbled Sentence (points = number of blanks) CA: Calculation (1 pt)
Question Titles Coded to identify all question components Will allow search by:
Topic Learning Outcome Question Set Question Type
Example – Question Title T16.A.A.04-1.1.MC.1
Example – Question TitleT16.A.A.04-1.1.MC.1
Topic 16
Example – Question Title T16.A.A.04-1.1.MC.1
Topic 16 Learning Outcome A for Topic 16
Example – Question Title T16.A.A.04-1.1.MC.1
Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16
Example – Question Title T16.A.A.04-1.1.MC.1
Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set.
Example – Question Title T16.A.A.04-1.1.MC.1
Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set. All questions in set worth 1 point.
Example – Question Title T16.A.A.04-1.1.MC.1
Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set. All questions in set worth 1 point. Multiple Choice question format
Example – Question Title T16.A.A.04-1.1.MC.1
Topic 16 Learning Outcome A for Topic 16 Question Set A for Topic 16 4 Questions in Set A – randomly assign 1 from set. All questions in set worth 1 point. Multiple Choice question format Question Label
Make Assignments Upload Topic Database to Blackboard Assignments open on first day of topic in lecture. Assignments close two class days after end of topic
in lecture. One attempt at each assignment. Question answers and feedback available to
students after assignment closes.
Learn from Student Responses Database structure allows analysis of student
performance at different levels. Topic Learning Outcome Question Set Questions
Electronic format provides “easily” accessible data on student learning.
IT’S NOT THAT EASY!!!
Student Response Output from Blackboard
Inputs for Report Generation Answer Key from Respondus
Question Titles Question text with answers Graphics
Student Responses from Blackboard List of Learning Outcomes
Turning Input into Output Functions in R
Parse graphics from Respondus Answer Key Clean Respondus Answer Key
Match Question IDs to correct Question Title. Match Student Responses to possible choices (ex. a, b, c,
d). Clean Student Response File from Blackboard Data summaries, graphics, and analyses
Reports Written in knitr/LaTeX Different Report Levels
Single Topic By Section Across Multiple Sections
Multiple Topics By Section Across Multiple Sections
Single Topic by Section How did my students do on this homework
assignment? Are there any particular learning outcomes for this
topic that are more difficult than others? Is there a pattern in student performance on the
learning outcomes for this topic?
Single Topic by Section Summary of Student Performance on Topic
Overall By Learning Outcome By Question Set By Question
By Learning Outcome
By Student• Outcomes E, A are fairly
easy for everyone.• Outcomes B, C tend to be
difficult only for lower scoring students.
• Outcomes G, I are difficult for low to medium scoring students.
• Outcomes D, F, H are difficult for broad range of students.
Heat Map of Student Percentile Rank by Learning Outcome
Single Topic Across Multiple Sections Are there are any sections whose performance is
higher than others? Overall By Learning Outcome By Question Set
Are there any problems with questions in the question set?
What are student responses for some of the more difficult questions?
Overall
Question T16.F.J.01-1.MC.1 • 58% of students
answered question correctly.
• Lowest performance of any learning outcome.
Multiple Topics by Section Student performance by Topic.
Overall Individual
Missing Assignment Analysis.
Missing Assignments (Spring 14)
Multiple Topics Across Sections Are there certain sections that either perform better
or worse consistently across topics? What are the most difficult/easiest topics in the
course?
Overall Performance (Spring 14) Insert graph from poster.
Informing Student Learning and Instruction What do students do well on? What do students struggle on? What changes can we make to instruction to
promote learning? Did our intervention work?
Example #1 – Bar Graphs, Histograms and Variability Current research* indicates students:
Do not understand the difference between a bar graph and histogram.
Confuse what is being displayed in the horizontal and vertical axes of a histogram.
Believe flatter histograms have less variability than unimodal histograms.
*Kaplan, J.J., Gabrosek, J.G., Curtiss, P., Malone, C. (2014). Investigating student understanding of histograms. Journal of Statistics Education, vol. 22, no. 2.
Bar Graph vs. Histogram Which of the following is/are appropriate displays for
the distribution of the Hair Color of Stat 101 students?
Question 1A Question 1BI. Bar ChartII. Frequency/Relative
Frequency TableIII. Stem and Leaf Plot
I. Bar ChartII. Frequency/Relative
Frequency TableIII. Histogram
Bar Graph vs. Histogram (F14)Question 1A Question 1B
Question Responses
Number of Students Percentage
Number of Students Percentage
I only 22 7.61 20 7.30II only 9 3.11 11 4.01I and II only 221 76.47 154 54.20I, II and III 37 12.80 89 32.48Total 289 100 274 100
Summary Statistics and the Histogram Two classes of 100 students each took the same
quiz. The possible scores on the quiz were 0, 1, 2, 3 or 4 points. The distribution of the quiz scores for the two classes can be found below.
Question Select the correct statement about the (statistic) quiz
scores of the two classes.A. The (statistic) scores of the two classes are the
same.B. The (statistic) score of Class 1 is greater than the
(statistic) score of Class 2.C. The (statistic) score of Class 1 is less than the
(statistic) score of Class 2.D. The relationship between the (statistics) cannot be
determined.
Student Responses (Fall 14)
Statistic Correct Answer
Percentage Correct out
of 534 studentsMedian A 69.66Mean A 70.22Range A 65.17IQR A 49.25Std. Dev. C 26.97
Current Work Further Investigation of Vocabulary Specific Questions on role of axes for bar graphs
and histograms. Does data representation (histogram, stem and leaf
plot, raw data) affect student responses to quiz question?
Compare different distributions.
Example #2 – Normal distribution and the z-table Outcome F – Find percentile or area values for a
given observation from a normal distribution. Outcome G – Find the value of an observation given
a percentile or area value from a normal distribution.
Example #2 (cont.) Spring 2014
Indications student performance related to: Probability – left-tailed or right-tailed Observation – less than mean or greater than mean
Database not structured for further testing.
Example #2 (cont.) Fall 2014
Restructure Database - Each student received one question of each type for each outcome: Left Tailed – less than mean Left-Tailed – greater than mean Right-Tailed – less than mean Right-Tailed – greater than mean
Build model to analyze results. Main effect plus interaction. Include additional variables – question scenario, section,
student.
Example #2 (cont.) Spring 2015
Students were shown applets for the normal distribution.
Reminded to use them on the online homework assignment.
Preliminary results indicate improved student performance.
Study effect on student understanding of statistical inference.
Future Work Develop set of equivalent summative electronic
assessments for each topic. Identify learning outcomes and questions for further
study. Develop and assess teaching interventions to
address problems with student learning.