fisher2012 jiaxing china keynote proms
DESCRIPTION
Keynote presentation given at the Pacific Rim Objective Measurement Symposium in Jiaxing China August 2012TRANSCRIPT
Scientific and Economic Value of the Metrological Point of View
William P. Fisher, Jr.University of California, Berkeley
Pacific Rim Objective Measurement Symposium
6-9 August 2012Jiaxing, China
Overview
• Some basic economic principles shared by science and commerce
• Three points of view on measurement in education
• The kinds of markets created by the three approaches to measurement
• A plan for the future
Economic Principles Shared ByScience and Commerce
• Separate local economies– Different currencies– Different weights and measures– Higher costs of exchange– Less efficient, harder to compare values
• Unified regional and global economies– Same currency– Same weights and measures– Lower costs of exchange– More efficient, easier to compare values
Example 1 of Scientific Market
• Biochemistry– Equipment calibrated in universal reference
standard metrics– Test results always reported in common units– Measures available on the spot– Easy to coordinate research across labs– Result: SARS virus sequenced in weeks by network
of labs, vaccine successfully synthesized
Example 2 of a Scientific Market
• Custom tailored suits– Tape measures calibrated in universal reference
standard metric– Results always reported in common units– Measures available on the spot– Easy to coordinate across tailors– Result: measures can be sent around the world
and a well fitting suit obtained with little trouble
Example 3 of Scientific Market
• Education– Tests typically not calibrated at all– If they are calibrated, they are in local units– Test results are usually reported in unique units– Measures available only after costly data analysis– Very difficult to compare outcomes outside of
special contexts– Result: Improvement efforts repeatedly fail,
quality uncontrolled, costs spiral higher
The Ideal Efficient Market
• Cost of estimating value is very low• Cost of comparing value for price is very low• Supply and demand easily match up• Low value for price: cannot compete• High value for price: rewarded• Improved value easy to recognize• Improved value pushes out old value
Readily available high quality information on product or service
High Cost Low Cost
Hard to match supply and demand
Easy to match supply and demand
Market Efficiency
Basic Economics
Hard to know how toimprove quality
Easy to know how toimprove quality
QualityImprovement
Hard for customers to find quality
Easy for customersto find quality
CustomerQuality-Seeking
Three Points of View on How to Present Information
on Educational Outcomes
• True Score Theory• Measurement Theory• Metrological Traceability
True Score TheoryDisconnected Scores and Tests
• School 1– Student A has a score of 22 on a reading test.– This classroom averages a score of 24.
• School 2– Student Z has a score of 18 on a reading test.– This classroom averages a score of 26.
True Score TheoryDisconnected Scores and Tests
• Who has more reading ability, A or Z? ??• What can one student read that the other
cannot? ??• Which classroom reads better on average? ??• Which student is more on track for college
readiness? ??
True Score TheoryDisconnected Scores and Tests
• School 1– Student A’s reading scores on 2 tests are 22 & 32.– The classroom average score goes from 24 to 30.
• School 2– Student Z’s reading scores on 2 tests are 18 & 32.– The classroom average score goes from 26 to 40.
True Score TheoryDisconnected Scores and Tests
• Who gained more in reading ability, A or Z? ??• What new texts can A and Z read? ??• Which classroom improves more? ??• Are both students on track for college
readiness? ??
• Result:– Very high cost, almost useless information
Disorganized, uncontrolled, decaying
Measurement TheoryConnected Measures and Tests
• School 1– Student A has a measure of 22 (+/- 2) on a reading
test.– This classroom averages a measure of 24 (+/- 1).
• School 2– Student Z has a measure of 18 (+/- 2) on a reading
test.– This classroom averages a measure of 26 (+/- 1).
Measurement TheoryConnected Measures and Tests
• Who has more reading ability, A or Z? A• What can one student read that the other
cannot?– Text with measures between 18 and 22.
• Which classroom reads better on average? 2• Which student is more on track for college
readiness? ??
• School 1– Student A’s measures on 2 tests are 22 & 32 (+/- 2).– The classroom average goes from 24 to 30 (+/- 1).
• School 2– Student Z’s measures on 2 tests are 18 & 32 (+/- 2).– The classroom average goes from 26 to 40 (+/- 1).
Measurement TheoryConnected Measures and Tests
• Who gained more in reading ability, A or Z? Z• What new texts can Z read?– Those with measures between 18 and 32.
• Which classroom improves more? 2• Are both students on track for college readiness?
??
• Result:– Very high cost, incomplete, but useful information
Measurement TheoryConnected Measures and Tests
Organized, expressive, preserved
Metrologically Traceable Measures
• School 1– Student A’s measure (22, +/- 2) is inferred when 73%
of the items built into a reading assignment targeted at 22 are answered correctly.
– This classroom averages a measure of 24 (+/- 1).• School 2– Student Z’s measure (18, +/- 2) is inferred when 76%
of the items built into a reading assignment targeted at 18 are answered correctly.
– This classroom averages a measure of 26 (+/- 1).
Metrologically Traceable Measures
• Who has more reading ability, A or Z? A• What can one student read that the other
cannot?– Text with measures between 18 and 22.
• Which classroom reads better on average? 2• Is one student more on track for college
readiness? Yes, A
• School 1– Student A’s measures on 2 tests are 22 & 32 (+/- 2).– The classroom average goes from 24 to 30 (+/- 1).
• School 2– Student Z’s measures on 2 tests are 18 & 32 (+/- 2).– The classroom average goes from 26 to 40 (+/- 1).
Metrologically TraceableConnected Measures and Tests
• Who gained more in reading ability, A or Z? Z• What new texts can Z read?– Those with measures between 18 and 32.
• Which classroom improves more? 2• Are both students on track for college
readiness? No, but A is
• Result:– Very low cost, complete and useful information
Metrologically TraceableConnected Measures and Tests
Coordinated, harmonized, growing
What to choose?True Score Theory Economics
School 1Average Grade 7 End of Year Teacher’ QuizReading Score = 89%
Average Gain in 7th Grade Readingas measured by in-classquizzes and tests: ??
Annual tuition = US$5,000 Cost of average gain in reading scores = US$??
School 2Average Grade 7 End of Year Teacher’ QuizReading Score = 94%
Average Gain in 7th Grade Readingas measured by in-classquizzes and tests: ??
Annual tuition = US$1,000 Cost of average gain in reading scores = US$??
Not enough information to decide!
Simulated data
What to choose?Measurement Theory Economics
School 1Average Grade 7 End of Year StatewideReading Measure = 32 (+/- 6)
Adjusted average gain in7th Grade ReadingMeasures = 10 (+/- 4)
Cost of adjusted average gain in reading measures = US$5,000.00
School 2Average Grade 7 End of Year StatewideReading Measure = 34 (+/- 5)
Adjusted average gain in7th Grade ReadingMeasures = 11 (+/- 3)
Cost of adjusted average gain in reading measures = US$1,000.00
Best buy
But do you really want to buy the average gain?
Simulated data
• My 7th grader’s gain – US$1,000 for 6 units– US$166.67 per unit gain
• Your 7th grader’s gain– US$1,000 for 9 units– US$111.11 per unit gain
What to choose?Measurement Theory Economics
50% greater cost!
What to choose?Measurement Theory Economics
ReadingAbilityScale
What to choose?Metrology Economics
Best buy
We might repeat the Measurement Theory outcomes…
School 1Average Grade 7 End of Year StatewideReading Measure = 32 (+/- 6)
Adjusted average gain in7th Grade ReadingMeasures = 10 (+/- 4)
Cost of adjusted average gain in reading measures = US$5,000.00
School 2Average Grade 7 End of Year StatewideReading Measure = 34 (+/- 5)
Adjusted average gain in7th Grade ReadingMeasures = 11 (+/- 3)
Cost of adjusted average gain in reading measures = US$1,000.00
Simulated data
• My 7th grader’s gain – US$833.40 for 6 units– US$138.90 per unit gain
• Your 7th grader’s gain– US$1,250.10 for 9 units– US$138.90 per unit gain
What’s a parent to choose?Metrology Economics
Same per unit cost!
Simulated data
Readily available high quality information on product or service
High Cost Low Cost
Hard for customers to find quality
Easy for customersto find quality
CustomerQuality-Seeking
Basic Economics
High stakes measurement theory
cost per test item:> US$3,000.00
Routine theory-informedmetrologically traceable
cost per test item:< US$0.01
What’s a teacher to choose?Metrology Economics
Cost per unit gain:US$180
Simulated data
Cost per unit gain:US$620
What’s a principal to choose?Metrology Economics
Three schoolsTwelve months each
A | B | C
Cost per unit gained
US$458 US$208 US$116
Simulated data
Bett
er R
eadi
ng O
utco
mes
Basic Shop Floor Questions
• What is variation trying to tell us? (Deming)• Which variations are due to common causes,
and which are due to special causes? (Shewhart)
• How far can educational outcomes be maximized, and unwanted variation reduced?
• Can variation in outcomes be reduced by bringing all students to the highest levels?
What’s needed?
• System of distributed units• Instruments measuring in uniform metrics• Predictive construct theories to bring down costs• Low cost items and administration• Immediate results• Continuous Quality Improvement (CQI) training
and tools• A culture that rewards innovation
What’s needed?
• We need commitment to a long range vision of quality education.
• But vision is not enough; we also need:– Skills– Incentives– Resources– Plans
What’s needed?
Vision + Skills + Incentives + Resources + Plan = Sustainable Change
+ Skills + Incentives + Resources + Plan = Confusion
Vision + + Incentives + Resources + Plan = Anxiety
Vision + Skills + + Resources + Plan = Resistance
Vision + Skills + Incentives + + Plan = Frustration
Vision + Skills + Incentives + Resources + = Treadmill
Adapted from Knoster, T. P., Villa, R. A., & Thousand, J. S. (2000). A framework for thinking about systems change. In R. A. Villa & J. S. Thousand (Eds.), Restructuring for caring and effective education: Piecing the puzzle together, 2nd Ed (pp. 93-128). Baltimore: Paul H. Brookes.
Disorganized, uncontrolled, decaying
Organized, expressive, preserved
Coordinated, harmonized, growing
Thank you