developing and using measures

8/11/2019 Developing and Using Measures

1/34

http://www.sii.soe.umich.edu/

http://sitemaker.umich.edu/lmt

1

Developing Measures ofMathematical Knowledge for Teaching

Geoffrey Phelps, Heather Hill,

Deborah Loewenberg Ball, Hyman Bass

Learning Mathematics for Teaching

Study of Instructional Improvement

Consortium for Policy Research in Education

University of Michigan

MSP Regional Conference

Boston, MA

March 30-31, 2006


2/34



2

Overview of todays session

1. LMT/SII Measures Development

2. Some Sample Results

3. LMT/SII Measures and Dissemination


3/34



3

Subtract:

What isContent Knowledge for Teaching?An Example From Subtraction

3002

783-


4/34



4

Analyzing Student Errors

3002

783-

2781

3002 - 783 = 4832


5/34



5

Analyzing Unusual Student Solutions

3002

783-

299

2219

12 3 0 0 2

7 8 3-

3-7-8-1

2 2 1 9


6/34

LMT/SII Measures Development


7/34



7

WhyWould We Want to MeasureTeachers Content Knowledge for Teaching?

To understand what constitutes mathematical

knowledge for teaching

To understand the role of teachers content

knowledge in students performance

To study and compare outcomes of professional

development and teacher education

To inform design of teachers opportunities tolearn content knowledge


8/34



8

Measuring Teachers MathematicsKnowledge: Background and History

Research on teacher behavior

Early research on student achievement

Proxy measures for teacher knowledge

Tests of basic skills 1985 on: the missing paradigm pedagogical

content knowledge or PCK

1990s: interview studies of teachers

mathematics knowledge (MSU -- NCRTE)


9/34



9

Study of Instructional Improvement

Study of three Comprehensive School Reforms;teacher knowledge a key variable

Instrument development goals:

Develop measures of content knowledge teachersusein teaching K-6 content for elementary school teachers

Not just whatthey teach - specialized knowledge

Develop measures that discriminate among teachers

(not criterion referenced) Non-ideological

But we faced significant problems.


10/34



10

Problems As We Began This Work

No way to measure teachers contentknowledge for teaching on a large scale Small number of items, many written by Ball,

Post, others appeared on every instrument Nothing known about the statistical qualities of

those items (difficulty, reliability)

Studies relied on single items, yet single items

unlikely valid or reliable measures of teacherknowledge


11/34



11

Early Decisions and Activity

Survey-based measure of CKT-M 3000 teachers participating in SII

Multiple choice

Specified domain map

5 people + 5 lbs cheese + 5 weeks = 150

items (May 2001)

Large-scale piloting, summer 2001


12/34



12

Early Decisions and Activity

Types of knowledgeMat

hematical

content

Content knowledge Knowledge of

content

and students

Number

Operations

Patterns, functions,and algebra


13/34



13

Early Analyses and Validity Checks

Results from piloting

We canmeasure teachers CKT-M

Reliabilities of .70-.90

Factor analysis shows distinct types ofknowledge

Knowledge of content and students (KCS)separate from CK

Specialized content knowledge (SCK) vs.common content knowledge (CCK)

Hill, H.C., Schilling, S.G., & Ball, D.L. (2004) Developing measures of teachers

mathematics knowledge for teaching. Elementary School Journal 105, 11-30.


14/34



14

Reliabilities (1PL-IRT): Elementary

Knowledge of content Knowledge of content

and students

Number and

operations (K-6) .72-.81 .58-.67

Patterns, functions,

and algebra (K-6) .70-.85

Geometry (3-8) .85-.86


15/34



15

Reliabilities (1PL-IRT):Middle School

Knowledge of content Knowledge of content

and students

Number and

operations (5-9) .74-.75

Patterns, functions,

and algebra (5-9) .86-.89

Geometry (3-8) .84-.86


16/34



16

Content Knowledge :Number and Operations

Common knowledge Number halfway between 1.1 and 1.11

Specialized knowledge Representing mathematical ideas and operations

Providing explanations for mathematical ideas and

procedures Appraising unusual student methods, claims, orsolutions


17/34



17

Representing Number Concepts

Mrs. Johnson thinks it is important to vary the whole when she

teaches fractions. For example, she might use five dollars to be the

whole, or ten students, or a single rectangle. On one particular day,

she uses as the whole a picture of two pizzas. What fraction of the

two pizzas is she illustrating below? (Mark ONE answer.)a)5/4

b)5/3

c)5/8

d)1/4


18/34



18

Providing Mathematical Explanations:Divisibility Rules

Ms. Harris was working with her class on divisibility rules. She told herclass that a number is divisible by 4 if and only if the last two digits of thenumber are divisible by 4. One of her students asked her why the rule for4 worked. She asked the other students if they could come up with areason, and several possible reasons were proposed. Which of thefollowing statements comes closest to explaining the reason for thedivisibility rule for 4? (Mark ONE answer.)

a)Four is an even number, and odd numbers are not divisible by evennumbers.

b)The number 100 is divisible by 4 (and also 1000, 10,000, etc.).

c)Every other even number is divisible by 4, for example, 24 and 28 butnot 26.

d)It only works when the sum of the last two digits is an even number.


19/34



19

Student A Student B Student C

x

3

2

5

5 x

3

2

5

5 x

3

2

5

5

+

1

7

2

5

5

+

1

7

7

0

5

0 1

2

5

5

0

875

+

1

6

0

0

0

0

8 75

875

Which of these students is using a method that

could be used to multiply any two whole numbers?

Appraising Unusual Student Solutions


20/34



20

Common vs. Specialized CK

Appears in exploratory factor analyses on2/7 forms; confirmatory on 3/7

Individuals can be strong in common butnot specialized; vice versa

Support from cognitive interviews ofmathematicians

Suggests there isprofessional knowledgefor teaching


21/34



21

Ongoing Work

Item and measures development

Middle school national probability study

Develop new measurement modules for dataanalysis and for probability

Validation efforts

Videotape study

Cognitive tracing studies Content validity checks


22/34

Some Sample Results


23/34



23

An Example:Establishing a Relationship to Student

Growth


24/34



24

Links to Study of Instructional ImprovementStudent Achievement Analysis

SII CKT-M measure38 items SII: .89 IRT reliability

Model: Student Terra Nova gains predicted by: Student descriptors (family SES, absence rate)

Teacher characteristics (math methods/content,content knowledge)

Teacher content knowledge significant Small effect (LT 1/10 standard deviation)

But student SES is also on same order ofmagnitude

Hill, H.C., Rowan, B., & Ball, D.L. (2005) Effects of teachers'mathematical knowledge for teaching on student achievement.AmericanEducational Research Journal 42, 371-406.


25/34



25

A Second Example:Evaluating Teacher Professional

Development


26/34



26

Tracking Teacher Growth

Items piloted in Californias MathematicsProfessional Development Institutes(MPDI)

Instructors: Mathematicians and mathematicseducators

40-120 hours of professional development

Focus is squarely on mathematics content

Summer 2001

Pre/post assessment format (parallel forms)Hill, H. C. & Ball, D. L. (2004) Learning mathematics for teaching: Resultsfrom Californias Mathematics Professional Development Institutes. Journalof Research in Mathematics Education 35, 330-351.


27/34



27

MPDI Teacher Growth (Year 1)

For all institutes forwhich we have data,teachers gained .48logits, or roughly standard deviation

Translates to 2-3 itemincrease onassessment

Consideredsubstantial gain

0

0.2

0.4

0.6

0.8

1

1.2

All institutes

Pre-test

Post-test


28/34



28

Results from Sample Institutes

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

MPDI I MPDI II MPDI III MPDI IV MPDI V

Pre

Post


29/34



29

MPDI Evaluation: Other Findings

Length of institute predicts teacher gains 120-hour institutes most effective, on average

But some 40-hour institutes very effective

Focus on mathematical analysis, proof, and

communication leads to higher gains Many questions remain

Effects of content (e.g., mathematics vs. studentthinking)

Treatment of content: common vs. specialized

Effects of teacher motivation

Long term learning from colleagues, curriculum,practice


30/34

LMT/SII Measures and Dissemination


31/34



31

Current Item Pool

Equated forms for elementary school:

Number & operations / Content knowledge

(K-6)

Number & operations/ Knowledge of contentand students (K-6)

Patterns Functions & Algebra/ Content

knowledge (K-6) Geometry (3-8)


32/34



32

Current Item Pool

Equated forms for middle school:

Number & operations / Content knowledge

(5-9)

Patterns Functions & Algebra/ ContentKnowledge (5-9)

Geometry (3-8)


33/34



33

Item Workshops andDissemination

Interested users attend a one-day workshop in

Ann Arbor

We cover

History of item development Analytic methods and validation studies

How to use technical materials

Users get

Access to measures

Support materials


34/34


http://sitemaker umich edu/lm 34

Dates and Contact Information

Learning Mathematics for Teaching http://sitemaker.umich.edu/lmt

Dates for LMT Workshops

May 19, 2006 August 10, 2006

Brenda Ely ([email protected])

Geoffrey Phelps [email protected]

734-615-6076

developing and using measures

Documents