improving and assessing instructional effectiveness

28
Improving and Assessing Instructional Effectiveness Theodore Frick Department of Instructional Systems Technology School of Education Indiana University Bloomington Invited Lecture to IUB Medical Sciences Faculty October 14, 2002

Upload: jeremy-castro

Post on 31-Dec-2015

37 views

Category:

Documents


3 download

DESCRIPTION

Improving and Assessing Instructional Effectiveness. Theodore Frick Department of Instructional Systems Technology School of Education Indiana University Bloomington Invited Lecture to IUB Medical Sciences Faculty October 14, 2002. Overview. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Improving and Assessing Instructional Effectiveness

Improving and Assessing Instructional Effectiveness

Theodore FrickDepartment of Instructional Systems Technology

School of EducationIndiana University Bloomington

Invited Lecture to IUB Medical Sciences FacultyOctober 14, 2002

Page 2: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 2

Overview

Confounding of instructional effectiveness by other factors

Types of knowledge about educationType 2, 3 and 4 examplesRecommendations

Page 3: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 3

The Logic of Effectiveness

Before instruction After instruction Conclusion

1. Student nonmastery Student mastery Instruction appears to be effective enough, but …

2. Student nonmastery Student nonmastery Instruction apparently was not effective enough

3. Student mastery Student mastery Can't tell if instruction is effective, since student had achieved the goal before instruction began

4. Student mastery Student nonmastery ??? Something wrong with the content - factual errors, incorrect procedures, bad models ??? Bad luck ???

Page 4: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 4

Confounding Factors

1. Learning occurs without intentional instruction.

2. Learning occurs despite poor instruction.

3. We’re not the only teachers a student has.

4. Results of instruction may occur LONG afterwards.

Page 5: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 5

Conflation of Research Methods and Outcomes

In addition to the confounding problem in determining effectiveness of instruction, there has been a lot of debate in education about inquiry methods – e.g., quantitative vs. qualitative.

Continuing problem of bridging research and practice in education

Page 6: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 6

Consider this Debate

Imagine for a moment three carpenters arguing about which tool is best. 

Quanta: "Hammers and nails are clearly superior." Qualia: "I disagree.  Screwdrivers and screws are

much more effective." Performa: "You're both wrong.  Saws are best for

cutting wood." Quanta: "Who said anything about cutting wood?  I

thought we were talking about fastening wood together."

Qualia: "Right. Who needs saws?" Performa: "I do. I need to cut this board in half."

Page 7: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 7

Premise

The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful.

Page 8: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 8

Types of Educology: Knowledge about Education

Methodology of Theory Building (Elizabeth Steiner, 1988) (link)

Non-axiological knowledge – what isAxiological knowledge – pertaining to values:

Instrumental value: good for – what is effective Intrinsic value: good in itself – what is worthwhile

The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles (Lee Cronbach, et al., 1972)

Page 9: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 9

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

See Frick: R690 Syllabus for further details

Page 10: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 10

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Example: Type 1 Knowledge Claim

‘Elizabeth Steiner is an educational philosopher.’

Page 11: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 11

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Example: Type 2 Knowledge Claim ‘Students who spend more time engaged successfully in arithmetic and reading tasks score more highly on achievement tests in those content areas.’ (Fisher, et al., 1976)

Page 12: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 12

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Example: Type 3 Knowledge Claim ‘The DISTAR program for teaching arithmetic and reading to elementary students works.’

Page 13: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 13

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6Example: Type 4 Knowledge Claim ‘Instruction is most effective when it is problem-based, activates prior learner knowledge, demonstrates what is to be learned, provides opportunities for guided practice, and encourages integration with everyday life.’ (Merrill, 2001)

Page 14: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 14

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Example: Type 5 Knowledge Claim

‘The corporal punishment policy in Houston schools is a bad policy.’

Page 15: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 15

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Example: Type 6 Knowledge Claim

‘Teachers and students should respect each other.’

Page 16: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 16

Do not confuse ends and means

The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful. Outcome of research is knowledge. Inquiry methods are a means to that end. Criteria for evaluating adequacy of research

methodology are NOT the same for each knowledge type – e.g.,

criteria for statistical inference to a broad population do not apply to Types 1, 3, and 5;

effectiveness not of concern in Types 1, 2, 5, and 6, but main concern in 3 and 4.

Page 17: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 17

For today…

Focus on a few examples of Types 2, 3 and 4 knowledge of educationType 2: Academic learning time researchType 3 outcomes: Program/product

evaluation – Kirkpatrick’s 4 levels (1959)Type 4 process: Effective Web Instruction –

Frick & Boling (2002)Type 4: Principles of 5-Star Instruction –

David Merrill (2001)

Page 18: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 18

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Type 2 Example: Relationship of academic learning time and student achievement

Page 19: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 19

Type 2: Academic Learning Time

Allocated Time

Engaged Time

Task Success

ALT is positively correlated with academic achievement in the same content areas (Fisher, et al., 1976; Rieth & Frick, 1983; Berliner; 1985).

Page 20: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 20

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Type 3: Evaluation of a particular instructional program or product

Page 21: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 21

Type 3: Program/Product Evaluation – Kirkpatrick’s Levels

Techniques for Evaluating Training Programs -- Donald Kirkpatrick (1959)

1. Reaction (satisfaction)

2. Learning (achievement)

3. Behavior (transfer to real context)

4. Results (impact on organization/context)

More on Kirkpatrick’s levels from Encyclopedia of Educational Technology

Page 22: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 22

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Type 4 Example: how to create effective instructional products – design theory

Page 23: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 23

Type 4: Effective Web Instruction: An Inquiry-Based Process – Frick & Boling

More

Page 24: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 24

Type 4: First Principles of Instruction – David Merrill

Problem

Activation

DemonstrationApplication

Integration

Page 25: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 25

Type 4: First Principles of Instruction (cont’d)

“Learning is facilitated when:1. Learners are engaged in solving real-world

problems.

2. Existing knowledge is activated as a foundation for new knowledge.

3. New knowledge is demonstrated to the learner.

4. New knowledge is applied by the learner.

5. New knowledge is integrated into the learner’s world.” (Merrill, 2001, p. 2)

Does your instruction rate 5 stars? A rating scale

Page 26: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 26

Six Types of Knowledge (Frick)

Purpose of Inquiry

Scope of KnowledgeUnique Generalizable

What is? 1 2

What is effective? 3 4

What is worthwhile? 5 6

Type 3: Indiana University physician education program

Page 27: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 27

Recommendations

In physician education, focus on Type 3 knowledge:Apply Merrill’s 5 principles as criteria for

design. See video (requires RealPlayer).

Use Kirkpatrick’s 4 levels of evaluation for your programs at IU.

Page 28: Improving and Assessing Instructional Effectiveness

October 14, 2002Improving and Assessing Instructional

Effectiveness -- Ted Frick 28

Questions?

This presentation is available at:

http://www.indiana.edu/~tedfrick/improveinstruction.ppt

Contact: [email protected]