1

Language, Learning, and Assessment: Improving

Validity

Eva L. Baker

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

26th Annual Language Testing Research ColloquiumOf the International Language Testing Association

Temecula, CA

March 26, 2004

2

Today’s Focus

Language and identity—a personal note

Quick assessment policy tour

Need for cognitive model-based assessment to link learning to external outcome measures

Examples

Research agenda involving language and assessment interplay

3

Assessment Policy Tour©

1965 F

an

tasy R

ecord

s

4

Assessment Policy Tour (Cont’d)

Accountability is pushing testing practice

Soon to have greater role in postsecondary

Assessment and testing high priority in K-12, language certification, and adult and professional learning

Students’ language background a major area

5

K-12 Accountability Requirements Goals (standards) to be linked to instruction,

classroom assessments

External examinations are to be lined up with goals and instruction

Few examples of alignment, unless goals are restricted

Lack procedures and measures to document level of practice

Early growth likely not sustainable

Transfer of learning a major issue

6

Description:Extra comfort for senior dogs. Our popular orthopedic pet bed, made extra thick for aging dogs. A full 4" of medical grade convoluted foam supports bones and joints, and the elevated headrest provides proper neck and spine alignment.

http://www.petdiscounters.com/dog/beds/cu_orthopedic.html

7

http://www.powerofyoga.com/

Joyful manifestation of the heart’s desire.

8

Cognitive Model-Based Assessment

Cross-curricular cognitive families define core constructs

Embed in content, like linguistic rules in AI language understanding systems

Derives from work in writing assessment and AI systems

Evidence that it works across subject matters, types of learning, and students

9

Intellectual Capital Cognitive Families

ContentUnderstanding

ProblemSolving

Teamwork andCollaboration

MetacognitionLearning to LearnCommunication

Learning

10

Cognitive Demands Examples

Factual knowledge Definitions, propositions (e.g., statements about events,

causes, relations; “math facts”; formulas), etc. Terms and symbols (beyond arithmetic)

Identify domain by letter: A for algebra, G for geometry, etc.

Conceptual knowledge Knowledge of “more complex, organized knowledge

forms,” including classification schemes, principles and generalizations, theories, models, and knowledge structures

Procedural knowledge Knowledge about how to do something, including

mathematical skills and algorithms, techniques and methods

Knowledge Categories

11

Cognitive Demands Examples (Cont.)

Recall information

Apply information

Interpret figures, charts, or tables

Transform representations Change or move from one representation to another

(e.g., language to diagram, text to equation, graph to equation)

Multi-step reasoning Requires “students to generate an intermediate image,

construct, or sub-problem before solving the original problem”

Process Categories

12

From Science to Models to Templates

DOMAIN-INDEPENDENT

COGNITIVEPRINCIPLES

CONTENT CONTENT CONTENT

CBA

TEMPLATE TEMPLATE TEMPLATE

MODEL

SCIENTIFICFINDINGS

COGNITIVE DEMANDS

SCIENTIFICFINDINGS

SUBJECT MATTERSPECIFIC MODELS

CONDITIONSTOPICS OF TASKSSPECIFIC SCORING FEATURES

13

From Templates to Tasks

CBA

TEMPLATE TEMPLATE TEMPLATE

TASK TASKTASK

TASKTASK

TASK

TASK

TASKTASK

TASK

TASK TASK

14

Templates for Model of Content Understanding

Primary source materials in each domain

Student required to integrate prior knowledge and principles to succeed

Scored by using expert model in subject matter

15

Content UnderstandingTemplate #1 Explanation

An array of primary source materials

A prompt that asks for an explanation in context

Constructed (written) answer

Evaluated by means of a scoring rubric

16

Excerpts from U.S. HistoryPrimary Source Documents

Judge Douglas made two points upon my recent speech at Springfield. Hesays they are to be the issues of this campaign. The first one of these pointshe bases upon the language in a speech which I delivered at Springfield, whichI believe I can quote correctly from memory. I said there that “we are now farinto the fifth year since a policy was instituted for the avowed object, and withthe confident promise, of putting an end to slavery agitation; under theoperation of that policy, that agitation had not only not ceased, but hadconstantly augmented.” “I believe it will not cease until a crisis shall havebeen reached and passed. ‘A house divided against itself cannot stand.’ Ibelieve this Government cannot endure permanently, half slave and half free.” “I do not expect the Union to be dissolved”—I am quoting from my speech—“I do not expect the house to fall, but I do expect it will cease to be divided.It will become all one thing or the other. Either the opponents of slavery willarrest the spread of it and place it where the public mind shall rest, in the belief that it is in the course of ultimate extinction, or its advocates will pushit forward until it shall become alike lawful in all the States, North as well asSouth.” [text continues]

ABRAHAM LINCOLN

17

U.S. History WritingAssignment: Civil War Era

Be sure to show the relationships among your ideas and facts.

Your essay should be based on two major sources:

1. The general concepts and specific facts you know about United States history, and especially what you know about the history of the Civil War.

2. What you have learned from the readings yesterday.

Imagine that it is 1858 and you are an educated citizen living in Illinois.Because you are interested in politics and always keep yourself well-informed, you make a special trip to hear Abraham Lincoln and Stephen A.Douglas debating during their campaigns for the Senate seat representingIllinois. After the debates you return home, where your cousin asks youabout some of the problems that are facing the nation at this time.

Write an essay in which you explain the most important ideas and issuesyour cousin should understand.

18

Excerpts from Hawaiian HistoryPrimary Source Documents

LILIUOKALANI

For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.

Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.

And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.

19

Content Knowledge Prompt:Hawaiian History Writing

Assignment—BayonetConstitution

Be sure to show the relationships among your ideas and facts.

Your essay should be based on two major sources:

1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.

2. What you have learned from the readings yesterday.

Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.

Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.

20

Content Knowledge Prompt (Cont’d)

*From Hawaii’s Story by Hawaii’s Queen, Liliuokalani (Boston: Lee and Shepard Publishers, 1898).

It may be asked, “Why did the king give them his signature?” I answerwithout hesitation, because he had discovered traitors among his mosttrusted friends and because the conspirators were ripe for revolution, andhad taken measures to have him assassinated if he refused.

It has been known ever since that day as “The Bayonet Constitution,” and the name is well-chosen; for the cruel treatment received by the king from the military companies. [text continues]

Explain to your friend who missed class the reasons and differences for the Queen and the Senator’s approach to Hawaii’s future.

Scoring Rubric •General impression (on task)•Principles and themes•Prior knowledge•Relevant concrete examples•Avoidance of misconceptions

21

History Template

22

Benefits

Architecture for design of classroom instruction, assessments, and external tests, not multipurpose test, but multipurpose architecture

Transfer of learning

Flexibility of teaching

Vertical coherence

Specific procedures

Evidence base

23

Measuring Transfer of Learning

Not teach x, measure y: but describe variations in conditions, task content, criteria that permit students to demonstrate their flexibility

Learning to learn measurement

Metacognition and self-regulation

Time to learn new domains or tasks

Safeguard against teaching to the test as a preferred intervention

24

How Language Testing Influences Achievement Testing

Rubrics and procedures for assessing open ended performance

Linguistic analysis as accommodations

Linguistic descriptors as key task dimension

Arguments about language, content, and context

25

Toward Cognitively Oriented Constructs for English Language

Development Exams (K-12) Ironies

Expedience and science

Social and academic constructs

Measure of beginning skill or transfer

Relationships to other external exams not clear

Characterizing text Readability

Lexiles

LEARNOME

26

Towards Common Language Across Domains

Genome: “Complete set of instructions”

LEARNOME: “Complete map of features in learning domain”

27

Preliminary Exploration of the LEARNOME

Fundamental and generalizable research on identification of components

Cognitive demands, e.g., problem solving

Linguistic requirements, i.e., discourse, syntax, lexicon

Task, e.g., context, constraints

Content (ontologies)

Descriptive language—granularity

Modeling and representation of results

Empirical verification

28

Research Interactions: Language Testing and Assessment

Model-based R&D

EFF Adult Literacy

Questions about differential validity and fairness

29

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

30

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

31

EFF Assessment Consortium: SRI International & Center for LiteracyStudies. (2004, January). EFF standards and performance levelDescriptors for: Reading, writing, math, speaking, andListening. Washington, DC: National Institute for Literacy.

32

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

33

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

34

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

35

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

36

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

37

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

38

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

39

EFF A

ssessm

en

t Con

sortiu

m: S

RI In

tern

atio

nal &

Cen

ter fo

r Litera

cy

Stu

die

s. (20

04

, Jan

uary

). EFF sta

nd

ard

s an

d p

erfo

rman

ce le

vel

descrip

tors fo

r: Read

ing

, writin

g, m

ath

, speakin

g, a

nd

Listen

ing

. Wash

ing

ton

, DC

: Natio

nal In

stitute

for Lite

racy

.

40

Research Interactions: Language Testing and Assessment (Cont’d)

Technology

TBALL project—NSF education, linguistics, electrical engineering—automated testing of ELL children

Essay scoring—towards propositional rather than regression solutions

Authoring systems with smarts

Simulation architecture and narrative structures

41

42

Research Interactions: Language Testing and Assessment (Cont’d)

Validity inferences for different groups

Comparability among tasks, paths

Alternative to norms for comparisons among groups

43

Priorities

Linking context, purposes, and evidence to an appropriate grain-sized framework

Document growth patterns associated with different instructional paths

Explore validity interpretations for ELL groups

44

CRESST Web Site

http://www.cresst.org