1 teacher performance assessment consortium november 10, 2010 andrea whittaker and nicole merino...

1

Teacher Performance Assessment Consortium

Teacher Performance Assessment Consortium

November 10, 2010Andrea Whittaker and Nicole

MerinoStanford University

1

© Stanford Center for Assessment, Learning, and Equity, 2010

Three Sponsoring Organizations

Three Sponsoring Organizations

• AACTE – overall project management, communication with programs

• Stanford University – assessment development and technical support

• Council of Chief State School Officers – policy development and support, communication with state education agencies

2


• Sign off from state education agencies

• Participation of preparation programs

• Up to 3 funded through grant

• Alignment and policy studies

• Adaptation of TPA to address future state

program and policy needs

3

TPAC State Participation Requirements

TPAC State Participation Requirements


Lineage of TPACLineage of TPAC

• National Board for Professional Teaching Standards (NBPTS) portfolio assessments – accomplished teachers

• Connecticut BEST assessment – teachers at end of induction

• Performance Assessment for California Teachers (PACT) – pre-service teachers

source: SCALE @ Stanford University 5


APPLE CriteriaThe National Board for Professional Teaching

Standards

APPLE CriteriaThe National Board for Professional Teaching

Standards

Administratively feasible

Publicly credible

Professionally acceptable

Legally defensible

Economically affordable

6


• Discipline Specific

• Integrated Assessment

• Student Centered

• Analytic Feedback and Support

• Represents a Complex View of Teaching: Multiple Measures 7

Design Principles: Educative Assessment

Design Principles: Educative Assessment

888

The TPAC Assessment System

The TPAC Assessment System

Assessments Embedded in Local Programs

— examples —

Assessments Embedded in Local Programs

— examples —

Observation/Supervisory Evaluation & Feedback

Observation/Supervisory Evaluation & Feedback

Child Case Studies

Child Case Studies

Analyses of Student Learning

Analyses of Student Learning

Curriculum /Teaching Analyses

Curriculum /Teaching Analyses

The Capstone Assessment

The Capstone Assessment

TPAC AssessmentDemonstrates :

• Planning Instruction

and Assessment

• Engaging Students

and Supporting

Learning

•Assessing Student

Learning

‣Reflecting

‣Academic Language

TPAC AssessmentDemonstrates :

• Planning Instruction

and Assessment

• Engaging Students

and Supporting

Learning

•Assessing Student

Learning

‣Reflecting

‣Academic Language


TPAC Assessment Records of Practice*TPAC Assessment Records of Practice*

Instructional and Social ContextLearning Segment of 3 to 5 Hours/Days

Planning•Lesson Plans•Handouts, overheads, student work•Lesson Commentary

Engagement•Video clip(s)•Teaching Commentary

Assessment•Analysis of Whole Class Assessment •Analysis of learning of 2 students•Feedback to 2 students•Next steps in instruction

• Evidence of Academic Language Development• Daily Reflections and Reflective Commentaries

9


Guiding Questions and Analytic Rubrics


• PLANNING‣ Planning Focused,

Sequenced Instruction

‣ Using Knowledge of Students to Inform Teaching

‣ Planning Assessments to Monitor and Support Student Learning

• ENGAGING‣ Engaging Students in

Learning

‣ Deepening Student Learning During Instruction

10

• ASSESSMENT‣ Analyzing Student Work

‣ Using Assessment to Inform Instruction

‣ Using Feedback to Guide Further Learning

• REFLECTION‣ Monitoring Student

Progress and Adjusting Instruction

• ACADEMIC LANGUAGE‣ Understanding Language

Demands and Resources

‣ Developing Students’ Academic Language Repertoire

10


Rubric LevelsRubric Levels• Is the candidate ready for

independent teaching (i.e., to be the teacher of record)?

• Rubric Levels

‣Level 1 – Some skill but needs more practice to be teacher-of-record

‣Level 2 – Acceptable level to begin teaching

‣Level 3 – Solid foundation of knowledge and skills

‣Level 4 – Stellar candidate, in the top 5% of candidates


Questions and AnswersQuestions and Answers

???

12


PACT ScoresInter-rater Reliability

PACT ScoresInter-rater Reliability

Level of Agreement Percent

Exact Match 46%

± 1 point 34%

± 2 points or greater 10%

Sample Size · 2,580Spearman-Brown Reliability Estimate · 0.88

13


• Content validity

‣ Development teams, Program directors, Program faculty, & Leadership team

‣ TPE alignment study

• Concurrent validity

‣ Evaluation of score validity

‣ Decision Consistency · Holistic vs. Analytic ratings

• Bias and fairness review

PACT Validity StudiesPACT Validity Studies

14

• Construct validity

‣ Factor Analysis(2002-03 Pilot Year):

• Reflection & Assessment

• Instruction

• Planning

• Predictive Validity (Carnegie/CT Study)

• Consequential Validity


Research PlansResearch Plans

• Value-added analysis

• Observations and interviews of candidates completing assessment + first year of teaching

• Surveys and interviews with faculty and program administrators on use of assessment data

15


Assessment DevelopmentAssessment

Development• 2009-10 Small-scale tryout tasks &

feedback from users

• 2010-11 Development of pilot prototypes based on feedback and piloting in at least 2 credential areas per institution. User feedback to guide revisions

• 2011-12 National field test of prototypes, producing a technical report with reliability and validity studies and a bias and sensitivity review. National standard setting.

• 2012-13 Adoption of validated assessment

16


Overview of Tasks and Rubrics

Overview of Tasks and Rubrics

• Artifacts and Commentaries

‣ What looks familiar?

• Level Three Rubric Descriptors

‣ What gets assessed?

• Tips for implementation

17


Handbook OverviewHandbook OverviewWhat looks like something we already do?

What looks new or different?

What will our candidates be good at now?

What might be a struggle?

18


Context MattersContext Matters• Candidates describe what they know about

their students as learners:

‣ Academic development

‣ Academic Language development

‣ Social-emotional development

‣ Family, community and cultural assets

‣ Special needs

• Candidates use information about students to plan appropriate instruction and assessment.

• Scoring rubrics examine extent to which needs of students are addressed

19




• PLANNING‣ Planning Focused,

Sequenced Instruction

‣ Using Knowledge of Students to Inform Teaching

‣ Planning Assessments to Monitor and Support Student Learning

• ENGAGING‣ Engaging Students in

Learning

‣ Deepening Student Learning During Instruction

20

• ASSESSMENT‣ Analyzing Student Work

‣ Using Assessment to Inform Instruction

‣ Using Feedback to Guide Further Learning

• REFLECTION‣ Monitoring Student

Progress and Adjusting Instruction

• ACADEMIC LANGUAGE‣ Understanding Language

Demands and Resources

‣ Developing Students’ Academic Language Repertoire

20


EM1: Planning Focused, Sequenced Instruction

EM1: Planning Focused, Sequenced Instruction

• Standards/objectives, learning tasks, and assessments are clearly aligned to a big idea or essential question. The learning tasks and assessments represent differing depths of understanding

• Candidate plans how to make clear connections among mathematical facts, computations/procedures, concepts, and reasoning.

• Learning tasks build on each other to promote an understanding of the designated mathematical concepts, computations/procedures, and reasoning skills. Learning tasks (or their adaptation) are justified by explaining their appropriateness for students with references to relevant research and/or theory.

21


EM2: Using Knowledge of Students to Inform

Teaching

EM2: Using Knowledge of Students to Inform

Teaching• Learning tasks and materials draw upon

students’ academic and social/emotional development, including strengths, as well as experiences and interests to help students reach the learning objectives.

• Planned support consists of strategically selected or modified tasks/materials and/or scaffolding of instruction that is closely tied to specific learning objectives. It is appropriate for specific individuals or subgroups.

22


EM3: Planning Assessments that Monitor and Support Student

Learning

EM3: Planning Assessments that Monitor and Support Student

Learning

• The set of assessments will provide evidence of student learning relative to the standards/objectives for each lesson. At least one lesson’s assessments provide evidence of student learning that extends beyond the formulaic application of computations or procedures.

• Assessments are aligned to clearly defined benchmarks or criteria for student performance.

• Assessments are modified or adapted to be appropriate for students having difficulty demonstrating their learning.

23


EM4: Engaging Students in Learning

EM4: Engaging Students in Learning

• Strategies for intellectual engagement seen in the clip(s) offer structured opportunities for students to develop their own understanding of mathematical concepts through discourse. These strategies reflect attention to students’ academic or language development, social/emotional development, and/or cultural and lived experiences.

• Candidate identifies successful and unsuccessful teaching practices. The proposed improvements are reasonable and address the learning of a subgroup or individual students.

24


EM5: Deepening Student Learning During Instruction

EM5: Deepening Student Learning During Instruction

• Candidates and/or other students build on what students are saying and/or doing, using reasoning to improve understanding of mathematical concepts.

• Candidate and/or other students prompt students to make connections between and among mathematical concepts and representations of content.

25


EM6: Analyzing Student Work

EM6: Analyzing Student Work

• Criteria are well-defined and reflect the depth of understanding stated in of the indicated standards/objectives from the learning segment.

• The analysis focuses on patterns of student errors, skills, and understandings in relation to standards and learning objectives. The analysis uses these patterns to understand student thinking.

• The analysis is supported by work samples and the summary of learning. Specific patterns are identified for individuals or subgroup(s) in addition to the whole class. 2

6


EM7: Using Assessment to Inform Instruction

EM7: Using Assessment to Inform Instruction

• Next steps follow from an accurate analysis of student learning and aim at improving student understanding of important features of the standards/learning objectives.

• Next steps focus on improving student performance through targeted support to individuals and groups to address specific identified needs.

• Next steps are based on whole class patterns of performance and some patterns for individuals and/or subgroups and are described in sufficient detail to understand them.

27


EM8: Using Feedback to Guide Learning

EM8: Using Feedback to Guide Learning

• Specific and accurate feedback helps the student understand what s/he did well, and provides guidance for improvement.

• Candidate describes how students will use feedback to improve their work or their understanding.

28


EM9: Monitoring Student Progress and Adjusting

Instruction

EM9: Monitoring Student Progress and Adjusting

Instruction

• Daily reflections identify what students could or could not do within each lesson and consider the implications for meeting the standards/objectives at the end of the learning segment.

• Adjustments to instruction are appropriate and focused on addressing some individual and collective learning needs.

29


EM10: Understanding Language Demands and

Resources

EM10: Understanding Language Demands and

Resources• Candidate describes academic language

strengths and needs of students at different levels of academic language proficiency.

• The language genre(s) discussed are clearly related to the academic purpose of the learning segment and language demands are identified. One or more linguistic features and/or textual resources of the genre are explicitly identified.

• Candidate identifies essential vocabulary for students to actively engage in specific language tasks.

30


EM11: Developing Students’ Academic Language Repertoire

EM11: Developing Students’ Academic Language Repertoire

• The candidate’s use of scaffolding or other support provides access to core content while also providing explicit models, opportunities for practice, and feedback for students to develop further language proficiency for selected genres and key linguistic features.

• Candidate articulates why the instructional strategies chosen are likely to support specific aspects of students’ language development for different levels of language proficiency.

31


Potentially New Competencies

Potentially New Competencies

• Academic language development

• Interpreting student work and identifying trends across the class

• Designing next steps in instruction based on assessment results

• Providing feedback to students 32


Issues to Think about for Spring 2011 PilotIssues to Think about for Spring 2011 Pilot

• Match with curriculum

• Which faculty? Which credential areas?

• Candidate support

• Timeline, from introducing assessment through scoring

• Preparation for videotaping

• Recruiting scorers

• Consideration of results33


What did successful PACT programs do? What did successful PACT programs do?

• Broad and collegial involvement in decision making

• Inquiry and program improvement rather than compliance

• Affirm and maintain program values and identity

• Broad involvement in assessment activities

• Regular practices and protocols for data analysis

• Learned from individual “case” studies

34


Questions and AnswersQuestions and Answers

???

35


Academic Language Overview

Academic Language Overview

Three “F” Words…

36


Academic LanguageAcademic Language

Academic language development is making the language explicit to expand students’ control over language and improve their language choices according to the purpose/FUNCTION and audience for the message.

Academic language also offers structures/FORMS for developing as well as expressing explanations, evaluations, and analyses.

Developing students’ FLUENCY in academic language forms and functions provides access to the “language of school” and academic success

37



The purposes of Academic Language are to clearly and explicitly define, classify, analyze, explain, argue, interpret and evaluate ideas for distant audiences.

38



The purposes of Academic Language are to clearly and explicitly define, classify, analyze, explain, argue, interpret and evaluate ideas for distant audiences.

= FUNCTION39


Academic LanguageAcademic Language• Language for academic purposes differs

greatly from every day purposes

• These differences include

‣ a better-defined system of FORMS (genres) with explicit expectations about how texts are organized to achieve academic purposes

‣ precisely-defined vocabulary to express abstract concepts and complex ideas

‣ more complex grammar in order to pack more information into each sentence

‣ a greater variety of conjunctions and connective words and phrases to create coherence among multiple ideas

‣ formatting conventions, graphics and organizational titles and headings to guide understanding of texts

40


Academic LanguageAcademic LanguageAcademic Language also includes instructional language needed to participate in learning and assessment tasks, including discussing ideas and asking questions, summarizing instructional and disciplinary texts, following and giving instructions, listening to a mini-lesson, explaining thinking aloud, giving reasons for a point of view, writing essays to display knowledge on tests.

41


Academic Language in TPAC

Academic Language in TPAC

• Vocabulary

• Genre/forms with particular functions

▪ Linguistic features (grammar,

organization)

▪ Textual resources (headers, table of

contents, illustrations, graphs, charts)

42


VocabularyVocabulary• Technical vocabulary: triangle, metaphor,

metabolize

• Words whose technical meaning is different than everyday language: “balance” in chemistry, “plane” in mathematics, “ruler” in history/social science, “force” in science

• Connector words: and, but, because, therefore, however

43


Genres/FormsGenres/Forms• Have a general structure – e.g., narratives,

explanations, arguments

• Structure of an explanation

- Description of what is being explained

- Statements of cause-effect relationships

- Sometimes ending with an interpretation

judgment

44


Subject-Specific Genres/FORMSSubject-Specific Genres/FORMS

• Representing word problems mathematically

• Procedures for a science experiment

• Literary interpretation

• Argument proposing causes of an historical event

45


Structures of Argument

Genre/FORM

Structures of Argument

Genre/FORM• Mental verbs used to express opinions:

like, believe

• Move from personal to impersonal voice

• Connectives used for logical relations and to link points

‣Temporal connectives: first, next

‣Causal conditional connectives: because

‣Comparative connectives to introduce counterpoints: consequently, therefore

46


Structure of Arguments Structure of Arguments (from Knapp & Watkins, 2005)

• Simple argument: point/proposition, elaboration

‣ “I like The Simpsons because they make me laugh.”

• Argument with evidence: Proposition, argument, conclusion

• Discussion: statement of issue, arguments for, arguments against, recommendation

• Elaborated discussion: statement of issue, preview of pro/con, several iterations of point/elaboration representing arguments against, several iterations of point/elaboration representing arguments for, summary, conclusion

47


Characteristics of Advanced Academic LanguageCharacteristics of Advanced Academic Language

• Abstractions: government, electron, linear equation, acid

• Nominalizations, verbs or adjectives becoming nouns to enable more dense text or more cohesive text:

organize into…→ this organization…

…were revealed. The trigger for this revelation was…

• More precise connector words and phrases, going beyond “and” or “but” to “in contrast” or “Given this, it follows that…”.

48


Academic Language Competencies Measured (Rubric

10)


10)• Understanding language demands and

resources for instructional emphasis

‣ Identification of linguistic features of a genre/form/function addressed within instruction

‣ Relation of vocabulary identified to content and to students’ academic language proficiencies

‣ Description of student language strengths and needs

49



11)


11)

• Expanding students’ academic language repertoires

‣ Making key linguistic features related to genre purpose (form and function) visible to students

‣ Modeling vocabulary and linguistic features and providing opportunities for practice (developing fluency)

‣ (at higher levels) appropriateness of models for students at different levels of language proficiency

50


• Analyze video for academic language using graphic organizer

http://www.learner.org/resources/series33.html

ActivityActivity

51




Academic Language Takeaways

Academic Language Takeaways

• Academic language is different from everyday language. Some students are not exposed to this language outside of school.

• Much of academic language is discipline-specific.

• Unless we make academic language explicit for learning, some students will be excluded from classroom discourse and future opportunities that depend on having acquired this language. 5

2


Upcoming WebinarUpcoming Webinar

Tuesday, November 16

1:30 Eastern Time

• To register contact Rachel Popham at

‣[email protected]

53

1 teacher performance assessment consortium november 10, 2010 andrea whittaker and nicole merino...

Documents

educative assessment

assessment engaging

stanford center

teachers national board

planning instruction

future state program

complex view of teaching

policy studies alignment