strengthen your program with common departmental performance assessments charlotte gifford...
TRANSCRIPT
Strengthen Your Program with Common DepartmentalPerformance Assessments
Charlotte Gifford
Greenfield (MA) Community College
ACTFLSan Antonio, TXFriday, November 21, 2014
Agenda: preparing for a pilot project
• Definitions for our purposes– Backward Design– Performance
• Institutional assessment– Leverage of required assessment work
• Steps for an effective pilot project:the cycle of assessment, analysis and application– Assessment design and development– Analysis of student work– Application of findings
• Our experience: lessons learned over three cycles
Backward Design
An invaluable organizing principle:
“Backward design, also called backward planning, is a pedagogical approach to unit or lesson planning in which the teacher first identifies the desired end task or product, then works in reverse from the assessment task(s) to identify the prerequisite learning tasks.”
--Annenberg Teaching Foreign Language Workshop
http://www.learner.org/channel/workshops/tfl/glossary.html
Backward Design
1. Curricular goals:Identify desired results– What should my students know and be able to do?
2. Performance assessments:Determine acceptable evidence– How can I find out what my students know?
3. Classroom practices:Plan learning experiences and instruction– What will it look like in the classroom?– What are the best practices to get the results I
want?
Wiggins & McTighe: Understanding by Design
Performance assessment
Determine acceptable evidence = performance
“During a performance assessment, students demonstrate their ability to use the target language in activities that parallel what native speakers might do. For example, students might create a newspaper, respond to a want ad, or conduct an interview to learn about a cultural topic. These assessments are best evaluated using clearly developed rubrics, although grades can be assigned in a more traditional way.”
--Annenberg Teaching Foreign Language Workshop
http://www.learner.org/channel/workshops/tfl/glossary.html
Performance assessment
A theater analogy:real-world situations within the classroomproviding a metaphorical stage on which to• rehearse,• role-play,• improvise, and eventually • perform.
This produces the evidence we need.
Performance assessment
Alternate analogies:Music: from practice through performanceSports: from training through performance
Whatever the analogy, we must convince our learners that we are helping them get ready for performance.
Building a Performance-based Model
What does it take?A department-wide paradigm shift–Application of Backward Design
principles1. Sequenced, realistic and attainable
learning goals2. Performance assessments that address
those goals3. Classroom experiences that aim for
performance
curriculargoals
performance assessment
teaching & learning
Institutional Assessment
Leverage of required assessment work:• Program reviews– Linked to work on Backward Design
• NEASC accreditation documentation– E1A forms (Standard 4)
• Student Learning Outcomes– General Education Abilities (tied to
LEAP)–World Language Program SLO’s (tied to
Gen Eds)
Our Project Cycle
1. Identify performance objectives to be assessed
2. Develop a common performance assessment to measure the chosen objective(s)
3. Choose, adapt or develop an appropriate rubric
4. Administer the assessment; collect student work
5. Analyze the artifacts against the rubric6. Apply the findings to curriculum redesign to
improve teaching and learning
Your Pilot Project
1. Identify performance objectives to be assessed
If you have existing performance objectives:– What do you want to measure?
Unit level? Midterm? Final performance?
If you don’t yet have performance objectives:– Examine the NCCSFL/ACTFL Can-Do
Statements
Your Pilot Project
2. Develop a common performance assessment to measure the chosen objective(s)
– What will students do to show what they know and are able to do?
– Make sure all instructors understand the key distinction:Goals vs. Tools
– Choose the communicative mode(s)– Prepare transparent instructions for learners
No assessment surprises!– Be prepared to revise the assessment for future use
Your Pilot Project
3. Choose, adapt or develop an appropriate rubric
– Determine your criteria– Develop clear prose descriptors to distinguish
between levels of performance (not just numbers on a scale)
– Share with students– Show them how to excel: make it a roadmap for
success– Use rubrics for summative, formative and self-
evaluation purposes
OVERALL RATING: Communication and comprehension:
Level of expression: Task completion according to instructions
on assignment sheet:
Delivery, flow and pronunciation:
Accuracy:
Exceeds expectations (20-18)excellent
Fully comprehensible; no areas of confusion. Fully understands questions and responds relevantly, adding to the conversation.
4
Language expressed by simple sentences and some strings of sentences with some elaboration.
4
Exceeds required elements;thorough exploration of task/topic
4
Clear delivery and natural flow with rare hesitations. Pronunciation does not significantly interfere with communication.
4
No significant errors, control of grammatical structures and vocabulary studied.
4
Meets expectations (17-15)strong (14-12)weak
Almost fully comprehensible; minor areas of confusion. Understands most questions and responds relevantly.
3
Language expressed primarily by simple sentences.
3
All required elements present;on task/ topic
3
Generally clear delivery and flow with only occasional hesitation. Pronunciation occasionally interferes with communication.
3
Some minor to moderate errors in grammatical structures and vocabulary studied.
3Minimally meets expectations (11-9)too limitedat risk
Partially comprehensible; some significant areas of confusion. Understands some questions or parts of questions; answers show partial understanding.
2
Language expressed primarily by short phrases and/or memorized chunks/clusters of language.
2
Some required elements present, incomplete;partially on task/topic
2
Pauses frequently, searching for words. Mispronunciation interferes with communication.
2
Significant to serious errors in grammatical structures and vocabulary studied. Some patterns of error
2
Does not meet expectations (8-5)insufficient
Barely comprehensible; significant difficulties in comprehension. Does not understand questions; answers are incomprehensible.
1
Language expressed primarily by single words.
1
Almost no required elements present;off task/topic
1
Halting speech, constantly interrupted by pauses. Barely comprehensible due to mispronunciation.
1
Constant errors in grammatical structures and vocabulary studied. Patterns of error.
1
Unratable / no attempt to meet expectations
No attempt to convey message.
0
No attempt to use target language.
0
No required elements present.
0
No attempt to convey message orally.
0
No attempt to speak in target language.
0
Comments:
Uses English to fill gaps in communication: subtract one point.
Sample rubric
Your Pilot Project
4. Administer the assessment; collect student work
– Ensure group commitment up front– Issue timely reminders– Schedule time for in-depth discussion (step 5)
Your Pilot Project
5. Analyze the artifacts against the rubric, determining:
– What makes for a performance that exceeds expectations?
– What is missing from a performance that does not meet the standard?
– Identify key student behaviors/actions for each case
Your Pilot Project
6. Apply the findings to curriculum redesign
– Determine action steps and stratgies– Refine the common assessment for future use– Refine the rubric and its application: build in
incentives for and recognition of desired aspects of performances
– Adapt the curriculum: make changes to teaching and learning experiences to address deficits in student performances
Three passes through the cycle
Background:• Existing shared and common assessments• Previous informal comparisons of student work at
the unit level• Analysis of quite specific, targeted performance
objectives
Three passes through the cycle
Year 1 – Program-level assessment projects:• Analysis of program-level SLO - #5: Communicate through
interpersonal, presentational and interpretive modes• Two samples, presentational mode
– Midterm oral presentation: autobiography– Final exam, written expression: a letter to fictional exchange student
• Review of embedded performance objectives for assignments• Instructors selected 3 performances: strong/acceptable/weak• Analyzed the artifacts against the rubric, determining:
– What makes for a performance that exceeds expectations?– What is missing from a performance that does not meet the standard?– Identify key student behaviors/actions for each case
Three passes through the cycle
Year 1 – Program-level assessment projects:• Strongest performances:
– exceeded expectations in all but structural accuracy– included extensive elaboration and expansion of the topic(s)– students were able to recombine and reform learned patterns to
express personal meaning: creating with language (moving toward Intermediate)
– accuracy was very strong, considering the errors that usually accompany this kind of stretching into the next level of proficiency
• Weakest performances:– showed either reluctance or inability to stretch in this way– performances were incomplete and underdeveloped. – in the final exam, language used was recycled from pre-midterm
memorized chunks without attempts at expansion or elaboration. – level of expression and text type: overly simple, list-like and
repetitive, in spite of explicit instruction in this area.– patterns of error persisted in even basic forms and expressions.
Three passes through the cycle
Year 1 – Curriculum redesign:• Clarify benchmark moments when the bar is
raised higher• Midterm: that was then, this is now:
underdeveloped language - no longer “meets expectations”addressing repeated errors in rudimentary forms
• Two-level rubric: a higher standard for “exceeds expectations”
• Shared set of brainstormed Best Practices for class application
Three passes through the cycle
Year 2 – Program-level assessment project A:• Replication and extension of the project in Year 1• Two samples, interpersonal and presentational
modes– Final oral interview:
role play with the housing coordinator in the fictional exchange
– Midterm project, second semester: presentation on vacations, past and future
• Confirmation of Year 1 key findings for raising expectations:– The role of the midterm project– The two-level rubric– All applications in the Best Practices continue
Three passes through the cycle
Year 2 – Program-level assessment project B:• Analysis of program-level SLO - #1 Compare
students’ own and other cultures, thus understanding both better
• The balancing act: analyzing cultural perspectives with novicesTarget language or English? Philosophically, a 90%+ department
Three passes through the cycle
Year 2 – Program-level assessment project B:• Spanish curriculum in place: study of the film Mi familia
– Linked to performance objectives: in target languagefamily relationshipsbiographical informationlikes and dislikesvalues and beliefs: Le importa …
– Discussion of characters’ identities and of cultural and generational differences in perspectives: in English
• New for this project: a reflection paper on one character– An additional in-class assignment, non-graded– A 4 point single-criterial rubric: understanding of
perspectives/valuesEmerging / Approaching / Meeting / Exceeding (adapted, Weber State U)
– End of semester crunch: too little time, not enough preparatory discussion
Three passes through the cycle
Year 3 – Program-level assessment project B:• Replication of the project in Year 2• Replication of the end of semester time crunch!• Development of new preparatory discussion activities
– scheduled earlier in the semester– drawn from the Coverdell Schools/Peace Corps teaching
materials,“Building Bridges”
– addressing the perspectives content of the reflection paper to come
• Current assessment redesign for Year 4 (this year!)– a clarified writing prompt– reflection paper to be written outside of class– required and graded
Three passes through the cycle
Year 3 – Program-level assessment project A:• Replication and extension of the project in Years 1
& 2Revealed weakness in interpersonal communication
• Another set of revisions to the oral rubric:– Criteria change: from
“Communication of message” to“Communication and Comprehension”
– Adaptable for presentational and interpersonal modes
OVERALL RATING: Communication and comprehension:
Level of expression: Task completion according to instructions
on assignment sheet:
Delivery, flow and pronunciation:
Accuracy:
Exceeds expectations (20-18)excellent
Fully comprehensible; no areas of confusion. Fully understands questions and responds relevantly, adding to the conversation.
4
Language expressed by simple sentences and some strings of sentences with some elaboration.
4
Exceeds required elements;thorough exploration of task/topic
4
Clear delivery and natural flow with rare hesitations. Pronunciation does not significantly interfere with communication.
4
No significant errors, control of grammatical structures and vocabulary studied.
4
Meets expectations (17-15)strong (14-12)weak
Almost fully comprehensible; minor areas of confusion. Understands most questions and responds relevantly.
3
Language expressed primarily by simple sentences.
3
All required elements present;on task/ topic
3
Generally clear delivery and flow with only occasional hesitation. Pronunciation occasionally interferes with communication.
3
Some minor to moderate errors in grammatical structures and vocabulary studied.
3Minimally meets expectations (11-9)too limitedat risk
Partially comprehensible; some significant areas of confusion. Understands some questions or parts of questions; answers show partial understanding.
2
Language expressed primarily by short phrases and/or memorized chunks/clusters of language.
2
Some required elements present, incomplete;partially on task/topic
2
Pauses frequently, searching for words. Mispronunciation interferes with communication.
2
Significant to serious errors in grammatical structures and vocabulary studied. Some patterns of error
2
Does not meet expectations (8-5)insufficient
Barely comprehensible; significant difficulties in comprehension. Does not understand questions; answers are incomprehensible.
1
Language expressed primarily by single words.
1
Almost no required elements present;off task/topic
1
Halting speech, constantly interrupted by pauses. Barely comprehensible due to mispronunciation.
1
Constant errors in grammatical structures and vocabulary studied. Patterns of error.
1
Unratable / no attempt to meet expectations
No attempt to convey message.
0
No attempt to use target language.
0
No required elements present.
0
No attempt to convey message orally.
0
No attempt to speak in target language.
0
Comments:
Uses English to fill gaps in communication: subtract one point.
Sample rubric
Three passes through the cycle
Year 3 – Program-level assessment project A:• Replication and extension of the project in Years 1
& 2Revealed weakness in interpersonal communication
• Another set of revisions to the oral rubric:– Criteria change: from
“Communication of message” to“Communication and Comprehension”
– Adaptable for presentational and interpersonal modes
• All applications in the Best Practices continue– Emphasis on not settling for simple and safe– Rewards for stretching, expanding, elaborating
Results and benefits of our projects
• A unified department, built on consensus
• Shared expertise and resources:an expanding bank of departmental assessments(prompts, situations and authentic texts)
• Invaluable for the integration of new members
• Positive leverage of institutional assessment
• Closing the assessment loop: immediate application to improving student learning is the highest priority
• Most valuable: the conversations, discussions and collegial debates about teaching and learning
Your Pilot Project Cycle
1. Identify performance objectives to be assessed2. Develop a common performance assessment to
measure the chosen objective(s) 3. Choose, adapt or develop an appropriate rubric4. Administer the assessment; collect student
work5. Analyze the artifacts against the rubric6. Apply the findings to curriculum redesign to
improve teaching and learning
Repeat the cycle!