myways exercise 3: assessment design as integrated as your definition of student success
TRANSCRIPT
1
Exercise 3
Assessment Design as Integrated as Your Definition of Student Success
developed by Dave Lash & Grace Belfiore
for Next Generation Learning ChallengesAndrew Calkins, Deputy Director
October 30, 2015 DRAFT
MyWays is supported by a grantfrom the William and Flora Hewlett Foundation
with additional support fromthe Bill & Melinda Gates Foundation.
© 2015 EDUCAUSEThis work is licensed under a Creative CommonsAttribution-NonCommercial-NoDerivs 3.0 License.http://creativecommons.org/licenses/by-nc-nd/3.0/
Putting MyWays to Work
2
This is the third installment in a series of MyWays exercise packets.
Exercise 3: Assessment Design as Integrated as Your Definitionof Student Success
Exercise 3a: Mapping Assessments Across the Competency ArenasExercise 3b: How Well Do Your Projects Integrate a Variety of Assessments?
Quick link to the MyWays Beta ToolboxRepository of all MyWays overviews, tools, and exercises.
For a detailed listing of documents, see the last slide in this deck.
Putting MyWays to Work
Putting MyWays to Work
The MyWays model, tools, and reports provide auseful framework for design thinking & model building.
The MyWays model provides school designers, teachers, parents, and students with a synthesis of 20 student competencies needed for success in college, career, and life. In addition, it offers a set of simple, visual tools for mapping progress towards those competencies. The tools can be used to support strategic assessment, system development, and system implementation at the level of individual students, specific learning experiences, or overall learning model.
MyWays draws on research across the broad “student success” landscape to provide a composite framework applicable to all students regardless of academic aptitude or socioeconomic circumstance, including those students who must overcome the extraordinary challenges of intergenerational poverty.
3
The MyWays project is attempting to help grantees and school designers answer three big questions:
How well are we defining and articulating what success looks like for students attending our school?
How well does our design for learning and the organization of our school directly support students' attainment of that richer, deeper definition of success?
How do we gauge students' progress in developing those competencies? And: How can we measure and articulate our school’s overall performance, beyond proficiency in ELA and math?
The exercises in this packet, 3a and 3b, address the third of these three big questions—how well do our assessment activities support and gauge student progress on broader competencies?
1
2
3
4Putting MyWays to Work
Recap: The MyWays model provides school designers, teachers, parents, and students with a set of 20 student competencies needed for success in college, career, and life.
5Putting MyWays to Work
Please see Putting MyWays To Work Exercise 1 and the Introducing MyWays
overview for further information.
Situated learning (Lave and Wenger, 1991) broadly describes many forms of formal and informal learning that involve novices learning from experts in authentic, real world settings—a key ingredient we believe for acquiring the competencies in the MyWays model.
Combining situated learning with other learning science, David Perkins at Project Zero has developed seven principles for whole game learning—that is, maintaining the essence of the authentic activity while creating conditions that support novice advancement. The resulting learning experiences Perkins calls “junior versions.”
Please see Exercise 2 and the Whole Game Learning overview for further information.
6
Recap: Applying the seven principles of Whole Game Learning enables development of the range of competencies and of agency within meaningful, holistic learning experiences.
Putting MyWays to Work
7
What should next gen assessment for broader and deeper competencies look like?The ability to “measure what we value” in a world that has shifted is a mission-critical challenge to next generation educators. There is not yet a finished blueprint for next generation assessment; no glossy catalog of proven assessment tools and methods; no formula for replacing low-cost, narrow accountability testing with thoughtful investments in gauging the progress of the whole learner. On the other hand, there are pockets of extensive research and some developing practice in measuring learning and competency in specific settings and skill areas.
The Next Gen Assessment overview that accompanies this slide deck provides a brief snapshot of the evolution taking place in assessment; discusses two key paradigm shifts needed for the assessment of broader and deeper competencies; and summarizes five recommended assessment strategies that we believe are essential to success. Some, like formative assessment and performance assessment, are already familiar to many educators although their use in measuring competency arenas like Habits of Success and Wayfinding Abilities are likely less familiar. The other strategies—multiple measures, badges and micro-credentials, and quality reviews of learning experiences—are important in handling both these emerging arenas and wider learning settings
The goal of the overview, slides, and exercises in this Exercise 3 packet is to help you build capacity to apply these strategies within your faculty and staff.
Putting MyWays to Work
Truly, assessment can be a powerful force for knowing our students… We simply have to move past the baggage that comes with the term assessment, and understand that it can mean a lot of things. We can assess for content and skills, yes, but we can also assess for passions, interests, success skills, and the like for the purposes of the right instruction at the right time.”
Andrew MillerEdutopia blog
8
The state of assessment today across the broader competency range is fragmented, uneven, and fails to support the goals of next generation learning. Some of the shortcomings in current assessment practices have to do with the narrow range of what is measured, but much of the challenge arises from the nature of the assessments commonly carried out, even for traditional competencies, due to the ongoing preoccupation with accountability. For some aspects of agency, as well as some Habits of Success and Wayfinding competencies, there is little consensus on valid, reliable, context-sensitive measures. See the Next Gen Assessment overview for much more on this topic.
Putting MyWays to Work
Assessment today varies widely across the four competency arenas with growing attention on Creative Know How and Habits of Success.
9
Two key paradigm shifts are needed for an effective, nuanced assessment of broader and deeper competencies.
To measure broader and deeper competencies comprised of both capability and agency will require new approaches to assessment. Not only must we begin to assess hard-to-measure competencies like creativity, social skills, and wayfinding abilities; we must also gauge how well students “own” these competencies and apply them in real-life settings.
Next generation assessment systems that can address this challenge are some ways in the future. The crucial first step for next generation educators, we believe, is to adopt these two paradigm shifts in assessment practice.
Putting MyWays to Work
StandardizedStudent Assessments Tasks in the Outside World
Knowledge is: Measured within a subjectApplied across disciplines, along with other skills, to solve real world problems, create products, and generate new knowledge.
Asked to address:
Facts and application of simple procedures to well‐defined problems
Complex, disorderly problems in real world contexts.
Work is done: Individually Individually and in groups of others with complementary skills to accomplish a shared goal.
Resources available:
Without access to outside information, and use only paper and pencil
The challenge is to evaluate information from a wide range of tools and resources to find what’s relevant to analyze problems and create solutions
10
The Shift to Greater Authenticity requires moving from poor proxies to measures of competence in tasks similar to those performed in the outside world. See the Assessment Overview for further information.
Putting MyWays to Work
Adapted for the Stupski Foundation from Transforming Education: Teaching and Assessing 21st Century Skills (Cisco, Intel, Microsoft), 2010
A substantial majority of the competencies identified in the MyWays model require a combination of thinking skills and real world abilities. Not surprisingly, in the arenas of Creative Know How, Habits of Success, and Wayfinding Abilities, in particular, “textbook learning” is insufficient.
This “field of learning” is a useful visual device for envisioning both learning and assessment activities in terms of the thinking skills and real life abilities they engender. For example, the Mayan Community Project discussed in this packet is at the high end of the thinking skills axis while spanning both simulated and bounded authentic settings.
The following two slides map traditional assessments and a broader, deeper student experience, helping highlight those learning activities that encompass both higher-order thinking skills and real world abilities.
Bloom’s Thinking SkillsThe “left field” axis uses Bloom’s taxonomy to key a familiar progression of thinking skills. While technically the taxonomy is not a hierarchy, in our usage here, the skills are cumulative as one moves out the axis: Applying, for example, includes Remembering and Understanding while Creating includes all five of the “earlier” skills.
Real World AbilitiesThe “right field” axis indicates growing competence as the authenticity of the setting increases. For example, a particular middle schooler might be competent in Self-Direction or Communication & Collaboration within “simulated authentic” settings, such as those within school, but not in “complex authentic” settings in the adult world. This progression also allows specific learning activities to be plotted by degree of authenticity.
11
Recap: Mapping learning and assessment activities along two crucial axes
For mapping of learning experiences on the same field, see Exercise 1.
Putting MyWays to Work
Traditional assessments, especially most state testing, focuses on performance on non-authentic measures like multiple choice questions. Better state tests and AP include essays that enable students to construct responses at higher orders of thinking, but do not incorporate more complex, authentic contexts or settings.
Putting MyWays to Work 12
NOTE: A set of field of learning slides is available for your use including an empty grid for your own plots.
Assessment of broader & deeper competencies is only fully possible through measurement embedded in whole game learning approaches involving rich simulation, extended projects, or immersion in real world settings – contexts that increase the development of student agency, capability, and adaptability.
Putting MyWays to Work 13
The Shift to Multiple and Varied Measures requires moving from single, narrow assessments to multiple forms of measurement that are more varied, more developmentally-nuanced, and better integrated.
While one can envision basic mastery of multiplication tables being confirmed by simple quizzes or tests, assessing broader and deeper competencies like creativity, social skills, and wayfinding abilities requires a more multifaceted approach.
Because multiple forms of measures for any given competency are not the norm in traditional school models, we turn to state driving requirements for new drivers as a concrete and familiar example of such a system. At first blush, one might associate the road test as the qualifier for getting one’s license. However, over the past century, states have evolved systems of multiple, mandatory requirements to ensure the safety of drivers, passengers, and the public.
Putting MyWays to Work 14
How do we assess important life skills?
The “driving test”
Putting MyWays to Work 15
Learning to drive is one important real life skill, for which a mature system of learning, assessment, and certification has been developed over time.
Think about one example, the Massachusetts driving test. To the left we list the common components of the test, and on the following slide, we analyze what kinds of assessment experience are embedded in each component, and how they come together as an integrated system.
MA Requirements Assessment involved Written test of road rules – to get permit
30 hours of classroom instruction, with test at end
Computer simulations
6 hours of official driving observation
12 hours of official driving practice
40 hours additional practice, usually with parents
2 hour parent education class
The Road Test
How do we assess important life skills? The “driving test” as a system of multiple, varied assessments embedded in a learning experience
The MA “driving test” is actually a series of assessment-embedded learning experiences with multiple forms of measurement, addressing content knowledge, application, and creative know how, through authentic performance.
MA Requirements Assessment involved Written test of road rules – to get permit
Multiple choice, fact based; summative gateway to learner’s permit
30 hours of classroom instruction, with test at end
Formative feedback; scenarios for understanding of skills, consequences ; summative test on simple analytical
Computer simulations Incorporating application of knowledge and skills
6 hours of official driving observation
Introduction to the authentic learning environment; group/peer learning
12 hours of official driving practice
Practice loops in authentic environment with instant instructor feedback
40 hours additional practice, usually with parents
Practice loops in varying circumstances - different adult, different car - confirming transfer
2 hour parent education class
No assessment. Requirement is “programmatic”/seat time
The Road Test Performance-based assessment in complex, authentic environment
Putting MyWays to Work 16
Similarly, a “next generation” system of learning assessments will need to integrate the use of five key strategies.
Integration of these strategies will align assessment with holistic learning, and help gauge and guide learner progress towards the new, broader goal-line: Formative assessmentTo provide the essential foundations for effective learning & personal developmentPerformance assessmentTo provide the rich context for development and measurement of Agency as well as CapabilityMultiple & varied measuresTo address the whole learner and the breadth of competencies within Next Gen leaning environmentsBadges & micro-credentialsTo integrate “anywhere, anytime learning” within personalized learning approachesQuality reviewsTo ensure the quality of the learner experience when outcomes can’t be measured (and even when they can)
Putting MyWays to Work 17
Going deeper with thefive assessment strategies For each of the five strategies, the Next Gen Assessment overview includes a one-page primer, like the one to the right. These primers are not intended to be comprehensive nor to provide in-depth analysis of the technical merits of each strategy, but rather to get design teams thinking and discussing the level of variety, nuance, and integration needed to develop assessments for broader and deeper competencies.
Each includes brief notes on:
f-- Why the strategy is important-- Examples of this type of measurement-- Comments “through the MyWays lens”-- A few resources as food for thought
The overview also includes a one page summary chart of all five assessment strategies for reference. Deeper exploration of the strategies will be provided in forthcoming practice briefings.
Putting MyWays to Work 18
19
It is important to stress that the use of all five assessment strategies extends across all the MyWays arenas & competencies. What will not work, we are certain, is to try to isolate each of the 20 competencies and create a (likely inauthentic) way to assess each one! Indeed, educators would not even want to take each of the four MyWays arenas and chose a different assessment strategy to address each of those.
That being said, the Next Gen Assessment overview provides a summary of considerations for assessing in each arena, like the one to the left, that summarizes:-- The current state of assessment-- Main approaches currently being used & new approaches being tried-- Challenges of particular relevance
Putting MyWays to Work For those charged with designing learning and its environment, issues relating to assessment approaches vary somewhat across the four competency arenas.
20
As Angela Duckworth and David Yeager urge, at the end of Measurement Matters:
“Given the advantages, limitations, and medium-term potential of such measures, our hope is that the broader educational community proceeds forward with both alacrity and caution, and with equal parts optimism and humility.”
The Next Gen Assessment overview provides a glimpse at the ways the field is moving forward, covering:-- Cautions-- Promising developments-- Ways to work together, including: - Improvement communities like Carnegie’s SAIC and NGLC’s ALP - Partnerships between schools and assessment experts - Adaptation of approaches from other sectors
Putting MyWays to Work
The field of next gen assessment is still emerging. Careful thought is warranted about the use of various assessment measures, especially for agency and Habits of Success; nevertheless, given the importance of these broader competencies, it seems equally important to avoid “analysis paralysis” and forge ahead in collaborative and thoughtful ways.
Exercise 3a:How well is your school currently employing the five strategies that support next generation assessment?
These exercises are designed to help you use MyWays and the five assessment strategies to address the third big question:
How do we gauge students' progress in developing those competencies? And: How can we measure and articulate our school’s overall performance, beyond proficiency in ELA and math?
Review the Next Gen Assessment overview before tackling the exercises.
21Exercise 3a and 3bAssessment Design as Integrated as Your Definition of Student Success
Exercise 3b:How well is your school using elements of the five strategies to assess student progress within your learning projects?
Use the tools on the next two slides as checklists to aid you in evaluating and improving assessment experiences with respect to:
• The shifts to greater authenticity & multiple and varied measures
• The five assessment strategies most relevant to promoting the expanded success definition
• The 20 MyWays competencies
The goal is to equip your assessment design team with a reliable process for critiquing emerging assessment approaches—strengthening the extent to which you integrate your assessment with your learning, move towards more authentic tasks, and increase the range of assessment approaches in order to address broader and deeper competencies. Even at a quick, conceptual level, these tools can flag key issues and “help change the conversation” within your team with respect to transforming assessment as a force for teachers to better know their students and how to guide them, as well as for learners to get to understand themselves more fully.
22
Using the tools in tandem to develop assessment designas integrated as your definition of student success
Recapping the earlier exercisesExercises 3a and 3b build on the exercises in packets 1 and 2.
Exercises 1a and 1b explored the MyWays competency model, first comparing your school’s student success definition to MyWays and then using the MyWays Competency Plotting Tool to generate a spider plot of an individual’s current strengths, needs, and goals across the full set of MyWays competencies.
Exercises 2a, b, and c utilized the whole game learning principles to explore how well your learning projects reflect those principles, harness the benefits of junior versions, and map to the MyWays competencies.
Of course, your competency objectives, learning design, and assessment design all work together—and the seven MyWays tools presented in these exercises can be used in multiple ways and combinations.
Exercise 3a and 3b — and a case study demonstration
23Exercise 3a Employing the five strategies that support next gen assessment
Exercise 3a.How well is your school currently employing the five strategies that support next gen assessment? Assemble a team to evaluate the assessment strategies you use in each of the four competency arenas.
Read and discuss the Next Gen Assessment overview to more fully understand the two shifts and five assessment strategies. It is important for group members to be working from the strategy descriptions provided, rather than from individual interpretations of the strategy titles.
Download the Assessment-Competency Correlation Tool from the MyWays Beta Toolbox.
Working as a group, list the various assessment methods your school is using currently for each arena. Capture especially methods designed to assess student performance in the outside world (authenticity); but also note that not all strategies need be employed in each arena. Discuss and describe strengths and problem areas worthy of attention.
Discuss the completed worksheet. Come up with a joint analysis of areas of strength and needed improvement. Generate an action plan based on that analysis.
Use the MyWays competency arenas to analyze your overall assessment system.
Note: you can also do this analysis at the more detailed level of each competency. Detailed worksheets for each arena are provided.
Objectives: Gaining familiarity with the five strategies; visualizing your assessment strategies across your learning model; identifying areas for strengthening or augmenting
Enlarged on next slide
24Exercise 3a Mapping Assessments Across the Competency Arenas
25Exercise 3b Using the five strategies to assess student progress
Exercise 3b.How well is your school using elements of the five strategies to assess student progress within your learning projects? This exercise in analyzing how multiple assessment strategies are used within a specific learning project is best understood by reviewing the demonstration using the Mayan Community Project.
Assemble a team to evaluate one of your existing learning experiences. We suggest selecting a multi-faceted experience (a junior version) that runs a semester or more.
Read and discuss the Next Gen Assessment overview, including descriptions of the five strategies (important to work from shared understanding of them) and Mayan demo.
Download the Assessment Strategy Analysis Tool from the MyWays Beta Toolbox.
Working as a team, describe how each of the five assessment strategies is used (or not used) to assess student progress within that learning project. Record strengths and weaknesses within each assessment strategy. (See the Mayan demo for guidance.)
Discuss and come up with a joint analysis and action plan based on that analysis.
26A case study demonstration of exercises 2a, b, and c
Introducing the Mayan Community Project
The Next Gen Assessment overview demonstrates the use of the Assessment Strategy Analysis Tool in performing an evaluation of assessment practice in the same project as the one used for demonstration of the learning design tools introduced earlier in Exercise 2.
In this slide deck, we have excerpted a brief synopsis of the Mayan project itself along with just one section of the completed worksheet for 3b.
We encourage you to look for the full demo, including extensive assessment strategy worksheet, in Next Gen Assessment—as well as to connect all of this to the Mayan project learning design demo in the Whole Game Learning overview. While we have addressed learning and assessment separately within this series, the two are, rightly, deeply integrated in the design and implementation of the Mayan project.
A case study demonstration of exercise 3b
Summary of the Mayan Community Project An extended, interdisciplinary project with individual & group research
on the Mayan culture and Mayan areas of present-day Guatemala Application of knowledge to collaborative writing & illustration
of a children’ alphabet book on the Mayan culture Publication, marketing and sales of copies of the book to fund
schooling for seven Guatemalan students from impoverished families
• Essential questions: Why is it important to learn about the Mayan civilization today? How are books published and marketed? What is life currently like for people of Mayan descent?
• Learning goals include: knowledge of Mayan culture, the reality of poverty in Central America, skills in writing and editing for publication, actual experience of job roles in the publishing process, business planning, marketing & sales, and the empowerment of “how to make a difference in a child’s life”!
• Process is in-depth and over time: 12-week project, with approximately 2 hours class time/day, group and individual research, 2-3 revisions of book pages, student choice of research topics & job roles.
• Authentic, culminating experiences and assessments: Peer editors wielding the “Changes Needed” or “Approved” stamps, “Book Signing” (exhibition), and book selling activities in and with the local community.
27A case study demonstration of exercise 3b
28A case study demonstration of exercise 3b
This is the first of five sections of the assessment strategy analysis for the Mayan Community Project.
You can find the full analysis in the Next Gen Assessment overview.
29
All files associated with these exercises can be found at the MyWays Beta Toolbox.
Related readingThis slide deckNext Gen Assessment overview
ToolsAssessment-Competency Correlation Tool (for Exercise 3a)Assessment Strategy Analysis Tool (for Exercise 3b)MyWays Field of Learning slides
Forthcoming ReportsThe MyWays Model: A Composite of Student Competencies for Success in College, Career, and LifeFoundations of the MyWays Model: A Brief Summary of Student Success ResearchMyWays by Design: A Set of Practice Briefings on Learning and Assessment for Broader and Deeper CompetenciesSimple Tools for Using MyWays in Your School Community
Resources for Exercises 3a & 3bAssessment Design as Integrated as Your Definition of Student Success