assessment playbook orientation
TRANSCRIPT
Assessment Playbook Orientation
February 24, March 3, March 10, March 17, 2021
Katy MattisDirector of Tools and Assets
Introduction • NCMPS Director of Tools and Assets• AMI trained in Elementary and Assistants to
Infancy• 20+ years in Montessori as Upper Elementary
Guide, District Peer Observer, Assistant Head of School, Principal
• 13 years at Denison Montessori School in Denver Public Schools (Upper Elementary Guide, Humanities Facilitator, Principal (6 years))
– 30+ year old magnet program– 400+ students (3 year old-6th grade)– Title 1 (60% FRL)– 25% ELL (Spanish/ Vietnamese)– 70% SoC (60% hispanic/ 7% Vietnamese)
NameSchool/ Roll
What brought you to this courseWhat you hope to get out of the course
Agenda
Week 1 (February 24)Introduction to the Montessori Assessment Playbook
Week 2 (March 3)Classroom Measures
Week 3 (March 10)Whole School/ Program Measures
Week 4 (March 17)Student Measures
ObservationSummative Assessment
By human flourishing we mean the capacity to thrive socially, emotionally, intellectually and economically; to participate meaningfully in family, community and civic life; and to live a life of curiosity, agency and satisfaction.
At NCMPS, our work is grounded in the premise that human flourishing, rather than achievement as defined by standardized tests, should be the goal of education.
When surveyed, Public Montessori Educators identified the current assessment culture as one of the top 3 significant challenges they face (funding and staffing= other 2)
Demoralized Montessori Educators sought ways to provide a full Montessori experience within a system that reduces their work to the number of students who can move towards proficiency on a test.
Through collaboration with practitioners in the field and and ongoing cycles of developing, testing, reflection and refinement, NCMPS developed 33 tools for assessment to be used with students, in classrooms and in the whole school.
It is possible to assess children, classrooms and schools in ways that are:• Constructive• Equitable• Rigorous• Advance the goal of Human Flourishing
Assessment is an essential component of fully implemented Montessori practice
Assessment as a System Rather than an Event
Build an assessment protocol around carefully coordinated tools that enable the system to operate with both precision and coherenceComprehensive system designed to support continuous improvement of:
• Students• Classrooms • Whole School
Measures what matters most to human flourishingStudents: Skills such as Executive Functions, Social Fluency, Emotional Flexibility and Academic Knowledge at the center for assessment protocols
Classrooms: Align goals when organizing environments, plan lessons and model behavior
Schoolwide: strategically selecting measures (tests, tasks, observation rubrics, surveys), reporting results in a way that emphasizes the school’s mission
The Montessori Ecosystem
33 TOOLS
Who Administers?
Aligning Inputs and Outcomes
Are our children, staff and families flourishing?How do we know?
Taken together, this collection of tools is designed to provide content to flesh out structures supporting a robust community of practice focused on continuous improvement. By attending vigorously to inputs at both the school and classroom level and by making the process of appraisal transparent and growth oriented, we aim to offer a pathway for meaningful staff assessment.
Assessing Inputs
Whole School Inputs
Classroom Inputs
Assessing Inputs
We must therefore create a favorable environment that will encourage the flowering of a child’s natural gifts.
-Maria Montessori
• DERS• Appraisal Instruments
– Teacher (Primary & Elementary)– Assistant (Primary & Elementary)– Executive Director
• Annual Review Portfolio (Executive Director)• Essential Elements Rubric (EER)• Reflective Practice Inventories
– Teacher– Executive Director– School Wide
• Annual Climate Survey
● Executive Functioning Skills○ Initiation &
Concentration○ Inhibitory Control○ Working Memory
These 5 domains should be the centerpiece of a comprehensive assessment system that is itself the centerpiece of school culture and operations.
Assessing Outcomes
● Social Fluency & Emotional Flexibility
● Linguistic & Cultural Fluency
Outputs
Assessing Outcomes
• Normalization Check (Primary)• Adjustment Check (Elementary)• Progress Report Templates
– Primary– Elementary
• New Family Questionnaire• Minnesota Executive Function Scale (MEFS)• Skills Inventories (Early Childhood, Lower Elementary, Upper Elementary)• Alumni Survey• Receiving Teacher Survey
Assessment, Accountability and Human Potential
When assessment systems are designed to align inputs and outcomes, every single element of what happens in every corner of the school matters. Every adult- teachers, assistants, receptionists, cooks, custodians, parents- matters. How environments- indoor as well as outdoor- are prepared matters. Leadership and governance matter. Culture matters.
Week 2: Classroom MeasuresMarch 3
Classroom Measures
DERS
What’s Going on in the Classroom?
Getting a Glimpse of the Classroom
Observing Wide-Scope Learning
Observing Work Engagement
Appraisal
DERS anchors the suite of classroom-level measures
Wide-Scope Developmental Outcomes...
...as indicated by developmentally-aligned child behavior...
...in environments designed for optimal human development.
Developmental Environment Rating Scale
Wide-scope developmental outcomesDevelopmentally-aligned child behaviorEnvironments designed for optimal human development
The DERS AppThe DERS can be administered at regular intervals, for use in school improvement, or to measure the effects of curriculum changes or staff development.
The assessment is administered in a 60-minute classroom observation.
Detailed instructions for administration of the DERS are presented in on-line training provided by NCMPS.
DERS App
The DERS is convenient and easy to administer. Observation is guided by the DERS app, and clear definitions of each target area are readily provided. Note-taking materials are not needed, as data are recorded and analyzed directly on the tablet.
DERS ReportDERS results are available immediately.
The app generates a report, which includes a numerical score summary of each domain, a narrative discussion of each domain, and a graphic display of all 60 items.
The app calculates, sums, and analyzes scores, eliminating the possibility of observer calculation errors.
Assessors can direct their full attention to classroom observation. When the observation is complete, the DERS report can be delivered automatically by email.
What’s Going on in the Classroom?Page 59
What’s going on? • The object of the question can be the child, the
environment, the lesson• Calls the observer to open ended reflection
versus “what’s wrong” or “what’s right”?
Observation informs all classroom decisions: • what lessons to offer • to whom lessons are offered• where to place materials and furniture in the
classroom• how to approach a child who is straining to
concentrate• how to respond to questions• when and when not to intervene
If you do not have the DERs App, the “What’s going on in the Classroom” form can be used to replicate a basic rendition of the DERS framework.
Getting a Glimpse of the ClassroomPage 33
• Same tool for all levels
• Observe for the What? Where? and With Whom elements of Children’s Activities
– Information about child work partner preference
– Time spent with peers
– Time spent on a single activity
– Are students using work as prop or engaged?
Observing Wide-Scope Learning (Primary & Elementary)
Pages 34 & 35
• Similar to Getting a Glimpse of the Classroom
• Best for observers new to Montessori. Focuses on specific look-fors
Observing Work Engagement (Primary, Elementary, Adolescent)
Pages 36, 37 & 38
• Primary and Elementary versions are similar• Adolescent version is different
• Focuses on patterns and themes in the classroom environment (not individual children)
– What areas of the room are in use? – Are students choosing work spontaneously or through adult
direction?– Is the room set up to accommodate a variety of dynamism in
where students work and with whom?• Two Minute Sampling:
– Once at the start and once at the end– The observer categorizes the behavior of each of the students.
• Rest of the observation:– Montessori materials worked with– Lessons Given– Areas of the classroom in which work is chosen or lessons are
given
Reflective Practice Inventory
Teacher Appraisal
Reflective Practice Inventory (Teacher, Executive Director, Schoolwide)
Pages 99 to 101
Reflective Practice should be the core of Professional Culture
Aligned to Appraisal Instruments and DERS
Appraisal Instruments (Primary & Elementary)
Pages 60 to 86Aligned to Reflective Practice Inventory & DERS
Assistant:
Work with Assistant
Work with Children
Care of the Environment
Teacher:
Assessment
Preparation
Invitation
Protection
• Self Evaluation Using Reflective Practice Inventory • Devise Annual Growth Plan• Observed several times per year using the tool• Coaching• Ongoing reflection • Annual review to launch the next improvement cycle
Executive Director Suite of ToolsPage 82-86, 87 & 100
Annual Review Portfolio
The Executive Director Annual Review Portfolio is a curated file of evidence and reflection on the performance of the school’s executive. The Portfolio is compiled by the executive and submitted along with the checklists/ cover sheet
Executive Director Appraisal Instrument
• Used 2 times per year• Leading Authentically• Leading Others• Leading the Future
Reflective Practice Inventory
Effective leaders must be systems thinkers with a service orientation, who have the knowledge, skills, and disposition to embody the school’s mission.
Week 3: Whole School MeasuresMarch 10
Whole School Measures
The Essential Elements Rubric anchors the suite of Whole School measures
Essential Elements Rubric
Annual Climate Survey
Alumni Questionnaire
Receiving Teacher Questionnaire
Essential Elements of Successful Montessori Schools
in the Public School SectorThe Essential Elements of Montessori were developed based on consensus work with several Montessori organizations.
Essential Elements Rubric
26 Standards Across Five Domains
Montessori Adults
• Montessori trained teachers — all classrooms• Assistants who have been Montessori-oriented• Between two and four teachers always in the
training pipeline• All adults — trained or not — embody core
Montessori principles of respect for development, children’s natural needs and tendencies, independence, grace & courtesy
Montessori Learning Environment
• Practice supported by Montessori Scope & Sequence
• Large, mixed-age groupings of children in all classrooms
• Specials integrated into work period (minimum interruption)
• Long, uninterrupted work periods (ideally three hours)
• Full complement of Montessori materials (limited, intentionally chosen)
• High degree of student choice on what, where, and how long to Access to the world outside the classroom
• Informed, but not driven, by state standards• Classrooms are inclusive of all learners and
learning styles — including SPED and ELL
Family Engagement
● Strong, meaningful partnership with all families characterized by invitation rather than summons
● Lots of organized opportunities to learn about Montessori, beginning prior to enrollment
● Family activity is focused on engagement in child development rather than the operation of the school
Leadership & Organizational Development
• Montessori expertise at the top• Structures for ensuring high-fidelity through
ongoing self-reflection and review• Membership in one of the national Montessori
professional organizations
Assessment
• Holistic protocols that integrate academic, EF’s and social/emotional learning
• Observation is core to all meaningful assessment activities — and drives all instruction
• Necessary assessments (e.g. state assessments) are delivered in the least intrusive way possible
• As student mature, they are increasingly involved in their own progress monitoring
Essential Elements Rubric Pages 88 to 98
Provides a clear, concrete, and detailed articulation of the program-level practices that encompass Montessori fidelity, while also capturing key institutional practices for organizational sustainability.
Montessori Fidelity• Adults• Environment
Organizational Sustainability• Leadership and Organizational Development • Family Engagement• Assessment
How to Conduct an Essentials Element Review
• External Audit• Self-Study• Supported Self-Study
EER anchors the suite of school-level input measures
Annual Climate SurveyPage 102
Part of a collection of tools (Appraisal instruments, Essential Elements Rubric, Reflective Practice Inventories, Annual Climate Survey) designed to flesh out structures supporting a robust community of practice focused on continuous improvement.
For staff and families
Elicits narrative feedback
Responses more complex than simple satisfaction ratings
Brief, by design
Alumni Survey & Receiving Teacher Survey
Page 136 and 137
● Completed by:○ students who leave the program○ teachers who receive those students
● Rate skills and attitudes not highlighted on other assessment frameworks
○ Curiosity○ Compassion○ Persistence○ Teamwork
● Overtime, results can be tracked, analyzed and displayed as part of the school’s overall assessment framework
Week 4: Student MeasuresMarch 17
Observation
Observation, it turns out, is a superior data source for determining how best to support children’s development: it’s the primary strategy Montessori used to develop and refine the pedagogy, and it’s the first and most important skill Montessori teachers learn when they take training.
Every time a teacher engages in observation, they are engaging in assessment.
Observation and Data-Informed Instruction
Really good Montessori teachers base all their decisions- what and to whom lessons are offered, where to place furniture and materials in the environment, how to approach a child who is straining to concentrate, how to respond to questions, when and when not to intervene- on observation.
• Observing Wide-Scope Learning– Primary– Elementary
• Observing Work Engagement – Primary– Elementary– Adolescent
• Individual Child Behavior Inventory– Primary– Elementary
• Observing an Individual Child & Graphing Work Curves
What’s going on in the classroom?– What’s going on?
• The object of the question can be the child, the environment, the lesson• Calls the observer to open ended reflection versus “what’s wrong” or “what’s right”?
– Observation informs all classroom decisions: • what lessons to offer • to whom lessons are offered• where to place materials and furniture in the classroom• how to approach a child who is straining to concentrate• how to respond to questions• when and when not to intervene
The Role of Observation (Why do it?)– Inform our understanding of
• what children know• what they can do
– Ensure rigor in continuous improvement,
– Ensure the environment is meeting the needs of the children,
– Help unconscious behaviors come to light, and
– Break the cycles of inequity in our classrooms.
Observing Wide-Scope Learning (Primary & Elementary)
Pages 34 & 35
• Similar to Getting a Glimpse of the Classroom
• Best for observers new to Montessori. Focuses on specific look-fors
Observing Work Engagement (Primary, Elementary, Adolescent)
Pages 36, 37 & 38
• Primary and Elementary versions are similar• Adolescent version is different
• Focuses on patterns and themes in the classroom environment (not individual children)
– What areas of the room are in use? – Are students choosing work spontaneously or through adult
direction?– Is the room set up to accommodate a variety of dynamism in
where students work and with whom?• Two Minute Sampling:
– Once at the start and once at the end– The observer categorizes the behavior of each of the students.
• Rest of the observation:– Montessori materials worked with– Lessons Given– Areas of the classroom in which work is chosen or lessons are
given
Individual Child Behavior Inventory (Primary & Elementary)
Pages 41 & 42
Primary & Elementary● Caring for the classroom● Conversation● Initiating work● Interrupting● Joy● Navigating the room with
care● Offering/receiving help
from peers● Persisting in the face of
challenge● Resolving needs with
words● Seeking adult approval or
permission● Shifting● Social graces● Waiting turn● Work as prop
Primary● Completing a work cycle● Disrupting● Engaging with purpose● Misusing materials● Handling materials with care● Observing without disturbing
others
Elementary● Collaborating● Comfort with adults● Disrupting● Frustration● Maximum effort● Planning and reflection
Observing an Individual Child & Graphing Work Curves
Pages 39 & 40
Before the graph can be created, the data must be collected.
Collect at least a week of daily observations
The x-axis tracks concentration over time. The y-axis captures the magnitude of engagement.
Work curve analysis is among the most intensive, research-like approaches to collecting and analyzing data
Work Curve ExamplesWork Curves are graphic displays of the ebb and flow of concentration over a period of time (usually a morning work period)
Illustrate how children move from disorder to concentration
The concept of work curves is crucial to understanding not only what children learn when they interact within the prepared environment but how they learn.
Gaining access to both the patterns that characterize an individual learner’s mode of concentration and the factors that may be impeding engagement constitutes a deep dive into the learning process itself.
Promote the Importance of Observation
Create a school culture that values observation
Each classroom should have an observer’s chair and the children should respect that sitting in it is work
Guides should Teach, Circulate, Observe
Guides should prioritize observation
Student Measures
Individual Child Behavior InventoryObserving an Individual ChildGraphing Work Curves
Early Childhood Family QuestionnaireNormalization Check/ Adjustment CheckSkills InventoryProgress ReportMinnesota Executive Function Scale (MEFS)
Early Childhood Family QuestionnairePages 121 to 124
● Completed by parents prior to the 1st day of school● Teachers review as part of the orientation process ● Teachers refer to it as questions arise regarding child’s interests and strengths● Similar to the Ages and Stages Questionnaire (ASQ) ● Similarities New Family Questionnaire and ASQ
○ Focus on developmental skills■ Walking■ Talking ■ Holding a fork■ Toileting
● Unlike ASQ, New Family Questionnaire invites parents to share their perceptions of their child’s
○ Strengths○ Challenges○ Interests○ Unique Qualities○ Home Routines
● Qualitative orientation paints a holistic picture of the child and the entire family
Normalization Check (Primary)Adjustment Check (Elementary)
Pages 115 and 116
● Designed to be internal tools● Rely on technical Montessori language● Used by school-based adults to understand
individual learners’ needs and design individualized instruction around those needs
● Completed by teachers/classroom teams 4-6 weeks into the school year.
● Primary: All children ● Elementary: All NEW children
Is this child developing the skills, habits and knowledge needed to thrive in an environment designed to nurture wide-scope outcomes like Executive Functioning Skills, Linguistic and Cultural Fluency, Social Fluency and Emotional Flexibility?
Skills Inventories (Early Childhood, Lower Elementary, Upper Elementary)
Pages 125 to 135The skills inventories are benchmarking documents that capture key outcomes that can be measured via observation or through specific tasks.
The Skills Inventories assess a child’s progress towards the expected outcomes of three years in a Montessori environment.
● Primary (3-6 years old)● Lower Elementary (6-9 years old)● Upper Elementary (9-12 years old)
They should be used as benchmarks and lesson planning guide during a child’s time in the environment, and as a summative assessment in the year before entering a new level.
● Primary to Lower Elementary ● Lower Elementary to Upper Elementary
● Upper Elementary to Adolescent Program
Progress Reports (Primary & Elementary)
Pages 117 to 120
Primary
Social, Emotional & Executive Development
Social Fluency & Emotional Flexibility
Executive Functions
Academic Development
Elementary
Social, Emotional & Executive Development
Social Fluency & Emotional Flexibility
Executive Functions
Critical & Creative Thinking
Academic Development
MEFs
• Minnesota Executive Function Scale • 3rd Party Vendor Reflection Science• MEFs is a Universal Screen• Used 2x per year to gauge growth in
Executive Functioning over time
Bridging Formative & Summative AssessmentObservation: The Critical Fulcrum
The Truth
» Instruction and assessment are inseparable
» Feedback is necessary for achievement
» Summative assessment is helpful in its proper context
The Perception of Standardized/ Summative Testing
NarrowMeasures a limited scope of skills.
“Waters down” teaching and learning.
DivisiveCategorizes, ranks and sorts students and schools based on cut scores, percentages and percentiles.
DisruptiveNot aligned with the Montessori measurement cycle.
Add external pressure to perform.
Value of Assessment
Montessori FIRST AlignmentContext
InformationDirection
What to Do with Test Results . . .
ContextualizeAnalyze the results in the context of the learning environment and in the context of the goals and objectives of the Montessori Method.
CelebrateAcknowledge and celebrate all achievements progress/growth, no matter how small.
Critique w/ CautionConsider the validity of areas in need of improvement and thoroughly discuss reasons for any shortcomings.
What about the students?
Making test-taking as non-disruptive as possible
» Don’t make it about results, make it about the process.
» The goal is to normalize the event and not make it an ordeal
» Acclimate the students to the experience and help them develop the stamina to prepare for what they will eventually face
» Meet the needs of the child » allow bathroom breaks» skip the test if having a bad day» test in tiny groups (5 or fewer)
Remember...» Testing is merely a tool.
» Observation is a more authentic form of assessment that evaluates the whole-child.
» The test data should reasonably match the observations.
» Deeper understanding and growth.
» Hands on experiential learning results in far deeper learning of transferable skills.
You have the tools
» Montessori Assessment Playbook
» Personal passion to facilitate human flourishing
» The understanding that test results are a PICTURE not a PUNISHMENT
Benchmarking Montessori
Our approach to benchmarking aims to eliminate the need for curriculum (inputs) alignments by clearly articulating expected outcomes.
Aligning and/or “crosswalking” Montessori curricular elements with CCSS or other State Frameworks degrades the integrity of the Montessori program resulting in negative outcomes program wide.
Montessori is larger, more complex, more integrated and nonlinear
There are many paths to attainment supported by multiple materials, lessons and extensions.
Benchmarking Project Elementary80 Topics
Primary: 40 Topics
● Art & Music● Earth and Space
Science● Economics● Geography● Geometry● History & Political
Geography● Language Arts● Life Science● Mathematics● Physical Science
● Geography & History
● Language Arts● Mathematics● Practical Life● Science● Sensorial
The Intersection of Assessment & the Human Potential
How well do we help children reach their full potential?
What obstacles do we remove from their human flourishing?
Do the children in our care retain and develop their curiosity, their drive, their love of learning?
● Assessment doesn’t have to look like this○ Current Framework is based on 2 fundamentals: efficiency &
simplicity○ Human Potential is a complex phenomenon that presents real
challenges to test developers aiming to create a useful instrument
● Assessment is an industry: Users are the market, and users wield market power
○ If we position ourselves as clients in needs of a better product, we can and should set the terms of our relationship with the vendor.
● Changing the Assessment Industry is a long game. School survival is a short game. We need to be playing both at the same time.
○ Maintain pedagogical integrity while fulfilling compliance expectations
○ Hold ourselves accountable in ways that advance the overarching goal of human flourishing
We are climbing and if all we think about is putting one
foot before the other, fatigue will overtake us long before we reach our goal. But if we walk in a group, happy at the
thought of the marvelous view we will surely discover up there, we will reach the summit without fatigue and will benefit in both joy and
health.
Montessori, M. (1948/1973). From childhood to adolescence.
Madras: Kalakshetra Publications. p. 28.
