Saginaw KEYS Data Analysis Training for
Continuous School Improvement
March 20 and 21, 2006
Jacques Nacson [email protected]
Gary Obermeyer [email protected]
Workshop Agenda
• Welcome and Introductions• Review of Agenda and Objectives• Your Expectations and KEYS Experiences • KEYS and Continuous School
Improvement: The Action Research Cycle• KEYS Data Analysis and Interpretations
--Indicators and Predictions--General Data-Related Considerations--Demo Report: Charts and Data Points--Tools and Data Analysis Worksheets--Navigating the Online Data Report and Online Resources
• The NEA KEYS 2.0 Toolkit and Use of the Action and Facilitation Guides
Workshop Objectives
• To examine KEYS data within the larger context of continuous school improvement processes
• To learn how to analyze and interpret KEYS data and how results can be shared with school staff and other stakeholders
• To consider strategies for how KEYS data might be used as a stimulus to drive your targeted school improvement efforts
• To learn how to use KEYS support materials, including online resources, to help you take next steps beyond preliminary analysis of KEYS data
Problem orOpportunity
Identified
Data Collection
Diagnosis & Refinement of
ProblemAction Planning
Action TakingImplementation&
Evaluation
Model for Understanding and Interpreting KEYS 2.0 Data
School completesKEYS 2.0 survey
School receives scores on KEYS 42 indicators of school
quality
Use other data, including student achievement to
confirm, validate or justify
Make inferences about our school’s quality based on
the KEYS results and assessments of other data
School makes decisions about next steps and plans for
improvement
Make judgments based on theories or
hypotheses (if, then) to explain “why”
Implementation of planned intervention
Process & product evaluation including effects on student
achievement
NEA 6 Keysto
School Quality
General Questions and Principles to Consider
Before Looking at Your KEYS 2.0 Data
Recognizing the limits of the bestsurvey instruments
Distinguishing between symptomsand root causes
Are the data surprising orconfirming?
Are you listening to all relevantvoices in your school community?
Moving from data to understandingto knowledge
Without theory, no organizationallearning and improvement canoccur
General Questions and Principles to Consider
Before Looking at Your KEYS 2.0 Data(continued)
Applying systems thinking: thewhole is greater than the sum of itsparts
Looking for patterns within andacross Keys
Slow and steady: the road tocontinuous improvement has noend (PDSA)
Celebrating your victories andlearning from your strengths
KEYS 2.0 – A diagnostic toolleading to a deliberative process,not a rigid, mechanistic, quick-fixprescription
Flow Chart of Suggested Activities/Processes For Analyzing and Interpreting Your Data
School Community Completes KEYS Survey
School Improvement Plan Implemented and Evaluated
Created a School Improvement Team (SIT)
SIT trained in action research, use of quality tools and analysis and interpretation of KEYS Results
SIT conducted preliminary analysis and interpretation of KEYS results and prepared for involvement of whole school community
SIT assisted whole school community to examine results and make preliminary recommendations
SIT considered whole school’s recommendations and begins deliberative process
SIT developed preliminary school improvement plan based on KEYS data and other data and present to school community for feedback
School improvement plan approved and ready for implementation
?No
Yes
No
No
Yes
Yes
?
?
?
?
?
Yes
Yes
No
No
No ?
Yes
Yes
No
Understanding the Graphs for Each Key
Horizontal Axis Measure of quality: 5 point scale Left side Disagree (low value) Right side Agree (high value)
Vertical Axis Indicators (groups of questions that measure the same concept)
KEY 1. Shared Understanding and Commitment To High Goals
Respondents Provide Direct Instruction to Students
Understanding the Graphs for Each Key
School Average All Schools Average
90th Percentile Score• Data Points
• School average (red)• All schools average (38 pilot schools) (Purple)• 90th percentile score of the pilot sample (Yellow)• Length of the horizontal bar (Purple) 1 standard deviation above and below the school average (measure of agreement) The Goals for your school in terms of continuous improvement for each indicator• The school average moving continuously toward the right side (agree – high value of quality for that indicator)• At the same time, reduce the standard deviation (narrow the length of the horizontal bar, meaning greater agreement among respondents)
Standard Deviation
District Reports
5.4 Safe Environment
Worksheet for Analyzing KEYS 2.0 Results (Analysis of Individual Indicators)
Preliminary AnalysisNOTE: THIS WORKSHEET HAS BEEN REDESIGNED
Measures of Quality Degree of Consensus
Low(Wide
>1.3 pt.)
High(Narrow
<1 .3pt.)
VeryHigh
(between 4.1 & 5.0)
High(between3.4 & 4.0)
Low (between2.1 & 2.6)
Very Low
(average scorebetween 1 & 2)
Indicators
Key:_________________ Constituency Group:_________________ Date of Survey:______________________
Worksheet for Analyzing KEYS 2.0 Results (Analysis of Cross Key Indicators)
(Secondary Analysis)
Related Indicators Indicators
Key:_____________________ Constituency Group:____________________
Date of Survey:________________________
score score score score score score score
score score score score score score
score score score score score score
score
score
score
score
scorescore
score
score
scorescorescore
scorescore
score
scorescore
• Action Guide
• Facilitation guide
Tool Box that contains a CD with all materials
Steps a School Might Take Once KEYS Preliminary Analyses are Completed
1. GAPS: Decide on one indicator or a group of indicators where gap (s) exist.
2. RELEVANCE: Reflect with the “team” on the relevance, importance and priority of the selection.
3. DATA COLLECTION-VALIDATION: Consider the need to collect additional data to validate the KEYS findings.
4. DATA COLLECTION-DIAGNOSIS AND REFINEMENT: Examine what is the “root” cause of the problem.
Steps a School Might Take Once KEYS Preliminary Analyses are Completed
5. THEORY OF ACTION: Identify and select the most appropriate solution for your context.
6. ACTION PLANNING: Set achievable goals and develop specific action/project plans.
7. IMPLEMENTATION: Action plans must be implemented for improvement to occur.
8. DATA COLLECTION-EVALUATION: Both process and product evaluations are necessary for learning to happen.
9. BACK TO STEP 1
Objectives for Advanced Training KEYS Data Analysis
• Understand the Value of Data for Making Good Decisions
• Distinguish Between Symptoms and Root Causes for Continuous Improvement
• Learn How to Analyze and Use Data through the Continuous Improvement Cycle (PDCA)
• Use Quality Tools to Improve Process Quality and Student Outcomes
• Understand the Concept of Variation and Use Control Charts to reduce variation and improve outcomes
Agenda for Advanced Training KEYS Data
analysis• Data and General Data Principles• W. Edward Deming – Data and Variation• Continuous Improvement Cycle (PDCA)• Quality Tools
Statistical Process Control-Quality Control ChartsGetting to the Root CauseCollecting, Organizing and Analyzing your DataSelecting the Right Intervention Based on Theory and Hypothesis TestingPlanning and Carrying Out the InterventionEvaluating the success of the Intervention