the center for educational effectiveness, inc. 1 staff perceptions of school conditions what are...
TRANSCRIPT
1The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Staff Perceptions of School Conditions
What Are They?Do They Reflect Reality?
WERAMarch 2008
Pete Bylsma [email protected]
Greg Lobdell [email protected]
2The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
• Field-based research, service, and data-centric tools to support School & District Improvement
• Partnerships with 500 Schools in 100 districts in WA• Assist all schools & districts in OSPI School & District
Improvement programs– What we do & how we do it varies based on serving districts from 80
students K-12, to districts over 30,000 K-12.
• WASL Analysis and “Educational Growth” repository – WASL growth data (student cohorts) for schools serving 380,000
students
• The largest database of school effectiveness information in the state of Washington (Nine Characteristics of High Performing Schools)– 40,000 Staff responses– 110,000 Students (30% from homes where English is not primary
language)– 38,000 Parents (30% from homes where English is not primary
language)
• Schools of Distinction: Creation and Design for OSPI and ongoing research into what’s happening in these buildings
Data and theCenter for Educational Effectiveness
3The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
4The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Partnership
• Expand Capacity• Provide Value-add services• Expertise in core areas• Partnership is critical
CEE Services
OSPI
ESDs
Districts
Schools & Classrooms
Demographics & Community CharacteristicsDemographics & Community Characteristics
PerceptualAcademic
AchievementContextual-Program and
Process
5The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Who We ServeWASHINGTON SCHOOLS
April 2006 WASL Data
0
0.5
1
1.5
2
2.5
3
3.5
4
0 10 20 30 40 50 60 70 80 90 100
Percent Poverty
Co
mb
ined
Rea
din
g /
Mat
h L
evel
In
dex
All Schools
6The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Who We ServeWASHINGTON SCHOOLS
April 2006 WASL Data
0
0.5
1
1.5
2
2.5
3
3.5
4
0 10 20 30 40 50 60 70 80 90 100
Percent Poverty
Co
mb
ined
Rea
din
g /
Mat
h L
evel
In
dex
All Schools CEE Clients
7The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Opening the CEE Repositories for Research
Activities• What?
– Educational Effectiveness SurveyTM: Staff, Student, and Parent editions
• Why?– Expands CEE capacity to get research
back into the field– Brings organizational effectiveness data
from 189,000 educational stakeholders in WA into critical conversations at all levels
– Improves our collective practice – impacts future research and development activities
8The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The EES Staff Repository
• Demographics and descriptive variables– Respondents: Position, School Level,
Department, Length of service (both “this school” and “in education”), and ethnicity (listed as “optional”)
– Extensive research variables are attached at CEE as the data enters the repository
– N= 40,236 staff respondents as of March 1, 2008
9The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Position
Administrator-School
3%
Classified Support Staff
21%
Certificated Support Staff
7%
Certificated Teacher
64%
Para-Professional /
Instr. Aide5%
School Level
K-120%
High School24%
Elementary48%
Middle School23%
Junior High5%
Department or Level
Generalist K-3
17%Generalist 4-
68%
Other42%
Lang. Arts / Soc. Studies
14%
Electives8%
Math / Science
11%
Staff Descriptors
10The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Length of Service- In Education
0
2000
4000
6000
8000
10000
12000
< 1year
1 - 3years
4 - 5years
6 - 10years
11 -15
years
15+years
# of
Res
pond
ents
Length of Service- This School
0
1000
2000
3000
4000
5000
6000
< 1year
1 - 3years
4 - 5years
6 - 10years
11 - 15years
15+years
# of
Res
pond
ents
Staff Years of Service
11The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Ethnicity (optional) Other2%
African-American
1%
Asian / Pacif ic Islander
2%
American Indian / Alaskan Native
1%
Multi-racial2%
White / Caucasian
84%
Hispanic / Latino(a)
8%
Gender
Male, 6775, 26%
Female, 19383, 74%
Student Ethnicity: School-Level
Ethnic Majority44%
White Majority56%
Staff Ethnicity & Gender
School Makeup- Students
12The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
ESD by Respondents
ESD10526%
ESD1128%
ESD1134%
ESD11411%
ESD12127%
ESD12313% ESD171
5%
ESD1895%
ESD1011%
Geographic Segmentation
Large Central City1%
Rural, inside MSA17%
Rural, outside MSA10%
Small Tow n12%Large Tow n
5%
Urban Fringe of a Mid-Size City
10%
Urban Fringe of a Large City
22% Mid-size Central City23%
Title I Eligible
Eligible69%
Not Eligible31%
Title I Schoolwide
Schoolw ide52%
Not48%
13The Center for Educational Effectiveness, Inc.
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
Lead
ersh
ip
InstructionStudentLearning
Met AYP- Latest Year?
NO46%
YES54%
Federal School Improvement Step Number
Step 119%
Step 213%
Step 38%
Step 42%
Step 50%
None58%
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Center for Educational Effectiveness, Inc.www.effectiveness.org
Organization
StudentLearning
The Challenge• Time is the educational leaders best friend and
worst enemy• Dissemination and use of broadly applicable
information and research takes far too long and often doesn’t reach the people who need it most
• Let’s not forget:– Ocam’s Razor: “one should not increase, beyond what
is necessary, the number of entities required to explain anything”
– Often shortened as: “all else being equal, the simpler solution is preferable” (The “K.I.S.” principle)
Staff Perceptions of School Conditions
What Are They?Do They Reflect Reality?
WERAMarch 2008
Greg LobdellPete Bylsma
Study Rationale• Self assessments are one element of the school
improvement process.• State Board of Education requires survey data from
staff, students, and family members to inform their annual school improvement plans.
• Staff typically examine their schools on the dimensions of effective schools and reflect on the results to address perceived shortcomings.
• No attempt to examine survey results across multiple schools, no norms when reviewing results
Concerns About Survey Results
• Self perceptions reflect self awareness• Self awareness may not reflect reality
Psychologists have documented our tendency to inflate ourselves and be blind to our shortcomings or suffer from “groupthink”
• If educators don’t have accurate perceptions of their condition, they won’t to identify their problems, which will lead to efforts not focused on the right solutionsNeeded changes won’t occur, outcomes may not improve.
Lead to discouragement, less effort to improve in the future, and a belief that external conditions are to blame for the problems
Phases of Learning
Unconscious Competence
Conscious Incompetence
Unconscious Incompetence
Conscious Competence
Conscious of Unconscious Competence
Matrix of Perceptions of School Quality
Unrealistic
Unaware of limitations
Realistic
Aware of success
Realistic
Aware of limits/success
Realistic
Aware of limitations
Unrealistic
Unaware of success
Staff Perceptions of School Quality (consciousness)
Student Outcomes adjusted for student characteristics
(competence)
HIGH
MEDIUM
LOW
LOW MEDIUM HIGH
Objectives and Scope
• Determine staff perceptions of school characteristics based on a school’s– Grade level (elementary, middle, high, multiple grades, alternative)– Size (number of staff)– Socioeconomic status (% eligible for free/reduced lunch)– Student performance (avg. % met on grade 4/7/10 reading & math
WASL)
• Results from 430 public schools in CEE database (only known data set across schools in multiple districts)– Not a representative sample: more likely to be poor, low performing– Schools enrolled about 25% of all students in the state– Surveys reflect the views of more than 19,000 staff– Only included results from first survey administration
• School-level analysis only
Survey Contents/Structure
• 86 statements covering 9 characteristics of high-performing schools (plus 14 other statements)
• Range of 5-16 statements per characteristic
• 5-point Likert scale1. Almost always true
2. Often true
3. Sometimes true
4. Seldom true
5. Almost never true
Survey Results: All Schools
3.55 3.58 3.66 3.73
3.98 4.02 4.08
3.803.823.79
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Com
munic
atio
n &
Colla
bora
tion
Fo
cuse
d P
rof.
Deve
lop
me
nt
Fa
mily
& C
om
munity
Invo
lvem
ent
Mon
itor
Teach
ing &
Le
arn
ing
Alig
ned
Sta
nd.,
Curr
.,In
st., A
ssess
.
Hig
h S
tanda
rds
&E
xpect
atio
ns
Suppo
rtiv
e L
ea
rnin
gE
nvi
ronm
ent
Cle
ar
& S
ha
red
Fo
cus
Effect
ive L
ead
ers
hip
ALL
CH
AR
AC
TE
RIS
TIC
S
Ave
rage
resp
onse
Scale: 1 Almost never true 2 Seldom true 3 Sometimes true 4 Often true 5 Almost always true
(All results are preliminary)
Survey Results By Grade Level
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Clear & SharedFocus
High Standards& Expectations
EffectiveLeadership
Commun. &Collaboration
Aligned Stand.Curr. Inst.Assess.
Monitor Teaching& Learning
FocusedProfessionalDevelopment
SupportiveLearning
Environment
Family &CommunityInvolvement
Ave
rag
e r
esp
on
se
Elementary Middle/Jr High High (Regular)
On every characteristic, elementary had highest scores, HS the lowest
Survey Results By Size
140120100806040200
Number of staff responses
5.0
4.0
3.0
2.0
1.0
Cle
ar
& S
ha
red
Fo
cu
s
Fit line for Total
Middle.Jr High
High
ElementaryGrade Level
R Sq Linear = 0.046
• Larger schools had lower perceptions, but only because HS were larger and had lower scores
• No variation by size at the grade level
Survey Results By SES• Weak negative relationship at elementary and middle
school levels (perceptions decline as poverty increases), no relationship for professional development
• Only one characteristic significant at HS level (PD, +)• SES had little effect in multiple regression
1.0000.8000.6000.4000.2000.000
Pct Low Income
5.0
4.0
3.0
2.0
1.0
Hig
h S
tan
dar
ds
& E
xpec
tati
on
s
Grade Level: Middle/Jr High
R Sq Linear = 0.177
Strongest Relationship Middle School Results for “High Standards and Expectations”(R-square = .177)
Most lines were very flat
Survey Results By WASLPositive relationship at elementary and middle school levels (perceptions increase as WASL increases), but no relationship for professional development or for any HS results
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Clear & SharedFocus
High Standards &Expectations
Effective Leadership Commun. &Collaboration
Aligned Stand. Curr.Inst. Assess.
Monitor Teaching &Learning
FocusedProfessionalDevelopment
Supportive LearningEnvironment
Family &CommunityInvolvement
Ave
rage
Sta
ff R
atin
g
Bottom 1/3 Middle 1/3 Top 1/3Results at elementary level
Survey Results By WASLStrongest relationship related to “Family and Community Involvement” at elementary and middle school levels
10080604020
WASL avg (reading & math), 1st survey yr
5.0
4.0
3.0
2.0
1.0
Fa
mil
y &
Co
mm
un
ity
In
vo
lve
me
nt
Fit line for Total
High Achieving Group
Middle Achieving Group
Low Achieving Group
ElementaryAchievement Level
R Sq Linear = 0.213
100806040200
WASL avg (reading & math), 1st survey yr
5.0
4.0
3.0
2.0
1.0
Fa
mily
& C
om
mu
nit
y In
vo
lve
me
nt
Fit line for Total
High Achieving Group
Middle Achieving Group
Low Achieving Group
Middle/Jr HighAchievement Level
R Sq Linear = 0.312
Perceptions and WASL Crosstabs• Schools grouped into three groups based on
achievement controlling for SES
• Schools grouped into three groups based on overall perception score for all characteristics
Student Outcomes (competence)
Staff Perceptions of School Quality (consciousness)
Hig h 6.3% 13.7% 14.5%n 24 52 55
M ed iu m 10.0% 12.4% 10.6%n 38 47 40
Lo w 14.8% 9.2% 8.4%n 56 35 32
Lo w M ed iu m Hig h
(N=379)
Summary and Implications
• On average, school staff rate themselves fairly high
• Most results seem to reflect actual conditions and are consistent with research on effective schools
– Elementary schools score the highest, high schools the lowest– Perceptions increase as performance increases
• Perceptions do not increase as school size decreases, not affected much by SES
• High schools and PD remain enigmas• Some schools appear to have inaccurate perceptions
• Need other assessments/surveys to gain less subjective data
• Schools can use these trends to compare themselves to others
• We need ways to compare similar schools to each other
Further Study Needed
We still need to understand …• Perceptions of alternative schools and those serving multiple grades
• What is happening with professional development activities
• Why similar schools generate such a wide range of perceptions
• How perceptions in the same school change over time
• How perceptions at the staff level (e.g., new vs. experienced teachers)
• What is happening in schools with unrealistically high & low ratings
• How staff perceptions compare to student and family views
• How the results are being used at the school and district levels