florida association of professors of educational...
TRANSCRIPT
www.FLDOE.org1
Fall MeetingOctober 3, 2016Orlando, Florida
Florida Association of Professors of Educational Leadership (FAPEL)
Florida Educational Leadership Examination (FELE)
www.FLDOE.org2
Postsecondary Assessment Team for Today
Phil Canto, Bureau Chief
Dr. Suleyman Olgar, Psychometrics, Scoring & Reporting Director
Dan Moore, Test Development Team
www.FLDOE.org3
Today’s Postsecondary Assessment Presentation
Present status of current FELE program
Discuss status of research for “Next Gen” FELE
2015-2016 FELE Testing Outcomes
FELE Subtest Order Study
Actionable Feedback And Available Resources
Q & A
www.FLDOE.org4
AERA, APA, NCME Standards
Standard 8.2, p.134
“Test takers should be provided in advance with as much information about the test, the testing process, the intended test use, test scoring criteria, testing policy, availability of accommodations, and confidentiality protection as is consistent with obtaining valid responses and making appropriate interpretations of test scores.”
Comment: When appropriate, test takers should be informed in advance about test content, including subject area, topics covered, and item formats.
www.FLDOE.org5
Educator Quality-BHAG
“Every Florida teacher and leader is prepared, developed, supported, and supervised by educators to make teaching better.”
www.FLDOE.org6
• No planned substantive changes to current FELE content, structure, passing scores, or examination fees in 2016 or 2017
• Ongoing enhancement and expansion of M/C item pools for all three subtests-We constantly monitor to ensure quality control
• Development of additional WPA prompts is a possibility-IHE faculty will be involved
FELE Program Updates
www.FLDOE.org7
• FLDOE continues to evaluate options for performance based or enhanced assessment(s)
• May impact FELE and/or other exams in the future (current or new)
• 2017 before pilot project could begin with statewide inputs
• Keep in mind the normal full test development process is at least 2 years
FLDOE Research for Next Generation FELE
www.FLDOE.org8
FELE Subtest Number Correct Percentage Correct
Subtest 1: Leadership for Student
Learning48/70 69%
Subtest 2: Organizational Development 48/69 70%
Subtest 3: System Leadership:
Multiple-Choice Section36/55 66%
Subtest 3: System Leadership: Written-
Performance Section7/12 59%
FELE Approved Passing Scores – Effective 01/01/2015
www.FLDOE.org9
FLORIDA EDUCATIONAL LEADERSHIP EXAMINATION (FELE)
2015-2016 Statewide Pass Rates
Test Name N
First-Attempt Best-Attempt
Pass Rate Mean Score Pass Rate Mean Score
FELE Subtest 1:
Leadership for Student Learning2,889 63% 202 74% 205
FELE Subtest 2:
Organizational Development2,835 59% 201 73% 204
FELE Subtest 3:
Systems Leadership
Multiple-Choice Section
2,889 52% 197 70% 202
FELE Subtest 3:
Systems Leadership
Written Performance Section
2,886 52% 7 72% 8
FELE Assessment Level 2,774 24% ** 54% **
Note: Data is based on examinees taking FELE from January 2, 2015 to August 1, 2016. Assessment level pass rates are based on examinees taking
and passing all subtests including both sections of FELE Subtest 3. **: Not Applicable
www.FLDOE.org10
FLORIDA EDUCATIONAL LEADERSHIP EXAMINATION (FELE)
2015-2016 Statewide Pass Rates
Note: Data is based on examinees taking FELE from January 2, 2015 to August 1, 2016. Assessment level pass rates are based on examinees taking and passing all subtests including both sections of FELE Subtest 3.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
FELE Subtest 1:Leadership for Student
Learning
FELE Subtest 2:Organizational Development
FELE Subtest 3:Systems Leadership
Multiple-Choice Section
FELE Subtest 3:Systems Leadership
Written PerformanceSection
FELE Assessment Level
First-Attempt
Best-Attempt
www.FLDOE.org11
FLORIDA EDUCATIONAL LEADERSHIP EXAMINATION (FELE)
First-Attempt Statewide Pass Rates Comparison 2015 vs 2016
Test Name N
2015
N
2016
Pass Rate Mean Score Pass Rate Mean Score
FELE Subtest 1:
Leadership for Student Learning1,804 63% 202 1,085 64% 203
FELE Subtest 2:
Organizational Development1,754 58% 200 1,081 61% 202
FELE Subtest 3:
Systems Leadership
Multiple-Choice Section
1,810 50% 197 1,079 54% 199
FELE Subtest 3:
Systems Leadership
Written Performance Section
1,808 54% 7 1,078 48% 7
FELE Assessment Level 1,706 25% ** 1,068 22% **
Note: Data is based on examinees taking FELE from January 2, 2015 to August 1, 2016. Assessment level pass rates are based on examinees taking
and passing all subtests including both sections of FELE Subtest 3. **: Not Applicable
www.FLDOE.org12
Florida Educational Leadership Examination (FELE) Competency Performance
2015-2016 Statewide Examinees
Test Name Competency NameStatewide
N % Correct
Florida Educational Leadership Examination (FELE)
FELE Subtest 1:
Leadership for Student Learning
01 (17-18) Facilitate student learning goals
4,088
67.9
02 (18) Prioritize student learning via leadership 73.2
03 (15-17) Develop/implement instructional framework 69.0
04 (17) School environment for student learning 68.6
FELE Subtest 2:
Organizational Development
01 (11-12) Recruitment and induction practices
4,166
72.9
02 (22-23) Faculty/staff development and retention 70.1
03 (16-17) Develop leadership w/in the organization 69.0
04 (17-18) Personal/professional behavior practices 68.4
FELE Subtest 3:
Systems Leadership
Multiple-Choice Section
01 (13-14) Decision-making processes
4,618
65.0
02 (13-14) School operation management practices 67.5
03 (14) Financial and resource management 64.0
04 (13) School legal practices and applications 65.1
FELE Subtest 3:
Systems Leadership
Written Performance Section
05 (1) Effective communication practices 4,500 59.0
Note: Data is based on all examinees taking FELE from January 2, 2015 to August 1, 2016. The numbers in the parenthesis indicate range of item counts per
competency. Percentages correct are based on average examinee performance on each competency. Counts Include retake attempts.
www.FLDOE.org13
FLORIDA EDUCATIONAL LEADERSHIP EXAMINATION (FELE)
2015-2016 First-Attempt Pass Rates by Program Type
* Statewide total includes state approved district FELE programs.Note 1: Data is based on examinees taking FELE from January 2, 2015 to August 1, 2016. Assessment level pass rates are based on examinees taking and passing all subtests including both sections of FELE Subtest 3. **: Not ApplicableNote 2: Matching was done based on Title II data.
Test Name
Non-program Program Statewide Total*
NPass
Rate
Mean
ScoreN
Pass
Rate
Mean
ScoreN
Pass
Rate
Mean
Score
FELE Subtest 1:
Leadership for Student Learning1,575 61% 201 1,227 66% 203 2,889 63% 202
FELE Subtest 2:
Organizational Development1,546 58% 200 1,204 60% 201 2,835 59% 201
FELE Subtest 3:
Systems Leadership Multiple-Choice
Section
1,566 52% 197 1,240 50% 197 2,889 52% 197
FELE Subtest 3:
Systems Leadership Written
Performance Section
1,563 49% 7 1,240 55% 7 2,886 52% 7
FELE 3.0:
Assessment Level1,505 24% ** 1,187 23% ** 2,774 24% **
www.FLDOE.org14
FELE Subtest Order Study
• FELE 3.0 Examination • Subtest 1: Leadership for Student Learning
• Subtest 2: Organizational Development
• Subtest 3: Systems Leadership
• Multiple-Choice Section
• Written Performance Section
• Chi-Square Test Study Groups: N• A: Only Subtest 3 81
• B: Subtests 1 & 3 and 2 & 3 16
• C: All 3 Subtests 3,549
Note: Due to N counts, Groups A and C comparison will be discussed.
www.FLDOE.org15
FELE Subtest Order Study Findings
• Due to very large sample size difference, 100 random samples of 100 examinees each were drawn from group C to compare outcomes with group A.
• Chi-Square Test Group A and C comparison is not significant for any sample• There is no significant difference in the order of the subtests were
given to the FELE examinees.
• Chi-Square Test statistics is less than 2.15 for all 100 samples (significance critical value = 3.841)
• Pass rates for the groups are quite similar:• 48.1% for Group A (N=81)
• 47.8% for Group C (N=3,549)
www.FLDOE.org16
FELE Written Performance Assessment (WPA) & Holistic Scoring
• Interpretation of data and writing ability as it relates to the data analysis and target audience;• Organization, sentence structure, and mechanics, as well
as the accuracy of data interpretation
• An examinee response is judged for its total effect;• Rater makes an overall evaluation taking multiple
performance factors into consideration
• Responses are rated independently by two raters and sent through a discrepancy resolution process if rater scores vary by more than 1 point.
www.FLDOE.org18
FELE Written Performance Assessment
The FELE WPA is intended to measure prospective educational leaders' and administrators' ability to analyze and interpret educational data, as well as the efficacy of their written communication.
The WPA written responses are evaluated for content accuracy, data trend analysis, organization, appropriate audience level, and grammar.
www.FLDOE.org19
Feedback from FELE RatersTrends in examinee submissions that are limiting ratings to a score point of 1, 2, or 3.
• Not using data to inform response
• Not citing specific data
• Lack of data
• Using generalizations instead of examples/evidence
• Failing to address all areas of the prompt
www.FLDOE.org20
Feedback from FELE RatersTrends in examinee submissions earning a score point of 4, 5, or 6.
• Thorough data analysis and well-developed plans
• Citing specific data, not generalizations of the data
• Explanation of data points used to inform response
• Providing examples/evidence from the data
• Addressing all areas of the prompt
www.FLDOE.org21
Feedback from FELE RatersCommon factors that distinguish a score point of 4 from a score point of 3.
• Depth of the data analysis and quality of the plan
• Specific data (numbers/percentages) instead of generalizations
• Examples/Evidence from the data to inform the response
www.FLDOE.org22
FELE Rater Feedback SummaryBased on the feedback provided by the FELE Raters and Chief Reviewers, common areas for improvement include:
• Citing Specific Data (numbers/percentages)
• Thorough Data Analysis (explanations/trends)
• Addressing All Prompt Requirements (bullet points)
www.FLDOE.org23
Competency and Skills AssessedSubtest 3, Competency 5Knowledge of effective communication practices that accomplish school and system-wide goals by building and maintaining collaborative relationships with stakeholders.
1. Analyze data and communicate, in writing, appropriate information to stakeholders.
2. Analyze data and communicate, in writing, strategies for creating opportunities within a school that engage stakeholders.
3. Analyze data and communicate, in writing, strategies that increase motivation and improve morale while promoting collegial efforts.
www.FLDOE.org24
FELE WPA Scoring Constructs
The FELE WPA is scored on two constructs.
Communication SkillsIdeas Focus OrganizationMechanics
Data AnalysisInterpretationEvaluationExplanation ApplicationImplicationsTrend Analysis
www.FLDOE.org25
FELE Resources (Links to documents)
1. 2015 Annual Administration and Technical Report
2. Curriculum Mapping Tool
3. FELE Test Information Guide
4. WPA Resources:
• Scoring Rubric
• Sample Prompt #1
• Sample Prompt #1 Supplemental Rating Criteria
• Sample Prompt #2
• Sample Prompt #2 Supplemental Rating Criteria
www.FLDOE.org26
Discussion & Questions
Call
850-245-0513
Visit
http://www.fldoe.org/accountability/assessments/postsecondary-assessment/fele/
www.FLDOE.org27
www.FLDOE.org