2021 epp annual report - msu billings

2021 EPP Annual Report CAEP ID: 10033 AACTE SID: 1030
Institution: Montana State University-Billings
Unit: College of Education
Section 1. EPP Profile After reviewing and/or updating the Educator Preparation Provider's (EPP's) profile in AIMS, check the box to indicate that the information available is accurate.
1.1 In AIMS, the following information is current and accurate... Agree Disagree
1.1.1 Contact person 1.1.2 EPP characteristics 1.1.3 Program listings
1.2 [For EPPs seeking Continuing CAEP Accreditation]. Please provide a link to your webpage that demonstrates accurate representation of your Initial-Licensure Level and/or Advanced-Level programs as reviewed and accredited by CAEP (NCATE or TEAC). https://www.msubillings.edu/coe/degrees.htm
Section 2. Program Completers 2.1 How many candidates completed programs that prepared them to work in preschool through grade 12 settings during Academic Year 2019-2020 ?
Enter a numeric value for each textbox. 2.1.1 Number of completers in programs leading to initial teacher certification or licensure1 85
2.1.2 Number of completers in advanced programs or programs leading to a degree, endorsement, or some other credential that prepares the holder to serve in P-12 schools (Do not include those completers counted above.)2
41
Total number of program completers 126
1 For a description of the scope for Initial-Licensure Programs, see Policy 3.01 in the Accreditation Policy Manual 2 For a description of the scope for Advanced-Level Programs, see Policy 3.02 in the Accreditation Policy Manual
Section 3. Substantive Changes Have any of the following substantive changes occurred at your educator preparation provider or institution/organization during the 2019-2020 academic year?
3.1 Changes in the established mission or objectives of the institution/organization or the EPP
3.2 Any change in the legal status, form of control, or ownership of the EPP.
3.3 The addition of programs of study at a degree or credential level different from those that were offered when most recently accredited
3.4 The addition of courses or programs that represent a significant departure, in terms of either content or delivery, from those that were offered when most recently accredited
3.5 A contract with other providers for direct instructional services, including any teach-out agreements
Evidence of Professional Growth Data Analysis (ELEM-JF-GP vs ST Report-EPG.)
Program Name: ELEM Date: 1/17/2020 Participants: Semesters: fall 2018, spring 2019, and fall 2019 data
What do you notice?
What are the implications of data for program improvement ?
Is this data valid and reliable?
How can we establish the validity of data?
How can we establish the reliability of data?
1. Improvemen t between jr field/grad practicum and student teaching
2. Domain 5 Analysis and assessment of data see some greater variability with more students performing below “adequate” level
3.
1. Who is scoring at each level?
2. What approaches are taking place to insure reliability across raters?
3. How many student are being required to do a third EPG in student teaching?
4. How many students are being given opportunities to revise to increase their scores? (What level of support are students at each level receiving to complete EPGs)
5. Should this be scaffolded or independent pererformance ? And at each level?
6. Who should be scoring the instrument?
Need to establish both reliability and validity However, we first need to determine if the current EPG is the measure we want to continue to use moving forward (e.g is this the right instrument to be using)
Appears to have good construct validity with room for improvement . No work has been done to examine concurrent or predictive validity. No ongoing practice for establishing and maintaining reliability
Once instrument determine d then use standard procedures to establish Construct, concurrent and predictive validity
Once instrument determine d then use standard procedures to establish Reliability (interrater and kappa).
What type of support (for example, feedback on unit/lesson plans prior to implementation) do junior
field/grad practicum students receive for each EPG from their supervisors/professor?
Jr. Field- All students given 1 page paper (EPG assignment) outlining how to write EPG. Also given
student teaching guidebook with outlines EPG. Students have option to review EPG with faculty teaching
course prior to submitting final version. However few act on this. Also have option to re-do if earn a
“low” score. Again, few students have done so. Meet with every student on EPG at end of semester and
provide feedback on what was done well and give advice on ways to improve performance in student
teaching.
Grad practicum- Full seminar of instruction (2 hours) on EPG with model provided. 2 models on quant
data available. Added models for qualitative data. Turn in lesson plan associated with EPG and receive
feedback on it. Submit written draft which is scored and receive feedback on areas to improve. Choice
to R&R- all have chosen to do so. Never had anyone earn below 3 on revised version.
What type of support (for example, feedback on unit/lesson plans prior to implementation) do
student teachers receive for each EPG from their supervisors/professor?
Given student teaching handbook which has EPG instructions, rubric and examples to follow.
EPG 1 – our group has no real knowledge beyond above.
EPG 2
Evidence of Professional Growth (EPG) Data Analysis for Elementary Education
Description of the analysis
1. This analysis compares elementary education student teachers’ junior field /grad practicum performance with student teaching performance using the Evidence of Professional Growth rubric.
2. Students who student taught in fall 2018, spring 2019 and fall 2019 are included in this analysis. 3. Junior field/grad practicum EPG evaluation was compared with the first and second student
teaching EPG evaluation. 4. Students with complete evaluations (junior field/grad practicum EPG, first and second EPG’s for
student teaching) are included in this analysis. 73 junior field/grad practicum EPG evaluations were compared with 73 corresponding student teaching EPGs’ (first and second.)
Observations
1. All questions except “Implications for Future Teaching” had at least one student having “some skills” in student teaching EPG evaluation one or/and two.
2. Some junior field students had little or no skills/inconsistent or weak skills for certain questions. However, when it comes to student teaching EPG 1 and student teaching EPG 2 performance, none of the students had little or no skills/inconsistent or weak skills. Performance for student teachers was either in “some skills,” “Adequate skills,” “Competency” categories.
3. There is an overall growth from junior field experience to student teaching experience.
Limitations
There are a few limitations to this analysis. First, the EPG instrument is not currently validated, and reliability is not established. Second, this is a purposive sample, and only complete evaluations (one junior field/grad practicum EPG, first and second EPG’s for student teaching) are included. For some questions, the percentages do not add up to 100% for the different categories; this is because there is at least one student for that question with a blank response. These missing data were not removed. According to CAEP guidebook (page 135) Data Quality guidelines, it is important to discuss the following areas. • Valid and consistent (or reliable) • Relevant • Representative • Cumulative (which includes “multiple measures”) • Actionable • Fairness, which means free from bias • Robustness referring to evidence that is a direct and compelling measure of the condition
intended to be informed
InTASC Standard
InTASC Category
PEPPS Standard Evaluation Area # Little or No Skills Inconsistent or Weak Skills Some Skills Adequate Skills Competency Elementary Education K-8-JF-GP 73 0.00% 2.74% 16.44% 65.75% 15.07% Elementary Education K-8-ST1 73 0.00% 0.00% 2.74% 41.10% 56.16% Elementary Education K-8-ST2 73 0.00% 0.00% 2.74% 36.99% 56.16%
PEPPS Standard: 10.58.501(d)
InTASC Standard: 4,9
Quality of Professional Writing in Terms of Mechanics, Communication of Ideas, and Completeness
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Some Skills Adequate Skills Competency
ELEM-F18,SP19,F19
Question
PEPPS Standard Evaluation Area # Elementary Education K-8-JF-GP 73 Elementary Education K-8-ST1 73 Elementary Education K-8-ST2 73
Little or No Skills Inconsistent or Weak Skills Some Skills Adequate Skills Competency 4.11% 1.37% 17.81% 50.68% 17.81% 0.00% 0.00% 5.48% 27.40% 67.12% 0.00% 0.00% 2.74% 27.40% 69.86%
PEPPS Standard: 10.58.501(b)
InTASC Standard: 2
ELEM-F18,SP19,F19
Question
InTASC Standard: 2
ELEM-F18,SP19,F19
Question
InTASC Standard: 2
ELEM-F18,SP19,F19
Question
PEPPS Standard: 10.58.311(c), 10.58.311(d), 10.58.501(e)
InTASC Category: Content/Instructional Practice
ELEM-F18,SP19,F19
Question
PEPPS Standard: 10.58.311(d), 10.58.501(b), 10.58.501(e)
InTASC Category: The Learner and Learning/Content/Instructional Practice
InTASC Standard: 2,5,7,8
ELEM-F18,SP19,F19
Question
PEPPS Standard: 10.58.501(a), 10.58.501(b), 10.58.501(c)
InTASC Category: The Learner and Learning
InTASC Standard: 1,2,3
ELEM-F18,SP19,F19
Question
PEPPS Standard: 10.58.311(d), 10.58.501(a), 10.58.501(c), 10.58.501(e)
InTASC Category: The Learner and Learning/Content
InTASC Standard: 1,3,5
Potential of Planned Activities to Engage Students in the Lesson Objective Concepts
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
ELEM-F18,SP19,F19
Question
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%
100.00%
Chart Title
Question
ELEM-F18,SP19,F19
Question
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
ELEM-F18,SP19,F19
Question
ELEM-F18,SP19,F19
Question
InTASC Category: Learner and Learning/Insructional Practice/Professional Responsibility
InTASC Standard: 1,2,3,7,9
ELEM-F18,SP19,F19
Question
Little or No Skills Inconsistent or Weak Skills Some Skills Adequate Skills Competency 12.33% 12.33% 34.25% 24.66% 16.44%
0.00% 0.00% 2.74% 42.47% 54.79% 0.00% 0.00% 1.37% 24.66% 73.97%
ELEM-F18,SP19,F19
Question
ELEM-F18,SP19,F19
Question
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
ELEM-F18,SP19,F19
Description of the analysis-ELEM
Evidence of Professional Growth (EPG) Data Analysis for Special Education
1. The initial objective of the study was to compare special education student teachers’ junior field performance with student teaching performance using the Evidence of Professional Growth rubric.
2. However, when matching student teachers’ EPG’s with their corresponding junior field EPG evaluation, it was revealed that some graduate students did not have a corresponding junior field EPG.
3. SPED-Generalist graduate students are not required to complete a junior field EPG evaluation. 4. For SPED endorsement, EDSP 404 course is required. However, by examining the plan of study
documents, it was revealed that for some students EDSP 404 course was substituted with another course. These students did not have junior field EPG’s.
5. An attempt was made to match fall 2019, spring 2019, and fall 2018 student teacher’s EPG’s with their corresponding junior EPG rubrics. However, less than ten student teaching EPG’S were matched with their corresponding junior field EPG. Due to the reasons in items 2,3, and 4, instead of making a direct comparison, the analysis was conducted based on available data for SPED junior field evaluations (fall 2016, fall 2017, and fall 2019 semesters).
6. For student teaching EPG evaluations, fall 2018, spring 2019, and fall 2019 evaluations were included in this analysis. Please note that fall 2019 double majors are not included in this analysis (due date for the evaluators to submit second placement evaluation packet is January 17th 2020.)
Observations
Some junior field students had little or no skills/inconsistent or weak skills in certain areas. However, when it comes to student teaching EPG one and student teaching EPG two performance, none of the students had little or no skills/inconsistent or weak skills. All questions had performances either in “some skills,” “Adequate skills,” “Competency” categories.
Limitations
There are a few limitations to this analysis. First, the EPG instrument is not currently validated, and reliability is not established. The sample size for each semester is small. Also, fall 2019 double majors (student teachers) are not included in this analysis. For some questions, the percentages do not add up to 100% for the different categories; this is because there is at least one student for that question with a blank response. These missing data were not removed.
According to CAEP guidebook (page 135) Data Quality guidelines, it is important to discuss the following areas related to the EPG rubric.
i. Valid and consistent (or reliable) ii. Relevant
iii. Representative iv. Cumulative (which includes “multiple measures”) v. Actionable
vi. Fairness, which means free from bias vii. Robustness referring to evidence that is a direct and compelling measure of the
condition intended to be informed
Question InTASC Standard INTASC Category PEPPS Standard
Semester Evaluation Area # Little or No Skills Inconsistent or Weak Skills Some Skills Adequate Skills Competency Fall 2016 Special Education P-12-JF-F16 10 0.00% 0.00% 50.00% 40.00% 10.00% Fall 2017 Special Education P-12-JF-F17 14 0.00% 7.14% 14.29% 7.14% 64.29% Fall 2019 Special Education P-12-JF-F19 11 0.00% 0.00% 27.27% 63.64% 9.09%
PEPPS Standard: 10.58.501(d) INTASC Category: Content/Professional Responsibility
InTASC Standard: 4,9 Quality of Professional Writing in Terms of Mechanics, Communication of Ideas, and Completeness
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
SPED Junior Field Experience-EPG
Semester Evaluation Area # Fall 2016 Special Education P-12-JF-F16 10 Fall 2017 Special Education P-12-JF-F17 14 Fall 2019 Special Education P-12-JF-F19 11
42.86% 7.14% 21.43% 0.00% 28.57% 0.00% 0.00% 45.45% 45.45% 9.09%
PEPPS Standard: 10.58.501(b) InTASC Category: The Learner and Learning
InTASC Standard: 2 School and Community Description
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
35.71% 7.14% 28.57% 0.00% 28.57% 0.00% 0.00% 9.09% 72.73% 18.18%
InTASC Standard: 2 Classroom Description
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
InTASC Standard: 2 Student Description
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
PEPPS Standard: 10.58.311(c), 10.58.311(d), 10.58.501(e) InTASC Category: Content/Instructional Practice
InTASC Standard: 5,6,8 Lesson Objectives Identified
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
PEPPS Standard: 10.58.311(d), 10.58.501(b), 10.58.501(e) InTASC Category: The Learner and Learning/Content/Instructional Practice
InTASC Standard: 2,5,7,8 Lesson Structure and Procedures
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
PEPPS Standard: 10.58.501(a), 10.58.501(b), 10.58.501(c) InTASC Category: The Learner and Learning
InTASC Standard: 1,2,3 Instructional Materials and Resources
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
PEPPS Standard: 10.58.311(d), 10.58.501(a), 10.58.501(c), 10.58.501(e) InTASC Category: The Learner and Learning/Content
InTASC Standard: 1,3,5 Potential of Planned Activities to Engage Students in the Lesson Objective Concepts
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
InTASC Standard: 5,6,8 Assessments Align with Lesson Objectives
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
InTASC Standard: 5,6,8 Validity of Assessments
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
InTASC Standard: 5,6,8 Assessments Include Pre and Post Assessment Data
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
InTASC Standard: 5,6,8 Analysis of Data
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
PEPPS Standard: 10.58.501(a), 10.58.501(b), 10.58.501(c) InTASC Category: Learner and Learning/Insructional Practice/Professional Responsibility
InTASC Standard: 1,2,3,7,9 Insights on Effective Instruction
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Insights on Effective Assessment
InTASC Standard: 1,2,3,7,9 Implications for Future Teaching
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
Series1 Series2 Series3
InTASC Standard: 5,6,8 Evidence of Impact on Student Learning
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Semester Evaluation Area # Little or No Skills Inconsistent or Weak Skills Some Skills Adequate Skills Competency Fall 2018 Special Education P-12-ST-EVAL#1-F18 16 0.00% 0.00% 12.50% 25.00% 62.50% Spring 2019 Special Education P-12-ST-EVAL#1-SP19 8 0.00% 0.00% 12.50% 25.00% 62.50% Fall 2019 Special Education P-12-ST-EVAL#1-F19 7 0.00% 0.00% 0.00% 14.29% 85.71%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
SPED Student Teaching-EPG#1
Semester Evaluation Area # Fall 2018 Special Education P-12-ST-EVAL#1-F18 16 Spring 2019 Special Education P-12-ST-EVAL#1-SP19 8 Fall 2019 Special Education P-12-ST-EVAL#1-F19 7
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Insights on Effective Assessment
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Semester Evaluation Area # Little or No Skills Inconsistent or Weak Skills Some Skills Adequate Skills Competency Fall 2018 Special Education P-12-ST-EVAL#2-F18 14 0.00% 0.00% 7.14% 28.57% 64.29% Spring 2019 Special Education P-12-ST-EVAL#2-SP19 9 0.00% 0.00% 22.22% 0.00% 77.78% Fall 2019 Special Education P-12-ST-EVAL#2-F19 5 0.00% 0.00% 0.00% 20.00% 80.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
InTASC Standard: 1,2,3,7,9 Insights on Effective Assessment
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Description of the analysis-SPED
SPED-JF
SPED-ST-1
SPED-ST2
EPG-SPED-Final.pdf
Evidence of Professional Growth Data Analysis -Secondary-JF-GP vs ST Report-EPG.pdf (If required please review the detailed reports as well in other data folder- these are detailed reports)
Program Name: Secondary: Cori, Kari, Amanda, Kathy, and Dave Date: 1/17/2020 Participants: Semesters: fall 2018, spring 2019, fall 2019 data
What do you notice?
What are the implications of data for program improvement?
Our students are performing well across evaluations. We notice that the secondary students perform low in assessment; one important implication is that our secondary students don’t have assessment. Concern: We are teaching to a test if this is across our curriculum. Concern that EPG might not be appropriate for all endorsement areas.
When a student is evaluated is it based upon their skills at the time at the field placement or are students scored upon their skills compared to a master teacher? How many different supervisors score student teachers? Is there a difference between out of area scores versus scores in the area.
Assessment is needed for secondary students, based upon what we see in the data; anecdotally, I think classroom management would be needed as well. Perhaps an analysis of student reflections would provide data concerning classroom management need.
Issues concerning standard alignment within tool.
What theories and/or published materials provided a foundation for construct validity? Formula for grading student teaching was created by field experience committee (starting 2015?).
Complete inter-rater reliability. Train supervisors and provide exemplars. Suggestion: D2L could be used for supervisor training.
field student receive for each EPG from their supervisors/professor?
I can speak for reading students, as their supervisor/professor we walk through EPGs and how to plan.
We look at examples.
What type of support (for example, feedback on unit/lesson plans prior to implementation) do student
teachers receive for each EPG from their supervisors/professor?
As a student teaching supervisor, I discuss with students their EPGs and provide feedback.
EPG 1
EPG 2
Evidence of Professional Growth (EPG) Data Analysis for Secondary Education
1. This analysis compares secondary education student teachers’ junior field/grad practicum performance with student teaching performance using the Evidence of Professional Growth rubric.
2. Students who student taught in fall 2018, spring 2019 and fall 2019 are included in this analysis. 3. Junior field/grad practicum EPG evaluation was compared with the first and second student
teaching EPG evaluation. 4. Students with complete evaluations (junior field/grad practicum EPG, first and second EPG’s for
student teaching) are included in this analysis. 35 junior field/grad practicum evaluations were compared with 35 corresponding student teaching EPGs’ (first and second.)
Observations
1. When it comes to junior field evaluations, there were many questions for which students had Little/No skills or Inconsistent/Weak skills.
2. None of the student teaching EPG evaluations had Little/No Skills or Inconsistent or Weak Skills. 3. All questions except “classroom description” and “student description,” had at least one student
having “some skills” in student teaching EPG evaluation one or/and two evaluation. 4. There is an overall growth from junior field experience to student teaching experience.
Limitations
1. There are a few limitations to this analysis. First, the EPG instrument is not currently validated, and reliability is not established. Second, this is a purposive sample, and only complete evaluations (one junior field/grad practicum EPG, first and second EPG’s for student teaching) are included. For some questions, the percentages do not add up to 100% for the different categories; this is because there is at least one student for that question with a blank response. These missing data were not removed.
2. According to CAEP guidebook (page 135) Data Quality guidelines, it is important to discuss the following areas.
i. Valid and consistent (or reliable) ii. Relevant
iii. Representative iv. Cumulative (which includes “multiple measures”) v. Actionable
vi. Fairness, which means free from bias vii. Robustness referring to evidence that is a direct and compelling measure of the condition
intended to be informed
JF- Junior Field, GP-Grad Practicum, ST- Student Teaching
Question InTASC Standard INTASC Category PEPPS Standard Semester (group) # Little or No Skills Inconsistent or Weak Skills Some Skills Adequate Skills Competency JF-GP-Secondary Education Combined 35 0.00% 0.00% 11.43% 37.14% 51.43% ST-Secondary Education Combined-EVAL #1 35 0.00% 0.00% 2.86% 31.43% 65.71% ST-Secondary Education Combined-EVAL #2 35 0.00% 0.00% 0.00% 28.57% 71.43%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
2
Question InTASC Standard INTASC Category PEPPS Standard Semester (group) # JF-GP-Secondary Education Combined 35 ST-Secondary Education Combined-EVAL #1 35 ST-Secondary Education Combined-EVAL #2 35
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
3
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
4
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
5
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
6
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
7
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
8
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
9
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
10
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
11
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
12
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
13
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
14
InTASC Standard: 1,2,3,7,9 Insights on Effective Assessment
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
15
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
16
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
17
Secondary Education Combined EPG
Evidence of Professional Growth Data Analysis
Program Name: SPED Date: 1/17/2020 Participants: Semesters: Junior Field Data-fall 2016, fall 2017, and fall 2019 Student Teaching Data-fall 2018, spring 2019, and fall 2019
What do you notice?
What are the implications of data for program improvement?
Junior field ratings are done by Susan Gregory. Student teaching ratings are done by University Supervisors that may or may not have special education experience. 2016 data is missing for multiple measures. Scores from 2016, on the whole are low…2017 are high…2019 in the middle.
Can we disaggregate undergrad and grad for student teaching? Junior field is the first EPG they’ve done…in student teaching, they’ve had practice and feedback. Should special education be using a different assessment…this can’t be validated for special education. A new assessment could be aligned with PEPPS special education standards.
Staffing issues for supervision of undergrad special education student teaching.
No. Raters for student teaching may not have special education expertise, which would make this assessment invalid.
Create a new assessment aligned to PEPPS and/or CEC standards.
Train raters. Have raters that have qualifications in special education (endorsed and/or sped teaching experience)
field student receive for each EPG from their supervisors/professor?
EPG is taught in junior field class by faculty in special education. Supervisor is a special ed faculty person
who provides feedback on completed EPG.
What type of support (for example, feedback on unit/lesson plans prior to implementation) do student
teachers receive for each EPG from their supervisors/professor?
EPG 1
EPG 2 unknown because not all the supervisors have a background in special education.
SPED-Evidence of Professional Growth Data Analysis Faculty Questions.pdf
1-17-2020
EPG Data
• Overall percentages • SPED • Secondary • Elementary Junior Field
EPG Data • File Location • Box\ETP\MINUTES\2019-2020
Minutes\Data Retreat
EPG Instrument
• According to CAEP guidebook (page 135) Data Quality guidelines, it is important to discuss about the following areas.
• Valid and consistent (or reliable) • Relevant • Representative • Cumulative (which includes “multiple
measures”) • Actionable • Fairness, which means free from bias • Robustness referring to evidence that is a
direct and compelling measure of the condition intended to be informed
EPG Data
• Group Discussion • Locate the main report for the program • Electronically complete the “Faculty Questions
document” located in the corresponding program folder.
Whats Next? EPG
3. Consider adopting Bozeman’s tool
4. Improve our EPG
1. edTPA Readily available, specifically tailored to pre- service teachers, valid and reliable
High cost for the students
2. Modify and use Danielson Proprietary tool Validity and Reliability needs to be established Could be time consuming compared to option 3
3. Bozeman’s tool Already developed, aligned with Danielson and InTASC, Was the the tool developed for Inservice teachers? Inter-rater reliability is tested (CAEP Annual report) Validity -waiting to hear from Dr. Ann Ewbank
Reliability needs to be addressed Validity has to be established? Less time consuming compared to option 2 and 4
4. Improve MSUB EPG The instrument is already developed, was developed with the pre-service teachers in mind, also developed with partners and supervisors.
Have to meet the Evaluation framework for EPG created instruments by CAEP standards Validity and Reliability has to be established Time consuming compared to options 1,2, and 3
Milestone/Task Date
Draft Self Study Report June 2022 to September 2022
Internal Review October 2022-November 2022 Revise December 2022-January 2022 Submit Self-study Report March 2023 Next CAEP site visit Spring 2024 (Date is not announced
yet-assume January 1st)
Proposed timeline for EPG Modifications • Faculty decide regarding the EPG, based on the options provided • Faculty individually vote (electronically) by February 7th regarding the
individual decisions. • Based on faculty input decide the timeline for next steps
• Request faculty input on when and how the new instrument will be established
• Note: Based on the CAEP timeline, we should pilot the newly adopted EPG in Spring 2021. By August 2020, the instrument should be finalized so that validity and reliability testing can be conducted and items can be revised if needed before January 2021 (is this a realistic plan?)
Other Instruments
• Validity and Reliability of • ACP • Summative • Dispositions
• Note: Waiting to hear about state wide decisions regarding dispositions rubric at MCDE.
Spring 2020COE- Data Retreat
EPG Data
EPG Data
EPG Instrument
EPG Data
Slide Number 7
Slide Number 8
Other Instruments
Content Knowledge (MACK)
Teacher candidates completing an accredited Educator Preparation Provider (EPP) program in Montana must meet the minimum content knowledge requirements described below to be recommended for licensure/endorsement. (Source: OPI) http://opi.mt.gov/Portals/182/Page%20Files/Educator%20Preparation/MCDE/Praxis%20WC/ Montana%20Assessment%20of%20Content%20Knowledge.pdf?ver=2019-06-07-090733- 330
ACP Forms are located at https://www.msubillings.edu/coe/forms/index.htm (In Fillable PDF Evaluation Forms section)
Praxis exam
Montana Assessment of Content Knowledge Verification
• Teacher candidates must earn at least 7 points on the Montana Assessment of Content Knowledge prior to recommendation for licensure/endorsement by an accredited Montana EPP. The possible range for the Content Knowledge Score (CKS) is 0-10.
• Teacher candidates earning fewer than 7 CKS points or who score zero on any of the three rubric components shall not be recommended for licensure/endorsement.
• For candidates receiving a score of 1* on rubric components 1, 2, or 3, each Montana EPP will conduct a further individualized review of the candidate’s content knowledge and teaching skills, based on established policy, to ensure that the candidate merits recommendation for licensure/endorsement (Source: OPI)
Data Analysis
• Data analysis reports for Praxis data are located at Box\ETP\MINUTES\2019-2020 Minutes\Data Retreat\3-10-2020
• A folder is designated for each program (contains the praxis points analysis report as well as reports from ETS)
• Please respond to the the corresponding questions located in each program folder and save your responses in Box.
• ALL RESPONSES SHOULD BE TYPED IN THE QUESTIONNAIRE AND UPLOADED IN THE ORIGINAL FOLDER.
• ONLY ONE QUESTIONNAIRE PER GROUP SHOULD BE UPLOADED
Data Retreat Montana Assessment of Content Knowledge (MACK)
MACK (Montana assessment of content knowledge)
Content knowledge GPA
Praxis exam
Montana Assessment of Content Knowledge Verification
Data Analysis
Second Data Retreat 3-10-2020-Presentation.pdf
Any change that means the EPP no longer satisfies accreditation standards or requirements: 3.6 Change in regional accreditation status
3.7 Change in state program approval
Section 4. Display of Annual Reporting Measures. Annual Reporting Measures (CAEP Component 5.4 | A.5.4)
Impact Measures (CAEP Standard 4) Outcome Measures 1. Impact on P-12 learning and development (Component 4.1) 5. Graduation Rates (initial & advanced levels)
2. Indicators of teaching effectiveness (Component 4.2)
6. Ability of completers to meet licensing (certification) and any additional state requirements; Title II (initial & advanced levels)
3. Satisfaction of employers and employment milestones (Component 4.3 | A.4.1)
7. Ability of completers to be hired in education positions for which they have prepared (initial & advanced levels)
4. Satisfaction of completers (Component 4.4 | A.4.2)
8. Student loan default rates and other consumer information (initial & advanced levels)
4.1 Provide a link or links that demonstrate data relevant to each of the Annual Reporting Measures are public-friendly and prominently displayed on the educator preparation provider's website.
1 Link: https://www.msubillings.edu/coe/accreditation.htm
Description of data accessible via link:
This link provides accreditation reports and annual Title II and CAEP reports. There are links to eight annual reporting measures as well as data collection timeline at the bottom of this page.
Tag the Annual Reporting Measure(s) represented in the link above to the appropriate preparation level(s) (initial and/or advanced, as offered by the EPP) and corresponding measure number.
Level \ Annual Reporting Measure 1. 2. 3. 4. 5. 6. 7. 8. Initial-Licensure Programs Advanced-Level Programs
4.2 Summarize data and trends from the data linked above, reflecting on the prompts below.
What has the provider learned from reviewing its Annual Reporting Measures over the past three years?
Discuss any emerging, long-term, expected, or unexpected trends? Discuss any programmatic/provider-wide changes being planned as a result of these data? Are benchmarks available for comparison? Are measures widely shared? How? With whom?
We have designated an area on our web page (the link is posted above) to publish updates related to the reporting measures. Montana Educator preparation providers have been working together in a Continuous Improvement Collaborative (CIC) to develop a statewide protocol for a three-year data collection to support continuous improvement. As a part of the CIC, we have already administered the completer and employer survey. The third component of the CIC protocol is conducting Case Study. We used employer and completer survey findings to craft questions for the completer case study. We have already completed the case study research and currently analyzing the data. Please see the timeline created by the State related to the Continuous Improvement Collaborative. This timeline can be found on the above webpage.
We collaborated with the Montana Council of Deans to evaluate the MT State criteria related to advanced standards. The College of Education (COE) also worked on aligning advanced programs with SPA/STATE/CAEP standards and revise key assessments. By analyzing completer and employer survey data, we also found that our secondary education programs lacked diversity, classroom management, and assessment competencies. The Elementary and Secondary Program committee coordinated with the secondary programs to change the professional core for secondary programs. As a result of this, the professional core for secondary programs now includes the two credit Strategies for Managing Diverse Learners (EDU 343) course. The committee was
also able to officially change the professional core for secondary to EDU 381, the 3-credit curriculum course instead of EDU 380, a 2 credit course. EDU 381 has some assessment integrated into the course as part of the curriculum planning process and teaching cycle. In addition, the math and science teaching majors also added the assessment course (EDU 383) to their majors. English program agreed to change the English methods course from a two-credit to a three-credit class so that assessment could be more fully integrated into that course. The English department is also strongly encouraging English majors to select the reading minor as their teaching minor. The reading minor has several courses that emphasize assessment as part of the intervention process.
Section 5. Areas for Improvement, Weaknesses, and/or Stipulations Summarize EPP activities and the outcomes of those activities as they relate to correcting the areas cited in the last Accreditation Action/Decision Report.
Section 6. Continuous Improvement CAEP Standard 5
The provider maintains a quality assurance system comprised of valid data from multiple measures, including evidence of candidates' and completers' positive impact on P-12 student learning and development. The provider supports continuous improvement that is sustained and evidence-based, and that evaluates the effectiveness of its completers. The provider uses the results of inquiry and data collection to establish priorities, enhance program elements and capacity, and test innovations to improve completers' impact on P-12 student learning and development.
CAEP Standard 5, Component 5.3 The provider regularly and systemat

2021 epp annual report - msu billings

Documents