creating a teacher education assessment system david c. smith, dean emeritus college of education...

Creating aTeacher EducationAssessment System

David C. Smith, Dean EmeritusCollege of EducationUniversity of FloridaEmail: [email protected]

David C. Smith Creating a Teacher Education Assessment System

2

Why We Need an Assessment System

We need to do a better job of assessing what we are doing and how it can be improved.

Our assessment systems need to accommodate a paradigm shift from assessing teaching to assessing learning.

We need to move from collecting data to implementing an assessment system.


3

We need to be seen as the sources of the best information available about the preparation of professional educators.Each SCDE needs an assessment system.One assessment system should be capable of addressing multiple demands for data.An effective assessment system can help to increase confidence in the preparation of professional educators.


4

There are Multiple Demands for Information

Internal - advocacy, analysis, and planning.Making a case for resources.

How do you compare with other units in your institution?

Analyzing units within your SCDE.How do you compare with “significant others”?

Who are “significant others”?

Annual Reports.


5

External - Accreditation and evaluation.Accreditation - NCATE and regional.Reports to the State Department of

Education.Reports to the State Higher Education

Board.Title II information.


6

The pressure is very real, and

it is a high stakes environment.


7

Considerations in Building an Assessment System.

The baseline is very important. That may become your standard. (Neither easy or trivial. High tide and low tide.)

Historical data are important insofar as they lead to an understanding of your current needs and direction. (The Past is Prologue. National Archives.)

Collect information that is likely to be valuable and relevant avoid the temptation to collect data which are not meaningful but readily available. Trivial information may be readily available. Have a reason for each item in your assessment system.


8

Further Considerations.

Consider your audience. (Bob and Gene show.)

A system that is not current is not of much value. (It must be kept up to date.)

It is imperative that your data be accurate. (The price is lost credibility.)

You get big numbers by adding little numbers. (Institutional research.)

Collect data elements at least one level lower than the unit. (College down 10%.)


9

Considerations. - (Still more.)

Data are inextricably related to resource allocation and the demonstration of productivity. (Without data – what did you do with what you got? Summer budget allocation.)

The first line of defense is to question the accuracy of the data or to ignore dissonance producing data. (Use mine until you have better numbers.)

Try to use data that are difficult for others to reject. (I got the information from your office.)


10

Considerations. - (Even more.)

Avoid asking for data which require calculations. (Avoid introducing errors.)

Precise measures are better than crude measures; crude measures are better than approximations; and approximations are better than no data at all. (Important variables may be difficult to measure.)

You can make better approximations for your unit than anyone else. (Yo0u need not apologize for reasonable approximations.)


11

Considerations. - (Still more.)

Do not expect sharp changes over a short time interval. (You can’t drive an 18-wheeler like a sports car. Complex organizations change slowly.)

An assessment system faces conflicting demands; the ability to compare and the need to tailor. (Data from comparable institutions are powerful. Yet your system must fit your needs.)

The system must be capable of easy modification.The assessment system must be capable of supporting a rapid response.


12

Considerations. - (Even more.)

People will attend to those things that are measured, rather than those things which are said to be important. (Achievement tests.)

It doesn’t make sense to think you need to measure with a micrometer if you are going to mark with a piece of chalk and cut with an axe. ( If it is important. Quantitative economics.)

Use data to make your problem important to others. (Abolish elementary education.)


13

Principles. (Enough – at last.)

A single system is much more powerful and useful than multiple data sources.Information is power. You need to be the best informed persons on the campus with respect to matters relating to the preparation of professional educators. (An assessment system can help you accomplish that goal.)


14

Data can help us get our act together.


15

The institution almost assuredly collects data.

Those data signal what the institution values.

The institution can hardly question the accuracy of the data it collects and reports.

Those data can be studied and interpreted. What do these data suggest about your

SCDE? Some of these data can be imported into

your assessment system.


16

What do we mean by the term “assessment system”?

A carefully designed and functional system for collecting, organizing, maintaining, analyzing and utilizing meaningful information.A system is made up of sub-systems.To be effective, the system must be incorporated into the regular operation of the SCDE.


17

A Performance Assessment System involves utilizing

information to support vital activities of the SCDE.

AnalysisAssessment Implementation


18

What kinds of data might be included in a Teacher

Education Performance Assessment System©? (Establishing a conceptual

framework.)Resource Data

Productivity Data

Candidate Performance Data

Faculty Performance Data


19

Structure of a Teacher Education Assessment System©

First Symbol - Classification√ A = Resource Data√ B = Productivity Data√ C = Candidate Performance Data√ D= Faculty Performance Data

Second Symbol - Level√ 1= Unit Data√ 2= Institutional Data√ 3= Calculated Data

Third Symbol - Category√ Sub-systems

Fourth Symbol - Number√ Element Number


20

TEAS© System and

Sub-systemsA. Resources

1. Unit

1. Human Resources

2. Fiscal Resources

3. Demographic Data

2. Institution

1. Human Resources

2. Fiscal Resources

3. Demographic Data

3. Derived

B. Productivity

1. Unit

1. Enrollment

2. Loads and Assignments

3. Program

4. Title II Data

5. External $ Generated

6. Student Demographic Data

2. Institution

1. .Enrollment




3. Derived

1. Enrollment





21

TEAS© System and Sub-systemsC. Candidate Performance

1. Unit1. Initial Program Admission2. Early in Program3. Mid-program4. Summative5. Graduate Follow-up6. Advanced Programs

Admission7. Mid-program8. Summative9. Graduate Follow-up

2. Institution1. Undergraduate2. Graduate

3. Derived1. Undergraduate2. Advanced Programs

D.Faculty 1. Unit 1. Teaching Evaluations 2. Scholarly Productivity 3. Professional Service 4. Institutional Service

2. Institution 1. Teaching Evaluations 2. Scholarly Productivity 3. Professional Service 4. Institutional Service

3. Derived 1. Scholarly Productivity 2. Institutional Service


22

Kinds of Information in TEAS.AACTE – NCATE Annual Report data.

Demographic information. A.1.3.1 Number White A.1.3.2 Number Black A.1.3.3 Number Hispanic A.1.3.4 Number Asian A.1.3.5 Number Native American A.1.3.6 Number Pacific Islander A.1.3.7 Number Eskimo A.1.3.8 Number Other A.1.3.9 Total Minority 0 A.1.3.10 Total Faculty 0 A.1.3.11 % Minority Faculty

#DIV/0!


23

Examples of Use - Resource Information.

Section A - Resource Data -- TEASUnit Data – Resource 1999

2000Human ResourcesA.1.1.1 FTE Faculty - Full TimeA.1.1.2 FTE Faculty - Part TimeA.1.1.3 Summer FTE FacultyA.1.1.4 Total FTE Faculty 0

A.1.1.5 FTE AdministrationA.1.1.6 FTE Support Personnel


24

Comparative Salary Information.

Section A - Resource Data -- DerivedA.3.1 Instr. $ Diff. Inst. – Unit #DIV/0!A.3.2 Asst. Prof. $ Diff. Inst. – Unit #DIV/0!A.3.3 Assoc. Prof. $ Diff. Inst. – Unit #DIV/0!A.3.4 Prof. $ Diff. Inst. – Unit #DIV/0!A.3.5 All Ranks $ Diff. Inst. – Unit #DIV/0!

This information can be tracked over time.


25

Productivity Data.

Section B - Productivity Data -- UnitEnrollmentB.1.1.1 FTE Enroll. Undergrad.B.1.1.2 FTE Enroll. Grad.B.1.1.3 Total FTE Enroll. 0B.1.1.4 Headcount Enroll. UndergradB.1.1.5 Headcount Enroll. GradB.1.1.6 Total Headcount Enroll. 0B.1.1.7 Total Undergrad. SCH B.1.1.8 Total Grad. SCHB.1.1.9 Total SCH 0B.1.1.10 Summer SCH Undergrad.B.1.1.11 Summer SCH Grad.B.1.1.12 Total Summer SCH 0


26

Data for Staff Planning.

A.1.2.18 % Instructors #DIV/0!

A.1.2.19 % Asst. Prof. #DIV/0!

A.1.2.20 % Assoc. Prof. #DIV/0!

A.1.2.21 % Prof. #DIV/0!

A.1.2.22 Avg. Instructor #DIV/0!

A.1.2.23 Avg. Asst. Prof. #DIV/0!

A.1.2.24 Avg. Assoc. Prof. #DIV/0!

A.1.2.25 Avg. Prof. #DIV/0!

A.1.2.26 Avg. All Ranks #DIV/0!


27

Budget Analysis.

A.1.2.27 Consumable ExpensesA.1.2.28 EquipmentA.1.2.29 Library BudgetA.1.2.30 Travel BudgetA.1.2.31 Graduate AssistantsA.1.2.32 OtherA.1.2.33 Total Budget

$0A.1.2.34 Faculty Support Factor

#DIV/0!


28

Equity in the System – Over Time.

A.3.10 Unit % Faculty #DIV/0!A.3.11 Unit % Summer Faculty #DIV/0!A .3.12 Unit % FTE Support Personnel #DIV/0!A.3.13 Unit % Part-time Faculty #DIV/0!A.3.14 Unit % Consumable Expenses #DIV/0!A.3.15 Unit % Equipment #DIV/0!A.3.16 Unit % Administration #DIV/0!A.3.17 Unit % Library #DIV/0!A.3.18 Unit % Travel #DIV/0!A.3.19 Unit % Budget #DIV/0!


29

Monitoring the System.

A.3.20 Unit % Faculty Change #DIV/0!A.3.21 Inst. % Faculty Change #DIV/0!A.3.22 Unit Total Budget % Change #DIV/0!A.3.23 Inst. Total Budget % Change #DIV/0!B.3.2.9 Unit % Change Total SCH #DIV/0!B.3.2.10 Inst. % Change Total SCH #DIV/0!

Patterns become apparent over time!


30

Using Productivity Data - Examples.

B.1.2.7 Hours Taught/FTE Faculty #DIV/0!B.2.2.4 Hours Taught/FTE Faculty #DIV/0!B.1.2.8 Sch/FTE Faculty #DIV/0!B.2.1.10 Sch/FTE Faculty #DIV/0!B.1.3.1 Cost/SCH #DIV/0!B.2.2.5 Cost/SCH #DIV/0! Compare productivity over time!

B.1.3.2 $ Prepare Elem. Teacher Approx.#DIV/0!

B.1.3.3 $ Prepare Sec. Teacher Approx.#DIV/0!


31

Candidate Performance Data.

Section C - Unit -- Candidate Performance Data

Basic ProgramsAdmissionC.1.1.1 SAT/ACTC.1.1.2 Writing SampleC.1.1.3 GPAC.1.1.4 Interview


32


Multiple Measures

Early in ProgramC.1.2.1 Praxis IC.1.2.2 Educ. GPAC.1.2.3 Classroom Obser. - Knowledge. C.1.2.4 Classroom Obser. – Application

Clear Expectations


33


Mid-ProgramC.1.3.1 Total GPAC.1.3.2 Tutorial KnowledgeC.1.3.3 Tutorial ESLC.1.3.4 Small Group KnowledgeC.1.3.5 Small Group ESL

ESL = Evidence of Student Learning.


34

Candidate Performance Data.SummativeC.1.4.1 Final GPAC.1.4.2 GPA in Non-Educ. CoursesC.1.4.5 PortfolioC.1.4.6 Student Teaching GradeC.1.4.7 No. Taking Praxis IIC.1.4.8 No. Passing Praxis IIC.1.4.9 Praxis II Success Rate

#DIV/0!C.1.4.10 No. Taking State License Exam.C.1.4.11 No. Passing State License Exam.C.1.4.12 State License Exam. Rate

#DIV/0!C.1.4.13 No. Candidates Rec. Elem. Lisc.C.1.4.14 No. Candidates Rec. Sec. Lisc.C.1.4.15 Total No. Candidates Rec. Lisc. 0


35

Candidate Impact Data.

How important are schools, colleges and departments of education?

C.1.4.16 Potential Candidate Impact

=(D345*25*25)+(D346*125*25)

Movers and shakers. One years’ graduating class.


36


Graduate Follow-UpC.1.5.1 Prog. Assessment - GraduatesC.1.5.2 Graduate Assess. - Principals


37

Faculty Performance Data.

Professional ServiceD.1.3.1 No.State Wrkshps, ElemD.1.3.2 No. State Wrkshps, SecD.1.3.3 No. State Wrkshps, Total 0D.1.3.4 Elem. Tchrs. @ State Conf.D.1.3.5 Sec. Tchrs. @ State Conf.D.1.3.6 Total Tchrs. @ State conf. (PS) 0D.1.3.7 Potential Student Impact 0D.1.3.8 No. State Wrkshps/Faculty

#DIV/0!


38

Much of this information is important only to deans.

NOT SO!

Without information of this kind the dean cannot effectively serve as an advocate for the unit.

Much of the information can be generated only by the faculty and program heads.

Often other units are able to make their data more compelling.


39

Relationships Among Assessment Systems

Program

Department

Unit

Remember, you get big numbers by adding little numbers.So you may need to have sub-unit assessment systems.

Institution


40

Kinds of Data

Top Down DataSCH

Budget Data

Enrollment Data

FTE Faculty

FTE Students

Student Demographics

Most of these data can be acquired.

Bottom Up DataFaculty performance data.Candidate performance

data.Student performance data.

Most of these data can only be generated at the program level and aggregated to the unit level.


41

Focus on Faculty PerformanceCritical Check Points

Admission (GPA – Essay – Interview)Knowledge SkillsDispositionsEvidence of student learning – N.A.

Early Program Assessment (Classroom Observation – Participation)Knowledge SkillsDispositionsEvidence of student learning – N.A.

Mid-Program Assessment (Diagnosis - Tutorial)Knowledge SkillsDispositionsEvidence of student learning


42

Critical Check Points

Exit Assessment (Student Teaching Evaluation – Tests)

Knowledge - Skills - Dispositions Evidence of student learning

Follow-Up Assessment (Principal Evaluation – Graduate Survey)

Knowledge - Skills - Dispositions

Evidence of student learning


43

Critical Check Points

KnowledgeSkill Disposition Student Learning

AdmissionGPA XXXXXEssay XXXXX XXXXXInterview XXXXX XXXXX

Early Program AssessmentClassroom Observation XXXXX XXXXX XXXXXClassroom Participation XXXXX XXXXX

Mid-Program AssessmentDiagnosis XXXXX XXXXX XXXXXTutorial XXXXX XXXXX XXXXX

Exit AssessmentPortfolio XXXXX XXXXXTest XXXXXStudent Teaching Eval. XXXXX XXXXX XXXXX

Follow-Up AssessmentPrincipal Evaluation XXXXX XXXXXSurvey Data XXXXX XXXXXContinuing Contract XXXXX XXXXX XXXXX

Consider: Comparability of data among programs.Aggregation of candidate performance data.The need for explicit criteria.The need for rubrics and scaling.


44

Attributes of this system.(Exclusive of Title II Data)

366 pieces of information are contained in the system. 216 pieces of information (59%) are entered. Many of the data are already routinely collected. It is very likely that the institution and unit currently

collects some of the data.

150 pieces of information (41%) are generated.Consider the kinds of information generated. Comparative data within the system. Comparative data over time – multiple years. Evaluative data. Data for planning.

Graphs and tables can be developed from the data in this system. (Enrollment and Budget, % of Faculty and Budget, % of Grad. SCH and Budget)


45

What You Have and What You Need

0

20

40

60

80

100

120

140

160

180

Prob. Have Prob. Need Derived

Series1


46

What You Have and What You Need (v1.0)©Section Prob. Have Prob. Need Derived TotalA.1 35 1 15 51A.2 34 1 15 50A.3 0 0 23 23B.1 32 4 20 56B.2 26 4 16 46B.3 0 0 25 25C.1 18 19 4 41C.2 4 1 0 5C.3 0 0 5 5D.1 13 5 11 29D.2 0 18 11 29D.3 0 0 6 6Total 162 53 151 366Percent 44.3% 14.5% 41.3% 100.0%


47

This System and Your System

The data in this performance assessment system can be used for: Unit planning, advocacy,annual reports and budgeting. Institutional planning and evaluation. External advocacy and information. Accreditation, program approval and review.

This system is easily modified.Remember, you do not need to create a performance assessment system annually.The entire assessment system does not need to be put into operation simultaneously.


48

Your Assessment System.

Think about the framework for your system.All data elements do not need the same baseline.You can implement your assessment system systematically and gradually.An assessment system can help make informed decisions about improving your programs.There is a relationship between resources and productivity. Make unconventional comparisons. (Compare SCH/FTE with candidate evaluations of programs over time.)


49

Your System

This system is available to you.

Email: [email protected]

But you need your own system.

This system is easily modifiable.

There are advantages in having some commonality in data sets - within the state, region and nationally.


50

Remember

The burden of proof rests with the unit.

You do not need to measure everything.

You need to present sufficient information for the BOE to make a reasoned and reasonable judgment.


51

The Real Question is.....

Does your assessment system provide sufficient information to permit you and others (in this case NCATE) to draw accurate and reasoned judgments about the quality and performance of your teacher candidates, faculty, unit and the context in which you function?


52

Closing thoughts.

Our ability to serve as effective advocates is essential.

No effective case can be made without data for support.

We must document the level of our effectiveness in a much more sophisticated manner. That requires an assessment system.

We need to become much more proactive in portraying the teacher education enterprise.

creating a teacher education assessment system david c. smith, dean emeritus college of education...

Documents

assessment systems

effective assessment

data elements

historical data

state department of

state higher education

trivial information

best information available