using measurable outcomes to evaluate tutor programs jan norton, presenter
TRANSCRIPT
Using Measurable Outcomes to Evaluate Tutor Programs
Jan Norton, Presenter
IntroductionJan NortonEmail: [email protected]: (319) 499-1372Master’s in Educational Research& Psychology
25+ years as director of multi-discipline tutoring centers with CRLA certifications
Former reviewer for CRLA tutor training certification; current NADE certification reviewer for tutoring, SI, and developmental education programs
NCLCA Lifetime Learning Center Leadership certification
Webinar ContentsTerminologyThe Black BoxTimetablesData/MeasurementData/Measurement CautionsProgram Outcomes & ExamplesStudent Outcomes & ExamplesClosing
TerminologyVisionsMissionsValuesGoalsObjectivesOutcomesProgram vs. Student
OutcomesLearning Outcomes
The Black Box
Something
Happens
The Black Box
Something
Happens
What is going on in there? How do we measure it?
TimetablesScope of assessment
◦≥ 6 years◦2 years◦1 year◦1 semester◦1 course◦1 assignment◦1 tutoring session
TimetablesPlanning for Assessment
◦Continual processes◦Multiple times per
semester◦Once or twice per
semester◦Two or three times per year◦Once per year◦Every two years◦Every 4 years
Data/MeasurementWhy Measure
What to measure◦Before (Benchmarks)◦During◦After (Outcomes)
Data/MeasurementHow to measure
◦Program-generated data◦Course-specific data◦Campus data◦Campus surveys◦Standardized tools◦Locally developed tools
Data/MeasurementMore about locally developed
tools◦Surveys◦Likert-type scales◦Explicated scales◦Behavioral rubrics◦Self assessment by student◦Tutor assessment◦Observer assessment
Data/Measurement Cautions
Direct vs. Indirect MeasuresData access and consistencyRealistic limitationsCooperationThe Baseline ChallengeTime & money
(and staff and comfort and ……)
Chat Time for Questions & Contributions
Finished the foundation: what we can and can’t measure, when & how to measure, and what data we will gain
Moving on to examples of program-level outcomes
Program OutcomesPossible Areas of Assessment
Focus◦Establishing baselines◦Usage◦Satisfaction or quality◦Level/variety of services◦Staff development◦Reaching certification or meeting
standards
Program Outcomes ExamplesOutcome Statement
◦As a result of the new marketing strategy, the number of students attending group tutoring sessions will increase by 10% in Spring 2013.
Timetable◦One semester
Starting Measure (Benchmark)◦Average of past attendance
Ending Measure ◦Workshop attendance records
Program Outcomes ExamplesOutcome Statement
◦During 2012-13, the number of tutors who earn CRLA Level I certification will increase by 5%.
Timetable◦One academic year
Starting Measure (Benchmark)◦Number of tutors achieving Level I in
previous year(s)Ending Measure
◦Number of tutors achieving Level I in 2012-13
Program Outcomes ExamplesOutcome Statement
◦Outside reviewers using the NADE self-study process will score the tutoring program at 85% or above on all sections.
Timetable◦One year
Starting Measure (Benchmark)◦ Internal review & self-study
Ending Measure◦External review
Program Outcomes ExamplesOutcome Statement
◦As a result of tutor training sessions, tutors will demonstrate an improved understanding of Bloom’s Taxonomy.
Timetable◦One semester or year, depending on
training scheduleStarting Measure (Benchmark)
◦Pre-testEnding Measure
◦Post-test
Chat Time for Questions & Contributions
Finished program-level outcomes
Moving on to examples of student outcomes
Student Outcomes: MeasuresQualitative
◦Interviews & focus groups◦Opinion surveys◦Observations or self reports
Quantitative◦Assignment & course grades◦Instrument scores◦Observation rubrics
Student Outcomes: TypesAffective
◦Build confidence◦Improve attitude◦Reduce anxiety
Cognitive◦Mastery of material◦Conscious strategies use
Student Outcomes ExamplesOutcome Statement
◦After being tutored for at least one hour, clients will be more confident in their ability to accomplish the academic task they focused on in tutoring.
Timetable◦One tutoring session
Starting Measure (Benchmark)◦Client self-report
Ending Measure ◦Client self report
Student Outcomes ExamplesOutcome Statement
◦ Students who work with the study skills tutors will demonstrate an increased ability to use memorization strategies.
Timetable◦One tutoring session
Starting Measure (Benchmark)◦Tutor or observer checklist
Ending Measure◦Tutor or observer checklist
Student Outcomes ExamplesOutcome Statement
◦75% of the students who work with a reading tutor for at least two sessions will score at least 75% on the main ideas reading test.
Timetable◦Within three weeks
Starting Measure (Benchmark)◦Reading quiz at start of tutoring
Ending Measure◦Chapter reading test in class
Student Outcomes ExamplesOutcome Statement
◦Students who attend at least 10 group tutoring sessions for Algebra Review 090 will earn an average grade of B or above in the course.
Timetable◦One semester, length of course
Starting Measure (Benchmark)◦Placement score into course
Ending Measure◦Course grade
Student Outcomes ExamplesOutcome Statement
◦After attending mandatory tutoring for all 5 of their developmental education courses, students will be retained until the start of their junior year at a rate at least 25% higher than developmental students for whom tutoring was not required.
Timetable◦Two years
Starting Measure (Benchmark)◦Developmental course placement
Ending Measure◦Enrollment level
Student Outcomes ExamplesOutcome Statement
◦Students attending SI for World Religions will “[apply] previously understood information, concepts, and experiences to a new situation or setting.” (CAS)
Timetable◦Within one semester
Starting Measure (Benchmark)◦Pre-survey
Ending Measure ◦Post-survey
ClosingCheck with your institutional
research or assessment officeReview early drafts of
outcomes statements with your supervisor
Pre-test any locally created measurement tool
Be prepared for outcomes that are not met or wildly exceeded
Closing
Remember the black box: we do not own or control the box, and our ability to see into it is very limited
Questions & Contributions?
