evolution of institutional capacity to support the assessment-change-effectiveness cycle

Evolution of Institutional Capacity to support the Assessment-Change-Effectiveness Cycle

Course Content & Design

Implementation Examples

Assessment Strategiesby Dr. Greg Witteman, U. Guam

The big questions• What are students’ abilities before taking the class?

– What Knowledge, Skills, or Abilities (KSAs) should the students enter the class with?

• What are the major objectives of the class?– What KSAs should the students leave the class with?

• What is the role of assessment in course design or delivery? – Can outcomes be improved? (and how would we

know?)

Type of Course Content

• Self contained Content (no prerequisites)– Examples: Anatomy and Physiology I, and II

• Prerequisite knowledge needed– Examples: Conservation Biology and

Comparative Vertebrate Biology labs

Abilities (student) Objectives (Course) Assessment (results)

Self contained course content

• Anatomy and Physiology I & II

– Assumes no prior knowledge – Each part of the course builds on knowledge

gained earlier in the course (hierarchy)

Chem. Cytology Histology Organs Systems


Course content assumes KSAs

• e.g. Without general biology, math and ecology species population size, distribution and probability of extinction can not be meaningfully presented

E.S policy Defining C.B. Demography Methods and tools

General Biology

Ecology

Envtl. Sci. Eco.

Lab Math

Cons.Bio.

Previouscourses


Course Design and Delivery Strategies

Course content – (what type of subject matter and students?)

Course goals – (what do we hope to accomplish?)

• What should students be able to do to finish the course?

• What should they know after the course ends?

– Delivery and Assessment methods • (how and when do we know if we accomplished our goals ?)

The Ranges of KSA’s and goals for Science Courses

No Knowledgelimited

background

Have Knowledge from: H.S.,

College, other

Lower Division non-Sci. Majors

Lower Division Science Majors

Upper Division Science Majors

Remember Understand

Remember Understand

Analyze Apply

Create

Entry KSAs

Courses

Course goals


Example 1: Anatomy and Physiology• Non-science majors (freshmen/sophomore)

– Extreme range of educational and cultural backgrounds

• Goal: “Remember and Understand”: – Content heavy (terminology for structures must be learned before

subsequent structures, function & concepts can be presented)

• Two semester course, concurrent lab. (one grade)• Large lecture hall, no blackboard (PowerPoint and

overheads only). Typical lab facilities.• No pre-requisites or assumed knowledge


Original Course Delivery model

Lectures Labs

Assignments

Quizzes Tests

Student outcomes (the scores)


Concepts build on those previously presented in class.

Students must leave course with knowledge and ability in every topic presented (which will be needed for subsequent courses and professional certification)

Advantages and Disadvantages(assessing this approach)

Lectures Labs

Assignments

Quizzes Tests


Pro: consistent delivery of material (Year-to-year)

Con: can not adapt to differences in student learning (can leave gaps in knowledge)


Problems with this approach


Changes in content delivery (availability & access): Addition of supplemental materials on web and in library, adopted new texts.

Addressing these problems (adding adaptive assessment TO THIS approach)

• Weekly adaptive quizzes– Goal = insure that content is covered and retained by

“recycling” or re-emphasizing poorly understood topics (subsequent content adapts to knowledge gaps indicated by objective measures of student performance).

• Pre-post tests (entry assessments, midterms and final)– Goal = overall course improvement by evaluating: successful

presentations of individual topics, retention of material after completion of course (i.e.:of first semester material in second semester)


Adaptive Assessment

Quiz 1

30 Neurons 3040 Neuroglia 5030 Function 20

Results 1

70%80%60%

Quiz 2

40 Neurons 4060 Neuroglia 400 (N. Function) 20

Needsreview?

Review


Assessment “feedback” loop

Pro: an adaptive approach can be used to close the gaps in student knowledge

Con: takes time and effort by instructor to implement.

Note: this can be used for both self contained and pre-requisite knowledge courses


Course Changes: Technology classroom (NIH-RISE)

• Interactive media & Video on demand – Dissections, models, background & supplemental

information

• View PowerPoint lectures

• Print lecture notes and outlines

• Automated practice quizzes and tests – (Web-based, secure (CGI),with feedback (in TC only)

• Equal access to course materials

Example 2: Comparative Vertebrate Biology

• Upper division biology majors – Juniors/Seniors (pre-med, pre-vet, pre-graduate school)

• “Remember, Understand & Analyze”– Content and concept intensive

• Lecture & lab (20 students)

• Assumes basic understanding of lower-division general biology, basic laboratory skills, understanding of correct terminology.


Course design & delivery (lab)

• Lab exercises with questions – (non-adaptive, “you must know this…” and “how is this structure similar to …”).

Assumes good study habits and skills.

• Student presentations of lecture summaries in lab– Identify difficult concepts with need for more in-depth coverage (identify gaps in

knowledge)

– Create participatory classroom dynamic

• Lab documentation projects– Students create class materials (self documenting review materials)

– “problem based learning”


Lectures Labs

Assignments

Quizzes Tests


NOTE: Student self-assessment/feedback is within lab prior to tests

Course changes: NIH-Rise-TC

• Visual review (Digital Video) of in-lab procedures, dissections, and specimens – A picture is worth a thousand words, and a video is

worth a thousand pictures!

• Equal access to information resources – (“go find everything you can about…” on the

web.)

Learner-centered approaches to assessment/improvement implemented

• Human Anatomy and Physiology– Use of adaptive weekly quizzes to “fine-tune” delivery of

individual lectures (year-to-year), guarantee a sufficient knowledge foundation on which to build additional concepts (within the semester)

– Entry and Exit assessments provide the information necessary to measure success in meeting the course goals

• Comparative Vertebrate Anatomy Lab– Use of self and peer assessment to identify and fill gaps in

knowledge, skills and abilities before they are formally tested.

Application of NIH-Rise Technology Classroom capabilities

• Increases in information resources availability– Online resources (web)

– Class notes, outlines and presentations.

• Dynamic content capabilities – Self serve video (TB server capable of 600+ hours of MP1

video)

– Interactive content (practice exams, data exploration)

• Provide all the Information Technology tools needed for “Problem based learning”

• …and, let’s not forget Technology Development support and training for faculty

The bad news

• There is no free lunch– Incorporating adaptive assessment (in A&P)

requires an additional 3 hours each week to quiz and lecture preparation and delivery (50% increase after initial development of course materials)

– Adding self and peer assessment (2 hours additional planning and setup for TC use).

Still to be addressed...• Why did I do this?

– Student outcomes poor relative to objective standards

• Why should others do this?• How can others incorporate these types of

assessment features in their courses?– Drink more coffee

– Give up sleep

• Conclusion: still some implementation obstacles

evolution of institutional capacity to support the assessment-change-effectiveness cycle

Documents

course content anatomy

course ends

course builds

students abilities

adaptive assessment

presentedtwo semester

assessment methods

role of assessment