Selecting and Developing Assessment Approaches and Methods

Selecting and Developing Assessment Approaches and MethodsPresented by Jennifer FagerXavier University forUniversity of Wisconsin-SuperiorEnhancement Day1/19/2011

Guiding PrincipleThe assessment of student learning should become a system whereby planning, data collection, analyses, and improvement are includedReactions?Burning QuestionsWhat sorts of things should a professor try to assess?Having decided on what to assess, how should a professor go about assessing it?What sorts of things should faculty try to assess in their programs?Having decided on what to assess, how should faculty go about assessing these things?

What is your Assessment Pattern?Why do you assess what you assess?

What are the reasons professors construct and use assessment instrumentsIdentify areas of deficiencyUnderstand end-of-instruction targets

Data-Driven AssessmentProfessors make inferences and then decisions based upon the inferencesWhat does an A mean? B? C? F?THUS It is important to clarify BEFORE the test is created the decisions that will be influenced by students performances.

How do you do that?Its time to give your mid-term. Before you do so, there are several questions that need to be addressed.What should be tested?What topics were discussed and for how long?What type of items will you use?How long will students have to take the exam?How many items/points should be used given the amount of time available?

Two Fundamental QuestionsWhat evidence do you have that students achieve your stated learning outcomes?

In what ways do you analyze and use evidence of student learning?Another Question or twoWhat changes have you made to your programs, your institution, or your courses based upon evidence collected?What evidence do you currently possess that might inform essential decisions that need to be made?Defining EvidenceInformation that tells you something directly or indirectly about the topic of interest Evidence is neutral -- neither good nor badRequires context to be meaningfulTwo types of assessment evidenceDirect and Indirect

Direct EvidenceStudents show achievement of learning goals through performance of knowledge, skills:Scores and pass rates of licensure/certificate examsCapstone experiences Individual research projects, presentations, performancesCollaborative (group) projects/papers which tackle complex problemsScore gains between entry and exit Ratings of skills provided by internship/clinical supervisorsSubstantial course assignments that require performance of learningPortfoliosCourse assignmentsOthers?Indirect EvidenceAttitudes, perceptions, satisfaction, and experiences of learning and the learning environment

Students self-assessments of learning Local student, alumni, employer surveys and questionnaires Course Evaluations. National Engagement Surveys and Satisfaction Surveys NSSE, CCSSE, FSSE, BCSSE, SSI (Noel Levitz)Focus Groups (student, faculty, employer) Interviews (student, faculty, employer)Others?

Finding Evidence: An Evidence InventoryLets you discover the evidence you already have, such as:Institutional Research dataStudent Life dataExit Surveys (seniors)Alumni SurveysStart with the obvious but dont stop thereFinding Evidence: Perils and PitfallsInstitutional historyWeve already done that, and it didnt tell us anything!Territory; PoliticsFighting for scant resourcesInstitutional policy/culture about sharing informationI dont want somebody policing my classrooms!Who owns the evidence?Finding Evidence: AppropriatenessDoes the evidence address student learning issues appropriate to the institution or the program?Does the evidence tell you something about how well the institution or program is accomplishing its mission and goals?The questions you have about student learning should guide your choice of appropriate existing evidence and identify gaps where a new type of evidence might be neededEvidence ExampleAttached to this packet are data and analysis examples. Writing Results RubricIR SurveyCAAP resultsStudents Will Think Critically FormStudent Affairs dataLibrary DataAssisting Academic Departments: Some AssumptionsFaculty are intensely interested in what students are learningAssessment occurs in classrooms and academic departments every dayEvidence of student learning already exists in academic departmentsThe challenge is not to convince academic departments to gather evidence, but rather to help them recognize and use evidence they already have

Assisting Academic Departments: Addressing Common BarriersThis is a lot of work!Use some sort of evidence inventory to help faculty understand how existing academic practices yield evidenceKeep expectations reasonable, given limited time and resourcesOffer assistance and rewards Remember: it is not necessary to gather all the evidence all of the time

Assessment Inventory: One ExampleInventory of Written Statements and PlansDo you have a written mission statement or statement of purpose? yes no If yes, please attach a copy or reference where this can be found: ________________________________________________________Do you have a written statement of intended educational outcomes describing what a student should know or be able to do when they have completed this program? yes noDo you have a written method of assessment for measuring student outcomes? yes noDoes your program have a separate accreditation process? yes no

Assessment Inventory: One ExampleDirect Methods of Assessment1. ________ Comprehensive Examinations2. ________ Writing proficiency Examinations3. ________ National Examinations assessing subject matter knowledge4. ________ Graduate Record Exam General Test5. ________ Graduate Record Exam Subject Test6. ________ Certification Examinations7. ________ Licensure Examinations8. ________ Locally developed pre-test or post-test for subject matter knowledge9. ________ Major paper/project10. ________ Program/course portfolios11. ________ Capstone coursework12. ________ Audio/video tape of presentations/performances

Assisting Academic Departments: Addressing Common BarriersHow do I know you wont use this against me?Be consistent and firm in the message that assessment is not faculty evaluation, that results will only be reported in the aggregatePartner with faculty willing to engage in the process and make her/his evidence publicLink assessment results to allocation of resources, ideally through a strategic planning processIf appropriate, develop policies regarding assessment

Assisting Academic Departments: Addressing Common BarriersMy students pass the tests. Why isnt that good enough?Tests often measure only content knowledge Learning = what student know (content knowledge) + what they can do with what they know (performance)Grades are generally not linked to specific learning outcomes and dont aggregate well Modify course tests to measure learning outcomes by adding performance assessmentsModifying Tests to Gather Direct Evidence of LearningIdentify questions on the test that provide evidence of a learning outcome:Five questions that require the use of deductive reasoning to arrive at the right answerOpen-ended questions that require students to solve a unique problem given knowledge/skills learnedIsolate those questions and look for patterns of performance:the average grade in the class was a B but 85% of the students missed four of the questions requiring deductive reasoning70% of students were able to use a particular theory/approach to resolve the problem

22Meaningful EvidenceSituated within the institutional and departmental mission and contextAddresses relevant questionsAnalyzed and interpreted in relation to other evidenceExamples?23Meaningful Evidence: Facts + Context Fact:National survey data indicates seniors do not feel a sense of engagement and belonging on our campus.Meaningful Evidence: Facts + ContextFact: Seniors feel disengaged from our campus (national survey data)Fact:Seniors would recommend this institution to other people (senior exit surveys)25Meaningful Evidence: Facts + ContextFact:Seniors feel disengaged from our campus (national survey data)Fact:Seniors would recommend this institution to other people (senior exit surveys)Context:Over the past five years, an average of 82% of first-year alums donated to the institution26This example of pulling together evidence from a variety of sources is at the institutional level and tells us something about how seniors feel about their educational experience. We can do the same sort of thing for academic departments.Recognizing Meaningful EvidenceHow compelling is your evidence?Does it make you want to do something? Will it make others want to do something?How relevant is your evidence?To what is it linked: departmental mission, institutional initiatives?How trustworthy is your evidence?How was it gathered? Who does it represent? Is it one piece? Several pieces?

HLC Expanded Fundamental Questions What evidence do you have that students achieve your stated learning outcomes?* Who actually measures the achievement of student learning outcomes?* At what points in the curriculum or co-curricular activities are essential institutional (including general education), major, or program outcomes assessed?* How is evidence of student learning collected?* How extensive is the collection of evidence?

In what ways do you analyze and use evidence of student learning?* Who analyzes the evidence?* What is your evidence telling you about student learning?* What systems are in place to ensure that conclusions are drawn and actions taken on the basis of the analysis of evidence?* How is evidence of the achievement of student learning outcomes incorporated into institutional planning and budgetingMeaningful Evidence: ExampleSenior exit surveys:Indicate a dissatisfaction with the amount of time spent on clinical skillsDepartmental assessment of skill ability and development finds that, of the critical skills required:students are outstanding on three of them, satisfactory on two, and not acceptable on twoInternship evaluations from supervisors consistently cite lack of ability in clinical skills 29Meaningful Evidence: Qualitative DataAppropriate uses:Exploring an issue in more depthAnswering specific questions about individual experience:Ex: How are you different now than you were before?Ex: how did living with a host family inform your understanding of the culture?Including student voices

Qualitative Data Analysis: Open-Ended QuestionsRead the dataStrip and code the data, while looking for themes and patternsPresent the data thematically---it will lead you somewhereAcademic AdvisingGeneral EducationStudent perceptions of particular courses

Qualitative Data Example 420 was a senior level course but I felt like a freshman! There was no way I knew all of that stuff.I thought I was going to fail 420 and Im a good student.I didnt know how to do anything in 420 and the instructor didnt care. We kept saying we didnt know but he just kept going. It was ridiculous.

Qualitative Data Example Drill down into the data by asking pertinent questions:What are the learning goals of 420?How did students perform in 420?What are the assumptions about students entering 420?Skill level?Knowledge base?Analyze the program curriculum map Where do students learn prerequisite skills and/or knowledge?How and where are program and course learning outcomes (expectations) assessed? Are they assessed?Leads back into the assessment cycle in a much deeper way because youre linking this information back to concrete student learning assessment processes.

33Using Assessment ResultsInform policy decisionsStrategic allocation/reallocation of resourcesMake changes in curriculumSupport new initiativesAccountabilityInform stakeholders about expectations and results Improve teaching and learning on campus34Presenting Assessment ResultsConsider audienceWho are they? Whats important to them? How will they use assessment information in their lives?Appropriate presentationPresent data thematically/topicallyLink data and interpretations to institutional initiatives or departmental strategic planning (provide a context)

35Assessing and Improving AssessmentWere the assessments reasonable and manageable?Did they answer your questions?Did they tell you something about student learning?Were you able to use the evidence you gathered?What else do you need to know?Questions and CommentsWhere do you need to go from here?What is your assessment system for the program in which you teach?Is the system understood by all stakeholders?Does the system reflect the discipline?Helpful SourcesDiamond, Robert M. Designing and Assessing Courses & Curricula (1998)Allen, Mary J. Assessing Academic Programs in Higher Education (2004)Huba, Mary E. and Jann E. Freed. Learner-Centered Assessment on College Campuses (2000)Suskie, Linda. Assessing Student Learning: A Common Sense Guide (2004)Walvoord, Barbara E. Assessment Clear and Simple (2004)