building testing committees that have the authority to create effective change
TRANSCRIPT
1
An ExamSoft Client Webinar
Building Testing Committees that have
Authority to Create Effective Change
Building Tes,ng Commi0ees that have authority to create effec,ve change
AINSLIE T. NIBERT, PHD, RN, FAAN
JANUARY 20, 2015
2
Why form a Tes,ng Commi0ee? Higher Educa-on Trends
• Increased demand for measurable achievement of program outcomes
• Surveillance for gaps in curriculum becoming an ongoing need
• Avoidance of inadequate/late response to academically at-‐risk students • Consequences affec?ng their admission, progression, gradua?on
• Increasing faculty re?rements – inexperienced faculty stretched thin • Curriculum evalua?on is a low priority un?l re-‐accredita?on looms • Less experienced faculty lack skills in test item wri?ng and exam analysis • Administra?on: tes?ng policies lack consistencies with their adop?on, maintenance, and revisions; policy-‐making should be supported by evidence
3
Tes,ng Policies & Procedures: Evidence-‐based & Consistent Well-‐wri6en test policies and procedures, carried out consistently, ensure:
• Defensibility of Tests used in each course across the curriculum • Test Blueprint aligns with Course Objec?ves • Standard scoring procedures are used for each test administra?on
• Policies are constructed using APA’s Code & are applied consistently • Sta?s?cal item analysis is systema?c and applied rou?nely to each test • Procedures for item nullifica?on, rescoring, second grader review are set up • Mechanisms for documen?ng test outcomes and analysis are established
• Consistent requirements established across all courses • Computerized reports used whenever possible to streamline effort for the faculty • Code of Fair Tes?ng Prac?ces in Educa?on by the American Psychological Associa?on applied
• See hSp://www.apa.org/science/programs/tes?ng/fair-‐code.aspx
4
A need for synergy: Tes,ng Policies rela,ng to Admission, Progression, and Gradua,on Tes-ng Commi6ee and Student Affairs Commi6ee – policy alignment
• The tes?ng commiSee’s recommenda?ons about tes?ng policy • Can overlap with policies adopted by the Student Affairs CommiSee
• Grading policies • Student grievance policies • Admission requirements • Condi?ons students must meet to be considered candidates for re-‐admission to the program
• Following academic failure of a course or courses • Following withdrawal from a course, or from the program, for any reason
• Progression requirements • Gradua?on requirements
The tes?ng commiSee does not formulate policy independently, but reports recommenda?ons to the curriculum commiSee for full faculty approval.
5
Curriculum evalua,on: Use aggregate student test data to find/close the gaps
• Total program evalua?on and curriculum revisions should be accomplished using an evidence-‐based approach that is systema?c. • Use aggregate student response data to establish benchmarks; track each cohort’s performance against benchmarks and compare to previous cohorts
• Use aggregate student response data gathered longitudinally from mul?ple cohorts to iden?fy weaknesses consistently seen by course or by concept across mul?ple courses; seek sources/causes of program weaknesses
• When curriculum evalua?ons are substan?ated by both faculty expert opinion (subjec?ve data) and aggregate student test results (scoring data, item analysis sta?s?cs that substan?ate test reliability and validity – objec?ve data,) the resul?ng revisions are specifically targeted at areas of program weaknesses • Saves faculty ?me by pinpoin?ng areas of concern & streamlining revisions • Completes the “evalua?on loop” -‐-‐-‐ just collec?ng the data is insufficient
6
Avoid the “too li0le, too late” response for the at-‐risk student Tes-ng commi6ees can make recommenda-ons about individual student interven-ons, but should not usurp the role of Student Affairs.
• Recommend updates to scoring benchmarks following data analysis • Create or adopt tools for longitudinal student data collected in each course • Require course coordinators to submit the datasets each semester • Establish remedia?on plans – create student learning contract templates • Formulate tools that summarize/document results following implementa?on of individual student ac?on plans to improve performance
• Coordinate with Student Affairs and Curriculum (Faculty) CommiSee • Recommend changes to Admission, Progression, and Gradua?on Policies
7
Commitment to adherence to Best Prac,ces in Tes,ng Ensure that tes?ng commiSee has proper representa?on and authority ◦ Representa?on across all levels of the program ◦ Sufficient resources to stay abreast of best prac?ces and elevate faculty skills
Make the commiSee a sub-‐commiSee of the curriculum commiSee or other faculty commiSee that represents the en?re faculty ◦ Policy and procedures recommended by the tes?ng commiSee ◦ But are approved by the overall faculty commiSee
Enforcement – item bank, exam bank, style guide, etc ◦ CommiSee must have an avenue through administra?on hierarchy to enforce policies established and carry out consequences when policies are ignored or thwarted
8
Best Prac,ce #1: Establish Tes,ng Commi0ee Design and revise the program’s tes?ng policies ◦ Publishes Guidelines for Exam Development ◦ Wri?ng style protocol – apply to all tests ◦ Cri?cal thinking items – increase Applica?on level
Exam Administra?on Policies ◦ Number of items/exam; length of ?me alloSed/exam ◦ Proctoring Guidelines ◦ Dissemina?on of Grades (When? How?) ◦ Test Review – to do or not to do; how to do it
…more Tes,ng Commi0ee Ac,vi,es Systema?c Item Analysis (see also #3 below)
◦ Use sta?s?cal parameters for analyzing overall test and individual test items
◦ Adhere to published psychometric standards ◦ Summarize analysis and include ac?on plan ◦ Review test blueprint as needed before next test ◦ Review individual items earmarked for aSen?on
Compile/analyze standardized test results ◦ Make recommenda?ons to curriculum commiSee for changes based on findings
10
Best Prac,ce #2: Wri,ng Style Protocol
Establishing clear guidelines for faculty leaves liSle room for ambiguity and helps insure uniformity in the presenta?on of exams throughout the curriculum: Will present or past tense be used for test items? Will op?ons end in periods, whether or not there are complete sentences? Will all op?ons begin with a capital leSer? When stressing a word in the stem, will it be highlighted, boldfaced, italicized, underlined? Will the term pa?ent or client be used?
Best Prac,ce #3: Increase propor,on of Applica,on-‐and-‐above Items
Which interven?on is most important? Which interven?on, plan, assessment data is/are most cri?cal to developing a plan of care?
Which interven?on should be done first?
What ac?on should the nurse take first? Which interven?on, plan, nursing ac?on has the highest priority?
What response is best?
12
Best Prac,ce #4: Use Uniform Sta,s,cal Parameters for Item Analysis
What is the minimum acceptable difficulty level for a test item? What is an acceptable discrimina?on level (PBCC) for a test item? What is the acceptable number of mastery items to include? What is an acceptable reliability coefficient for the exam?
13
Systema,c Item Analysis used for all tests: Use a 3-‐Step Method
1. Review Difficulty Level
2. Review Discrimina?on Data Item Discrimina?on Ra?o (IDR)
Point Biserial Correla?on Coefficient (PBCC)
3. Review Effec?veness of Alterna?ves Response Frequencies
Non-‐distracters
14
Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd ed.).Houston, TX: Health Education Systems, Inc.
15
Parameter Level of Acceptance
Item Difficulty 30%-‐90%
Item Discrimina?on Ra?o 25 % and above
PBCC (point biserial correla?on coefficient)
0.20 and above
KR20 (Kuder-‐Richardson 20) 0.70 and above
Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd ed.).Houston, TX: Health Education Systems, Inc.
Use these Recommended Item Analysis Standards
PBCC & KR-‐20: Standards of Acceptance for Nursing versus General Educa?on exams 16
PBCC: 0.15 and above KR20: 0.60 -‐ 0.65 and above
Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd ed.).Houston, TX: Health Education Systems, Inc.
Analyze Student Response Frequencies Target revisions of op?ons with ‘0’ responses These op?ons are not plausible; even poor performers know enough to avoid these choices on the exam, which increases prospect for success by guessing
When an op?on is chosen by fewer than 30 test takers, this generally indicates poor item discrimina?on and should be edited. If fewer than 30 test takers took the test, frequencies are not reliable; cumula?ve data should be used.
17
Summary: Tes,ng ‘Best Prac,ces’ Improve Test Blueprints
Adopt a uniform item style/format; publish the guidelines
Invert the propor?on of Applica?on-‐and-‐above test items as compared to Knowledge/Comprehension items
Use a Systema?c Item Analysis methodology
Include 10% alterna?ve item formats – across all tests
Remove items that no longer align with na?onal standards or current clinical prac?ce, which should be reflected in your program outcomes ◦ In Nursing: Remove obsolete nursing diagnoses ◦ In Nursing: Eliminate Trade names for most medica?ons (use generic only)
For competencies that must be tested in each course, be consistent: ◦ In Nursing: 10% of all test items are medica?on calcula?ons – across all tests in all courses
Revise the test length/?me allotments – be consistent across all tests
21
….more Tes,ng ‘Best Prac,ces’ Establish scoring benchmarks that are evidence-‐based/outcome-‐based
Determine consistent weight of standardized exam in overall grade
Iden?fy consequences for failure to achieve benchmarks ◦ Offer addi?onal (parallel) versions of the exam – mix these up! ◦ Allow enough ?me for remedia?on to occur between aSempts. ◦ Assign specific remedia?on ac?vi?es (mul?ple strategies)
◦ Date/?me for submission of these should be specified in the learning contract. Monitor remedia?on usage for student adherence to contract.
◦ Selec?ons should be based on weak areas iden?fied on scoring report. ◦ Focus on building confidence – this is not a puni?ve ac?vity. Model effec?ve study habits and test
taking strategies.
Use proctoring guidelines; take test security measures ◦ Inves?gate suspected breaches and enforce published policies.
22
A few words about vigilance with Test Security 1. Encourage moral behavior (Academic honesty program at your school with clear
language placed in handbooks)
2. Discourage chea?ng a. Before tes?ng
1. Minimize access to exams and viewing of exam content 2. Use highest levels of security available in LMS for teacher-‐made tests
and all security features available in a standardized tes?ng plaqorm -‐ protect logins & access codes; ac?ve dashboarding
3. Train proctors for live proctoring ac?vi?es; re-‐train/re-‐cer?fy regularly b. During tes?ng
1. Establish secure environment 2. No devia?ons to test procedures or breakdown of environmental
security allowed. Ex: leaving room equates to the test being over for that student regardless of reason
3. Vigilant proctoring (proctor physically walks around the room or rou?nely accesses ac?ve dashboards for remote tes?ng
3. Detect chea?ng with Data Forensics and take ac?ons as needed
23
References American Psychological Associa?on. (2004) Code of Fair Tes?ng Prac?ces in Educa?on. Washington, DC: Joint CommiSee on Tes?ng Prac?ces. hSp://www.apa.org/science/programs/tes?ng/fair-‐code.aspx
Morrison, S., Nibert, A., & Flick, J. (2006). Cri$cal thinking and test item wri$ng (2nd ed.). Houston, TX: Health Educa?on Systems, Inc.
Na?onal Council of State Boards of Nursing. (2013) 2013 NCLEX-‐RN test plan. Chicago, IL: Na?onal Council of State Boards of Nursing. hSps://www.ncsbn.org/3795.htm
Nibert, A. (2010) Benchmarking for student progression throughout a nursing program: Implica$ons for students, faculty, and administrators. In Capu?, L. (Ed.), Teaching nursing: The art and science, 2nd ed. (Vol. 3). (pp.45-‐64). Chicago: College of DuPage Press.
Schroeder, J. (2013). Improving NCLEX-‐RN pass rates by implemen?ng a tes?ng policy. Journal of Professional Nursing, 29(2), S43-‐S47.
Sewell, J., Culpa-‐Bondal, F., & Colvin, M. (2008). Nursing program assessment and evalua?on: Evidence based decision making improves outcomes. Nurse Educator, 33(3), 109-‐112.
24