an investigation of student use of passpoint and nclex-rn...

L I P P I N C O T T F O R L I F E

White Paper

by Dr. Julia Phelan, National Center for Research on Evaluation,

Standards, and Student Testing, University of California-Los Angeles

SUMMARY

• Overall NCLEX pass rate was 94.59%, across all four study

schools.

• The national NCLEX pass rate for 2013 was 83.04%.

• For students who passed the NCLEX:

– The average quizzing Mastery Level (ML) was 3.05,

average exam ML was 7.52.

– Median number of quiz questions was 556 and

quizzes was 38.

– Median number of exams was 3, and median exam

ML = 7.78.

• As students answered more questions their quizzing ML

increased.

• As students took more practice exams, their exam ML

increased.

• HESI exit data (if available) was not associated with any

PassPoint usage/mastery variables.

Introduction

Lippincott NCLEX-RN PassPoint (Wolters Kluwer, 2013) was introduced to a set of trial users in spring and fall, 2013. PassPoint incorporates the features of the Lippincott NCLEX 10, 000 with some additional ways for students to practice and master content as they prepare for the NCLEX. Practicing in an authentic environment is an effective way to learn and PassPoint provides students the opportunity to take practice quizzes in an adaptive learning environment, as well as take longer NCLEX-style exams.

Instructors can use settings to create and deliver a comprehensive NCLEX assignment of 75 questions or more across all NCLEX Client Need categories. This assignment will mimic the NCLEX experience for students and help gauge their current preparedness for the actual NCLEX test. The simulated NCLEX exams in PassPoint follow the NCLEX test design blueprint are designed to resemble an NCLEX exam situation a student might find themselves in.

Previously, students could take quizzes with a maximum of 25 questions. In PassPoint, students are able to choose from a range of longer-length practice exams with the minimum of 75 and the maximum 265. These limits reflect the potential range in length of the actual NCLEX. It is important for students to gain experience and build stamina for test day—this includes answering a larger number of questions in one sitting which they are able to do in PassPoint. And students can take as many practice exams as they want, thereby

An Investigation of Student Use of PassPoint and NCLEX-RN® Outcomes

An Investigation of Student Use of PassPoint and NCLEX-RN Outcomes

2 Transforming Nursing Education and Nursing Practice Together

continuing to practice, build stamina and strengthen their nursing knowledge over time. Students don’t know how many questions they will get when they sit for the NCLEX, so it is crucial they have the opportunity to practice.

PassPoint also includes remediation links. These links take students to content related to their individual strengths and weaknesses as indicated by their score on the simulated NCLEX or their day-to-day quizzing. The remediation links provide students chance to review concepts which may be causing them difficulties, before taking more quizzes or practice exams.

Goals of the efficacy project

The goals of the NCLEX-RN PassPoint efficacy project are to help determine the efficacy and impact of using an adaptive learning tool (NCLEX-RN PassPoint) in NCLEX preparation courses. We do this by gathering information on student and instructor usage as well as related student achievement data. Specific questions are:

• How are student usage patterns in PassPoint related to their NCLEX and other course outcomes?

• How does performance on the simulated exams correlate to other predictor tests schools may be using (if applicable)?

• To what extent does use of PassPoint impact student learning, mastery of course material, NCLEX simulated exam performance and ultimate success on the NCLEX?

While many instructors and students provide qualitative feedback, the efficacy project aims to dig a little deeper and make inferences based on data analysis and measurable evidence from students and instructors using PassPoint in real-world educational contexts. Student engagement with the product is important, but we must also understand how PassPoint is being implemented and utilized by students. Only then can we begin to determine what’s working and how it is impacting student learning outcomes. Crucial to building this body of evidence is studying not only how students are using PassPoint, but also how use of PassPoint impacts exam performance along with other student outcomes. Data supplied by instructors collaborating on the efficacy study provides us with evidence we can analyze—along with usage and mastery data from PassPoint—to help address these questions.

PassPoint Usage and Mastery Data

PassPoint usage and mastery information allows us to explore patterns and frequency of usage as well as relationships between factors such as number of questions answered and mastery level (for

example). The next piece of the puzzle comes from both learning how instructors are implementing PassPoint and also gathering exam scores and grades from the relevant courses. Once we have these data we can start to examine how using PassPoint can impact student performance on myriad external measures and ultimately the NCLEX.

Within PassPoint students can both take practice quizzes as well as practice exams. Data analysis, therefore, is divided into student usage of each of these areas and in some cases (if appropriate) creating sum totals.

Study Sample

Instructors were recruited to participate in a trial use of PassPoint during the fall 2013 semester. Here we report data on four individual studies conducted at four different schools. Table 1 shows NCLEX pass rates and size of the student population at the four study schools. All available data are included and any differences in the data in Table 1 (related to format) reflects the institution-level data made available by the state boards of nursing.

The data in Table 1 reflect the overall pass rates at each of the study schools for 2012 and 2013. Important to consider here is that in 2013 the passing standard was changed (increased). The passing standard is defined as “A cut point along an ability range that marks the minimum ability level requirement. For the NCLEX, it is the mini-mum ability required to safely and effectively practice nursing at the entry-level,” (NCBSN). The NCSBN Board of Directors re-evaluates the NCLEX passing standard every three years when the test plan is reviewed. In 2009 the NCLEX-RN passing standard was changed from -0.21 to -0.16 logits, and then in April, 2013 the passing standard was raised again to 0.00 logits. As the passing standard has changed (in-creased) this means that a student must be at a higher ability level to achieve the passing standard.

When the passing standard changed (in the second quarter of 2013), the pass rate for first-time US educated test-takers was 83%, down from ~90% in 2012. This drop is not unexpected as, according to the NCSBN, pass rates are historically lower immediately following a passing standard increase. Typically, however, the rates rebound within 3 years.


Details of usage and findings for each study school are first presented individually, and in later sections, for a larger, combined data set.

School A: Study 1: Sample and Design

Course and students: The study sample students were enrolled in an NCLEX prep course. All senior students in the 8th semester for the first time are required to take the course which is intended as a review of the nursing program and to help students prepare for the NCLEX. In the fall, 2013 semester, 31 graduating seniors were enrolled in the course which comprised two sub-groups – traditional undergraduate baccalaureate students and second degree students who already had received a baccalaureate or higher degree. This course was graded as satisfactory or unsatisfactory (no letter grade).

Implementation of PassPoint: Students were assigned to take weekly practice quizzes, across the 37 topics under the Nursing Topics category, and required to meet a mastery level of 4 on each topic for a grade of satisfactory.

NCLEX PassPoint topics chosen for the earlier weeks of the semester were content areas identified as weaknesses from the mid-curricular HESI exam as well as a survey completed by the students at the beginning of the semester. Students took their Exit HESI exam (E2) on at the end of September, 2013. It was hoped that reviewing previ-ously weak content areas might help students on the E2. A score of > 850 on the E2 was required for successful completion of the senior capstone course.

At the end of the semester students were taken to the computer lab as a group and completed a 75-question practice exam. The outcome of this exam was not included as part of the semester grade.

Design: We submitted and IRB application and received IRB approval for the PassPoint efficacy study from School A. The study employed a retrospective, descriptive, correlational design to explore student us-age of PassPoint and associated outcomes. Two surveys were created and deployed to students via an online survey tool, and per the guidelines set forth by the IRB, completion of at least one of the surveys constituted consent to participate in the study. Completion rates for the surveys were low, with just twelve students completing the post-semester survey, and three the post-NCLEX survey.

PassPoint Quizzing Usage and Mastery

Overall usage and ML in PassPoint is shown in Table 2.

Students answered an average of 1714.67 questions (SD = 327.56) and achieved an average ML of 4 (SD = .38). Given the implementa-tion of PassPoint within the course, we would expect the ML variance to be lower than usual. This is, in part, because there were specific course requirements set around PassPoint ML and so we see a more uniform pattern overall.

The largest variance within this group was in remediation views. Students looked at an average of 11.75 remediation links with one student looking at 81 and four students looking at zero.

PassPoint Practice Exam Usage and Mastery

Table 3 shows student usage and ML for PassPoint practice exams. All students took at least one exam within PassPoint. The average num-ber of exams was 2.33 and the average number of exam questions 175. Average exam ML was 7.87. Given the large standard deviation for the number of questions, the median was calculated. Median number of questions was 75 (as was the mode)—with 10 students each taking one, 75 question exam.

For this course, practice exam usage overall was low. Only one stu-dent used the practice exam feature to any significant extent and this student took 15 practice exams. All other students (except one who took 3 exams) took only the one proctored practice exam (described above).

A Pearson product moment correlation was used to explore the relationship between PassPoint usage and subsequent ML (which is a reflection of the average question difficulty students are answer-ing correctly). There were no significant correlations within the usage and ML variables. This is likely because of the low variance in ML. If all students are around the same ML and seem to stop there, it is harder to differentiate between students which is typically easier with a different implementation/usage pattern.



Course Outcomes

Course outcomes for the student group are shown in Table 4. The average course grade was 78.83 (SD = 2.69), the average mid- curricular HESI score was 975.83 (SD = 94.81) and the average E2 score1008.58 (SD = 85.83).

HESI exam scores were positively correlated with one another, but not with any of the other course outcomes or ML data. Of the group included in this analysis, only one student scored below 850 on the mid-curricular HESI; on the E2, all but one student scored above 850. NCLEX Outcome

Ultimately all of the students in the fall 2013 cohort (the complete sample, as well as the 12 reported on here) passed the NCLEX on their first attempt.

Overall Summary

• 100% of students passed the NCLEX on their first attempt• Average quizzing ML was 4 (this was the required ML to pass the

course)• Average exam ML was 7.87• No correlation between ML and number of questions, likely due

to the low level of variation within the sample• Overall practice exam usage was low as was remediation

link usage

School B: Study 2: Sample and Design

Graduation Requirements: In order to graduate from School B, students are required to score at least 850 on the HESI E2. Students are able to take the exam up to three times. If after three attempts, students are unsuccessful, they continue remediation, but are not allowed to progress to graduation. In the preceding semester there were six students who had to take the HESI E2 for a fourth time prior to enrolling in another long semester.

Course and sample: PassPoint was implemented in a course for stu-dents in need of extra help/remediation. Students may have failed a

course, or not done well on their HESI exam, and are required to take it to help with remediation. Other students, however, self-select into the course. The course is a whole semester course and students can take it at different points in the program.

All students enrolled in the course were given access to PassPoint (the group is a mix of juniors and seniors). Access continued into the spring semester with an overall group of 43 students participating. One student dropped out of the program and was not included in the subsequent analysis. Overall PassPoint Quizzing usage for all students is shown in Table 5 and for PassPoint practice exams in Table 6.

The average ML was 2.61 (SD = 1.62). The median number of questions answered across the sample was 329. The median was calculated given that there was a very large standard deviation in questions answered with one outlier student who answered 9,634 questions.

While all but two students used PassPoint for practice quizzes, 62.8% of students did not use the PassPoint practice exams. Practice exam frequency is shown in Figure 1. Sixteen students took practice exams and of those who did, the average number of exams was 6.93 (SD = 1.71).


Figure 1: Frequency of Practice Exam usage


Students either took the course online, or in a traditional face-to-face instruction context. Across the sample, there were two students who logged in to PassPoint one time, but did not take any quizzes or prac-tice exams. Both of these students were in the online section.

Overall usage and mastery was compared between the online and traditional students. Comparisons revealed no significant differences in PassPoint quizzing or ML between the online group (N = 18) and the traditional group (N = 24). Nor were there significant differences in practice exam usage and ML between the two groups.

Students with NCLEX Outcome Data At the point of data collec-tion and analysis, only 15 students in the study group had taken the NCLEX and reported back results. This is because the course is open to students at both the junior and senior level and at the time of analysis, some had not yet completed the program, or had only re-cently completed it (and so no NCLEX data were available). PassPoint Quizzing usage for this NCLEX group (N = 15) is shown in Table 7.

One student in the group did not use PassPoint and so was eliminated from the analyses. It is interesting to note that one student who did not pass the NCLEX on the first attempt (but who was successful on the second attempt) was the one student who did not use PassPoint. Table 8 shows PassPoint usage and mastery for those students who used PassPoint and took the NCLEX.

Five of the students who took quizzes in PassPoint and passed the NCLEX, did not use the practice exam feature. Practice exam usage as well as remediation link views for those who did use the practice exam feature are shown in Table 9.

The average number of exams for this group was 4.67 (SD = 4.09) and average exam ML was 6.78 (SD = 2.22). The average number of remediation links accessed was 24.67 with the maximum of 96. The median number of remediation views accessed was 8. Three students did not access any of the links; one student accessed one link, and the rest between 8 and 96. The student who accessed 96 remediation links had the highest overall mastery and usage across both sections of PassPoint.

Of the 14 PassPoint users, all but one passed the NCLEX exam. Thirteen students passed on the first attempt, and one did not pass. PassPoint usage is shown for these two groups below (Table 10).

In prior studies we have explored the trends within PassPoint including the relationship between ML and usage. A Pearson product moment correlation revealed a significant positive correlation between number of questions answered and overall ML, r (14) = .913, p < .01 (see Table 11). Thus, as students answer more quiz questions, the difficulty level of the questions increases—indicating that their content mastery is increasing.



Overall Summary

• 92.9% of students (who used PassPoint) passed the NCLEX on their first attempt

• Average quizzing ML was 2.54• Average exam ML was 6.78 (among the 9/14 students who took

practice exams)• There was a strong positive correlation between ML and num-

ber of questions answered.• Students took an average of 4.67 practice exams and just over

half of the students who used PassPoint and took the NCLEX, viewed more than 8 remediation links.

School C: Study 3

At School C, PassPoint was implemented in the fall, 2013 with two groups of students:

• Group 1 (N = 12): Students who had already graduated and were given access to help prepare for the NCLEX. Students were part of a class and received assignments within PassPoint. They were encouraged to practice taking longer exams in the practice exam section.

• Group 2 (N = 12): These were students who received access to PassPoint during their Advanced Medical Surgical course.

We report here on Group 1 as this is the group who (to date) has taken the NCLEX. Overall PassPoint usage is shown in Tables 12 and 13. All 12 students took quizzes in PassPoint. The average number of quizzes was 153 (SD = 42.06), the average number of questions was 1062.75 (SD = 953.53) and the average ML was 3.18 (SD = 1.18). Three students answered over 2,000 quiz questions.

The average number of remediation links accessed was 7.25 with the maximum 29. Two students did not access the links at all and 75% of the group accessed six or fewer.

Table 13 shows student usage and ML within the practice exam area of PassPoint.

Two students did not take any exams within PassPoint and so data are shown for those students who used this feature. The average number of exams was 13.30 and the average number of questions 1439. Average exam ML was 7.39. Given the large standard devia-tion for the number of questions, the median was calculated. Median number of exam questions answered was 600.

NCLEX Pass Rates

To date 9 students in Group 1 took and passed the NCLEX on their first attempt. Two students used PassPoint to some degree but owing to personal circumstances (e.g., marriage and birth of a child) did not utilize the resource as they should have and also did not sit for the exam until 3-6 months after graduation. And the remaining student has not yet taken the exam. Delaying taking the NCLEX after graduation is frequently associated with poor outcomes and is discouraged by most nursing faculty. Students in Group 1 made use of PassPoint in the time after graduation and ultimately those who used the program at a meaningful level, passed the NCLEX on their first attempt. The instructor reported that students who did not use PassPoint “correctly” –were the students who ended up not passing the exam.

We compared usage and mastery in PassPoint based on NCLEX outcome (see Table 14).

Students who passed the NCLEX on their first attempt (N = 8) or second (N = 1) answered an average of 971.11 questions, although as the SD was large, the median (440) is a better measure. Average quizzing ML was 2.8 (SD = .62) and the average exam ML was 6.58



(SD = 2.51). Students in the group who passed the NCLEX accessed more remediation links than the group who did not pass. Because two of the students did not pass the NCLEX (at the time data were ana-lyzed) and continued using PassPoint to prepare for a second attempt, usage of the two groups reflects different amounts of cumulative time. Thus we might expect higher numbers of questions and mas-tery for those students who failed the first attempt and continued to use PassPoint. Indeed, one of the students who failed (after waiting 6 months) was still using PassPoint in June, 2014 (at the time when data were analyzed).

Overall Summary

• 81.8% of students (who used PassPoint) passed the NCLEX on their first, or second (one student) attempt

• Average quizzing ML of those who passed the NCLEX was 2.8.• Average exam ML was 7.4 (among the 8/10 students who took

practice exams and passed the NCLEX )• Students who used the feature took an average of 13.63

practice exams.• 7/8 students who used PassPoint and passed the NCLEX

accessed the remediation links. And the average number viewed was 9.13.

School D: Study 4

School D uses HESI Exit Exams as part of an NCLEX preparation strategy. Program policies at the study school require a HESI E2 exam score of 900 (in semester V) to graduate; multiple failures and repeti-tions can lead to low morale, frustration and drop out. School D made the decision to incorporate PrepU and PassPoint into their nursing programs as part of the NCLEX preparation strategy. PassPoint was offered to students starting in late spring 2013. During the spring, 2013 and fall, 2013 semester students in the study sample who had not yet taken the NCLEX were provided access to PassPoint. Usage of PassPoint was not required and students were given optional access and used it independently.

Usage of all features of PassPoint are shown below. The data in Table 15 show student use of both practice quizzing and practice exams within PassPoint.

Forty-two students used both features of PassPoint (quizzing and practice exams), and two students used the practice exam feature only. Usage was varied with some students taking very few quizzes or exams, and others displaying more frequent usage. The median number of PassPoint quiz questions answered was 420 and the aver-age quizzing ML was 3.03 (SD = 1.36). Students took a median of four practice exams and achieved an average exam ML of 7.64 (SD = .57). The variance in exam ML is very low, which is interesting given the large variance in number of exams taken.

Figure 2 below shows student usage of the longer-length practice exams. 40% of students took 1 or 2 exams only. Nine students took 10 or more practice exams, and one student took 30 altogether.

Figure 2: Number of practice exams taken by PassPoint users

Within PassPoint Correlations

We used a Pearson product moment correlation to explore the relationship between PassPoint usage and subsequent ML (which is a reflection of the average question difficulty students are answering correctly). There was a significant, positive correlation between PassPoint quiz ML and the number of practice quiz questions answered, r (40) = .681, p < .01, as well as PassPoint exam usage and mastery. This pattern indicates that those students with a higher quiz ML were more likely to have a high exam ML and higher level of usage overall.



HESI Outcomes

Of the 42 students who used PassPoint, 66.7% (N = 28) passed the HESI exam on the first try, 4 passed on the second attempt, 6 passed on the third attempt, and 4 students took the HESI a fourth time. Of these 4 students, only one passed the HESI. Thus, to date, not all students in the cohort were eligible to sit for the NCLEX.

PassPoint Usage and NCLEX Success

Of the 42 students who used PassPoint, NCLEX data were available for 37 students. Of this group, 36 (97.3%) passed the NCLEX. 35 stu-dents passed on the first try and one student passed on the second attempt. PassPoint usage is shown below for all students, grouped by NCLEX outcome or status.

Students who passed the NCLEX answered an average of 615.53 questions, although as the SD was large, the median (420) is a better measure. Average quizzing ML was 2.98 (SD = 1.42) and the average exam ML was 7.62 (SD = .61). Students in the group who passed the NCLEX accessed an average of 18.66 remediation links. As seen in Study 3, the student who did not pass the NCLEX had a higher cumu-lative usage (at the time of data capture). This pattern is likely due to continued usage following the first, failed attempt at the NCLEX.

Overall Summary

• 97.3% of students (who used PassPoint) passed the NCLEX on their first or second attempt

• Average quizzing ML of those who passed the NCLEX was 2.98.• Average exam ML was 7.62 (among the students who took

practice exams and passed the NCLEX )• There was a significant, positive correlation between PassPoint

quiz ML, number of questions answered and PassPoint exam usage and mastery

• For students who used the practice exam feature, the median number of exams was 3.5 (maximum 26).

• 6/36 students who used PassPoint and passed the NCLEX accessed the remediation links. And the average number viewed was 18.66 (maximum 51).

Analysis of all Student Data Combined

Data for students in the four schools (N = 80) were analyzed together to further explore patterns in student usage and ML. Overall group usage and ML is shown in Table 18. Students across the four schools took an average of 71.62 quizzes and achieved an average ML of 3.09 (SD = 1.23). The average number of practice exams taken was 5.83 (SD = 7.46), with an average exam ML of 6.88 (SD = 2.32).

Comparison between Schools

The number of quiz questions answered by students was found to be normally distributed for schools A, B, and C, but not for School D. Thus, we conducted a logarithmic transformation on the questions answered data, following which all data was normally distributed as assessed by Shapiro-Wilk’s test (p > .05). The same transformation was made for overall quizzing ML. ML for School A was very narrowly distributed given the requirement for students to reach a ML of 4 to succeed in the course.

Table 19 shows results of a comparison between the schools.



A one-way ANOVA was used to compare usage and ML between groups.

There was a significant difference in all variables presented in Table 19. The log transformations were used for quiz questions and quizzing ML. A post-hoc analysis revealed the following:

The information in Table 20 can be read in the following way (here we provide two examples): • for number of quizzes school A had significantly more than schools

B, C, and D• for number of practice exams, school C had significantly more than

schools A, and B.

Results of the one-way ANOVA are shown in Table 21.

NCLEX Outcome

Of the 80 students in the sample, 70 (87.5%) took the NCLEX and passed, 4 students did not pass, and at the time of data analysis, 6 had yet to take the exam. Of those students who took the NCLEX, the pass rate was 94.59%. Comparisons of usage and ML data for the group who have taken the NCLEX are shown in Table 22.

Given the wide distribution of number of quiz questions across the sample, median is a more appropriate measure. The median number of questions was 602, and median number of quizzes was 39. The low percentage of students who did not pass the NCLEX renders compari-sons between these groups inconclusive. Prior analyses have found patterns indicating that students, who are not successful on the NCLEX, continue to use PassPoint/PrepU to prepare and so cumula-tive usage can be much higher for those students.

Table 23 shows usage and mastery data for those students who passed the NCLEX. Median number of quiz questions for this group was 556.5 and the median number of quizzes 38.

Prior analyses have shown an average overall quizzing ML of around 4 for students who pass the NCLEX and have used PrepU NCLEX 10,000 or PassPoint as one component of their preparation strategy. Figure 3 shows the distribution of quizzing ML for those students who used PassPoint and passed the NCLEX. Note, data in this table include ML scores for 68 of the 70 students who passed the exam. Two students were excluded as they did not use the quizzing feature of PassPoint.



Figure 3: Distribution of ML for students who passed the NCLEX

Two students within the sample had very high quiz ML (7.3 and 6.8). Removing these student outliers, resulted in an overall average quiz-zing ML of 2.93 (SD = .98).

Not all of the 70 students who passed the NCLEX took practice ex-ams in PassPoint. Exam ML distribution, for those who took at least one practice exam (N = 65), is shown in Figure 4. Students who used the practice exam feature, and passed the NCLEX, had an overall practice exam ML average of 7.52 (SD = .98).

Figure 4: Distribution of Exam ML (students who took >1 exam, and passed the

NCLEX)

Within the group of students who passed the NCLEX, 65 took at least one practice exam, and 68 took at least one quiz in PassPoint. The distribution of ML for the practice exams was much narrower than was the overall practice quizzing ML. This pattern relates to the difference in ML reporting between the quizzing and the practice exam section of PassPoint. This difference is a result of the fact that students are answering many more questions on a practice exam and so the overall ML associated with an exam is weighted more towards the more difficult questions answered correctly.

Overall Practice Exam Usage

The number of exams taken by the NCLEX passing group (N = 70) is shown in Figure 5.

Figure 5: Overall exam usage for NCLEX Passing Group

The median number of exams taken was 3, with most students (N = 19) taking only one. Five students took no practice exams.

Within PassPoint Correlations

Using a Pearson correlation analyses we explored the relationship between PassPoint quizzing usage and mastery variables. A Pearson’s product-moment correlation was run to explore the relationship between the number of questions answered in PassPoint and overall ML (see Table 24). The analysis revealed a strong, positive correlation between the number of questions a student answered and overall ML, r (78) = .739, p < .001, as well as number of quizzes r (78) = .717, p < .001.



We also found a positive correlation between exam usage and exam ML (see Table 25), although the association was not as strong as that seen for quizzing. This pattern reflects that seen in the distribution of ML (discussed above). There is a much smaller variance in exam ML which makes it more difficult to differentiate students based on exam ML.

In both cases, as student usage increased, so did their ML. Thus we see as students answer more questions their mastery of the course material increases accordingly. This finding is in accordance with prior results where we have observed that as students practice more within PrepU, the difficulty of the questions they answer correctly increases as does their overall ML.

HESI Outcome

Most students in the sample (N = 65) took the HESI E2 (at least once) as part of their course requirements. The distribution of HESI scores across the group (from the three study schools which use the HESI E2) is shown in Figure 6.

Figure 6: HESI Score Distribution

The median HESI score was 989 and 20% of the students scored below 900.

Figure 7 shows the HESI E2 score for all students who took the NCLEX exam (and who had a HESI exam score) (N = 60).

Figure 7: HESI Score Distribution for NCLEX group

Of the students who took and passed the NCLEX and had HESI data (N = 60), 11.7% scored below 850 on the HESI E2, and 15% scored below 900. These results are interesting because some schools insti-tute policies to prevent students from graduating unless they reach a certain threshold score on the NCLEX. The typical threshold score is



850-900 and if such a policy were in place for the study population, 15% of students who did pass the NCLEX would not have met the criteria and been allowed to sit for the NCLEX in the first place.

We furthered explored the group of students who had a HESI score, used PassPoint and passed the NCLEX (N = 57). We were interested to see how much variance existed within the usage and mastery variables as compared to the final HESI score. Table 26 shows results of a Pearson product moment correlation between PassPoint usage and ML and HESI score. We used the final (highest) score, if students had taken more than one HESI E2.

Quiz ML was significantly, positively correlated with all other usage and mastery variables, but was not related to HESI exit exam score. Thus we saw that students with a higher quiz ML had answered a larger number of questions, r (57) = .708, p < .01; a larger number of quizzes, r (57) = .737, p < .01, and took more exams, r (57) = .347, p < .01.

Summary of Findings

Table 27 below shows data for all students who took and passed the NCLEX and who also used PassPoint. In situations where not all students used the practice exam feature, they are not included in the calculation of the average.

* Two students used PassPoint to some degree but owing to personal circumstances (marriage and birth of a child) did not utilize the resource as they should have and also did not sit for the exam until

3-6 months after graduation, and one student has not yet taken the exam. Thus this percentage pass rate may not be representative of the sample who utilized PassPoint as hoped.

School 1: All students in the study group made use of PassPoint, although they used the practice quizzing feature more than the prac-tice exams. Students reported that the large number of questions within PassPoint was an important feature, as was the ML feature as well as the adaptive functionality.

All students passed the NCLEX on their first attempt and had an av-erage quizzing ML of 4.00 and an average exam ML of 7.87. Exam ML is typically higher because of the way that the exam ML is calculated.

School 2: All of the students who used the PassPoint practice exam feature passed the NCLEX. The student who did not pass had very low PassPoint quizzing usage, quizzing ML of 1.2 and had no practice exam usage.

Of the students who passed the NCLEX and used PassPoint, the aver-age ML was 2.54 and the average exam ML was 6.78. This number is for students who actually used the practice exams (N = 9). The quiz-zing ML is lower than we often see, but with the introduction of the practice exam feature, students are dividing their quizzing (in some cases) between regular adaptive quizzing and the longer, NCLEX-style exams in PassPoint. The average exam ML for those passing the NCLEX was 6.78 (SD = 2.22).

In prior studies we have explored the trends within prepU and PassPoint and the relationship between ML and usage. A Pearson product moment correlation revealed a significant positive correla-tion between number of questions answered and overall ML, r (41) = .774, p < .001. There was also a strong positive correlation between number of chapters quizzed on and ML, r (41) = .610, p < .001. Thus, as students answer more quiz questions on a wider breath of topics, the difficulty level of the questions increases—indicating that their content mastery is increasing.

School 3: Students in Group 1 made use of PassPoint in the time after graduation and ultimately those who used the program at a meaningful level, all passed the NCLEX on their first attempt. The instructor reported that the students who did not use PassPoint “cor-rectly” –were the students who ended up not passing the exam. One of the students used PassPoint, but waited more than six-months to take the NCLEX exam. The other student who did not pass the exam cited test anxiety as the reason she did not pass. A third stu-dent failed the NCLEX on the first attempt and then began to use PassPoint to study for the second attempt (which was successful).

School 4: Of the 42 students who used PassPoint, NCLEX data were available for 37 students. Of this group, 36 (97.3%) passed the



NCLEX. 35 students passed on the first try and one student passed on the second attempt. There was a significant, positive correlation between PassPoint quiz ML and PassPoint exam usage and mastery

Higher ML scores reflect a greater command of more difficult ques-tions. ML has been shown in many of our other studies to significant-ly predict course outcomes such as number of total course points, or final grade (see Phelan & Smith, 2012; Phelan & Orm, 2012; Phelan and Phelan, 2012; Phelan & Phelan, 2011). In the current study we analyzed the relationship between the average Mastery Level of those students who passed the NCLEX and also used PassPoint in a mean-ingful way. Those students who passed the NCLEX and used PassPoint had an average Mastery Level of 3.05 +/- 1.19 and (in two of the studies) ML was significantly positively correlated with the number of questions students answered in PassPoint (Studies 2 and 4). In Study 1, students had a target ML of 4 to achieve. We see in situations with this type of implementation, very little variance in ML and so this finding is not surprising.

With a new product where students can choose to take practice quiz-zes as well as longer-length practice exams, exploring the connection between usage and mastery becomes more complicated as usage is effectively spread across two different areas—each with somewhat different goals. The practice quizzes serve to provide a practice and learning environment in which students take short, adaptive quizzes and answer questions on particular content areas (which they select). When students choose to take practice exams, they are engaging in a longer-length testing experience which helps to build stamina and endurance (useful on the day of the exam) and also allow them exposure to content across all NCLEX-tested client needs. And so when we are exploring usage and mastery outcomes, we have two measures of usage to consider as well as two mastery outcomes. The positive correlation between number of quiz questions and number of exam questions does, however, suggest students who have high usage in PassPoint practice quizzing, also answer more practice exam questions.

Limitations

Sample sizes in studies 1, 2, and 3 are small, although in studies 2 and 3 the sample size reflects the size of the cohort. In study 1, a consent form was required and although all students used PassPoint, not all students consented to be in the study. This limitation, however, is not unique in the nursing domain and in all similar studies we reviewed sample sizes tended to be similarly small. Indeed, in most of the senior-level nursing courses included in the broader efficacy study, class size tends to be between 18-25 students and larger sample sizes are less common. In contrast to the earlier PrepU and NCLEX Success White Paper (Wolters Kluwer, 2014) NCLEX outcomes were reported by faculty from participating schools and not students themselves—

thereby eliminating the self-report issue present in the earlier work.. A further limitation exists in that generalizability of the findings may be affected by potential differences in schools, curricula, content of courses, grading practices, student demographics and so on. Again, this is not a unique issue and is present in other work of this nature. We report correlational analyses in the report above and important to keep in mind is that correlations are measuring the strength and direction of a relationship between two continuous variables—for ex-ample number of questions answered in PassPoint and the number of points earned in a course, or ML achieved. What we are exploring with a correlation is the existence of a relationship between two variables, but we cannot make causal inferences based on the results. There are of course many unknowns in projects such as this one: Individual differences between students play a large role in student performance and there are not always pre/post data against which to measure growth or change. In addition, as any instructor will tell you, there will always be students who achieve a high grade without seeming to put in much effort and those who apply themselves with serious effort who somehow don’t make the grade. Many educational interventions seek to focus on those students between the two ex-tremes; in other words, those who are perhaps within reach of achiev-ing something more, if only they had some more motivation or extra help. We continue to collect more data and design and implement additional studies in ongoing efforts to address the above issues and further explore the efficacy of PassPoint with a larger representative sample.

Conclusions, Implications and Next Steps

Findings from the studies reported here are promising and confirm previous trends relating to usage and student outcomes in other subject areas (Phelan & Phelan, 2011). In these studies, however, we typically have access to course-level data (midterms, final exams, final grades etc.) and so are able to perform analyses investigating the relationship between use of PrepU, Mastery Level and course outcomes. The primary tool we use to generate predictions about student outcomes based on PassPoint performance is regression analysis. This is a technique that helps us to identify the relationship between a dependent variable (performance on an exam, or grade in the course, for example) and one or more independent variables (final Mastery Level, number of quizzes taken, etc.). We can then estimate a student’s performance based on their Mastery Level or one of the other variables.

In the NCLEX review courses we studied, this type of analysis was not always possible as typically the courses were graded on a pass/fail basis and almost all students pass the courses. Therefore, as with the NCLEX itself, there is little or no variation in student outcomes rendering it impossible to make such predictions. Operating within these constraints we explored the relationship between usage of



PassPoint—in multiple contexts—and subsequent success on the NCLEX. We can make no assertions as to causal relationships in the above analysis—in other words we can’t say that using PassPoint caused students to all pass the NCLEX. These findings do, however, reflect other findings in which PrepU ML for students who passed the NCLEX was ~4.0. Across all students who passed the NCLEX, the av-erage quiz ML was 3.05 +/- 1.19, and the practice exam ML was 6.98 +/- 2.97. In addition, when we make comparisons to other factors, such as HESI exam score, we see that ML is a more consistent factor. For example, the average quizzing ML for students using PassPoint, who passed the NCLEX, and had a HESI score was 2.93 +/- .98. For the same group, the average HESI score was 974.75 +/- 101.27. The above mentioned studies reflect early efforts to better under-stand student use of PassPoint to help prepare for the NCLEX and also the impact of usage on NCLEX success. As discussed earlier, the nature of the NCLEX data (pass/fail) and the high percentage of stu-dents passing the NCLEX on their first try renders analysis complex as there is little or no variation in student outcomes. Future endeavors will focus on securing more detailed data from students and faculty about NCLEX question numbers, pass rates, and prior student course performance. This information may help better predict students at-risk for failing the NCLEX and provide more insight into factors which may predict success. We will also continue to explore the extent to which PrepU usage influences student learning, mastery of course material, and ultimately student success on the NCLEX.

Note: We are grateful to all of the students and instructors who have participated in this study. Their cooperation and generosity in sharing both their experiences and data from their courses is invaluable in helping determine how PassPoint can best help students prepare for success on the NCLEX as well as a fulfilling life as well-trained nurse.

References

NCLEX Research Report. The NCLEX Delay Pass Rate Study. (2002). National Council of State Boards of Nursing, Inc

NCLEX Statistics (2012). https://www.ncsbn.org/Table_of_Pass_Rates_2012.pdf

NCLEX Statistics (2013). https://www.ncsbn.org/Table_of_Pass_Rates_2013.pdf

Phelan, J.C. (2012). An investigation of student use of PrepU and NCLEX success. Wolters Kluwer Health: Los Angeles, CA.

Phelan, J., & and Orm, J. (2012). Impact of Introductory Nursing Student Usage of an Online Adaptive Quizzing System: Case 2. Unpublished manuscript.

Phelan, J., & Phelan, J.C. (2012). Impact of Introductory Non-Majors Biology Student Usage of an Online Adaptive Quizzing System. Un-published manuscript.

Phelan, J., & Phelan, J.C. (2011). Improving Biology Mastery Through Online Adaptive Quizzing: An Efficacy Study. In J. Herman (Chair), Toward Formative Assessments Supporting Learning: Design, Validation, and Mediating Factor. Poster presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

Phelan, J., & and Smith, B. (2012). Impact of Introductory Nursing Student Usage of an Online Adaptive Quizzing System. Unpublished manuscript.

Wolters Kluwer Health: Lippincott, Williams & Wilkins (2014). Lippincott’s adaptive learning system. Retrieved from www.lww.com.

Wolters Kluwer Health (2014). Wolters Kluwer Health launches Lippincott’s NCLEX-RN Passpoint, offering student nurses the first fully adaptive and content integrated NCLEX-RN tool. Retrieved from www.wolterskluwerhealth.com.


an investigation of student use of passpoint and nclex-rn...

Documents