the effects of various mastery criteria on student performance and attitude in a mastery-oriented...

The Effects of Various Mastery Criteria on Student Performance and Attitude in a Mastery-Oriented Course

Robert A. Reiser Marcy P. Driscoll Dale S. Farland Adriana Vergara Martin C. Tessmer

Robert A. Reiser, is a professor, Marcy P. Driscoll is an associate professor, and Dale S. Farland, Adriana Vergara, and Martin C. Tessmer are graduate students in the Department of Educational Research, Develop- ment, and Foundations, College of Education, Florida State University, Tallahassee, FL 32306.

The purpose of this study was to determine the effects of various mastery criteria on student performance and attitude in a course in which mastery learning strategies were employed. Undergraduates in an introductory course in educational psychology were randomly assigned to one of three treatments- one in which mastery criteria gradually increased from 70% to 90%, a second in which mastery criteria gradually decreased from 90% to 70%, or a third in which mastefl/criteria remained constant at 80%. Results indicated that although the high mastery criterion [90%] had a positive effect on some aspects of quiz performance, it did not have an effect on final examination performance. Results also indicated that students preferred that mastery criteria remain constant during a semester. These findings, when examined in light of previous research, call into question some prior notions regarding the levels at which mastery criteda should be set.

ECTJ, VOL. 34, NO. 1, PAGES 31-38 ISSN 0148-5806

In describing the origins of the mastery learning strategies he and his associates de- veloped, Benjamin Bloom (1976) has stated, "Basic to this work was the problem of defining what was meant by mastery on an achievement test" (p. 4). The issue of how mastery shall be defined is still an important one today.

Many people involved in the mastery learning movement have advocated the use of high standards in courses in which mastery learning strategies are employed. For example, Keller (1968) has indicated that 100% should be set as the cut-off score for identifying mastery level performance on a unit quiz. Others have suggested that a unit mastery requirement of 90% may be opti- mal (Block & Bums, 1976; Sherman, 1974), and some have recommended the use of high standards without specifying a par- ticular criterion level (Hursh, 1976; Johnson & Ruskin, 1977; Jacobs, 1983).

Some researchers have found that performance on comprehensive examinations (i.e., examinations covering the contents of several units) is better among students who are required to attain high standards on unit quizzes than among students who are required to meet lower standards. Semb (1974) found that a high mastery criterion (100% correct) resulted in better examination performance than did a low mastery criterion (60% correct). And Block (1972) found that a 95% criterion level produced better performance than lower standards. Results of other studies (Davis, 1975; Johnston & O'Neill, 1973; Johnston & Pen-

32 ECTJ SPRING1986

nypacker, 1971) indicate that quiz performance may be optimized when a 90% criterion level is set; however the effect of this standard on comprehensive examination performance was not examined in these studies.

Although there is some evidence to indicate that student learning may be improved when the standard for passing unit q-lzzes is set at relatively high levels, there is also some evidence to the contrary. Hochstetter and CaldweU (1977) found that students who were required to attain quiz scores of at least 90% performed no better on a final examination than did students who were required to attain lower scores (ranging from 60% to 85%). In two of the four comparisons they made, Parsons and Delaney (1978) obtained similar results: Students faced with either a 70% or 80% mastery criterion performed as well on an examination as did students faced with a 90% criterion. In another study (Burkman & Brezin, 1981), it was found that students who were required to pass unit tests at "medium ex- pectation levels" (70% correct on relatively easy material; 50% on relatively difficult content) performed better on a final examination than did students who were required to meet higher standards (80% and 60%). The students faced with high standards performed no better on the final examination than did students who were required to meet low standards (60% correct on difficult items and 40% correct on easy items).

Research studies examining the effects of various mastery criteria on variables other than comprehensive examination performance have also produced conflicting findings. Results from some studies indicate that quiz performance may be optimized when a 90% criterion level is set (Davis, 1975; Johnston & O'Neill, 1973; Johnston & Pennypacker, 1971). However, Carlson and Minke (1975) found that students who were required to attain 80% on unit quizzes, as well as students who were faced with a critierion level that increased from 60% to 90% over 15 units, performed better on quiz7es than students who were faced with a constant 90% standard. Carlson and Minke also found that students who were required to attain 90% on each unit quiz

progressed through the course at a slower rate than students in the other treatment conditions. There were no differences, however, in the percentage of students who withdrew from each treatment group. In contrast, Hochstetter and CaldweU (1977) found a 90% standard did not adversely affect student progress rates but did lead to more student withdrawals.

In light of the conflicting results obtained in previous studies, this study was conducted to determine the effects of various mastery criteria on student performance and a~ tude in a course inwhich mastery learning strategies were employed. Three treatment groups were used in the study. In one group, the criterion for mastering o unit quiz gradually increased from 70% correct to 90% correct. In a second group, the criterion gradually decreased from 90% to 70%, and in a third group the criterion remained constant at 80%. This design enabled us to compare the effects of three criterion levels--70%, 80%, and 90%--and to bal- ance the treatment conditions so that the average mastery criterion for each group was the same.

The design used in this study also enabled us to compare the effects of ascending, descending, and fixed criteria. In most mastery-oriented courses, the criterion for mastering a unit quiz remains constant, or fixed, across all units. There is some evidence, however, to suggest that by gradually raising mastery criteria over a series of units, instructors may shape student behavior and thus find that student quiz performance improves and student progress rates increase (Carlson & Minke, 1975). On the other hand, Carlson and Minke indicate that the study habits students develop in a course may depend upon the mastery criteria they are required to attain on early units. If criteria are set high initially and are decreased gradually, student performance may not slacken and more positive student attitudes may be engendered.

METHOD

Subjects Subjects in this study were 121 under- graduate students enrolled in an introductory course in educational psychology at

MASTERY CRITERIA 33

Florida State University. Most of the students in the course were juniors and seniors majoring in education. Students were randomly assigned to one of the three treatment conditions.

Course Structure The course in this study incorporated many of the features typically found in mastery- oriented courses. Instruction in the course consisted of 15 self-instructional units. Stu- dents were given a list of objectives for each unit and purchased the instructional materials needed in order to attain those objectives. The materials included two textbooks (Dick & Carey, 1978; Gagn6, 1974) and a study guide prepared by the instructor. Lec- tures were not given in the course; however, the instructor and two graduate as- sistants were available to work with the students on an individual basis.

Quizzes consisting of multiple-choice and short-answer items were used to assess student attainment of the objectives for each unit. The students could take these quizzes during any of the regularly sched- uled testing times. Many of the quiz items required students to classify or generate examples of concepts they had read about. For example, students were asked to write test items for given instructional objectives and were asked to identify the types of learning outcomes (Gagn6, 1974) the objectives exemplified. There were usually 15 to 20 items on a quiz.

Students were required to master one unit before proceeding to the next one. In order to get credit for mastering a unit, a student had to score at or above the mastery level his or her group was required to attain on quizzes for that unit. Students who failed at their first attempt to demonstrate mastery of a unit were required to take al- ternate versions of the quiz for that unit until their scores met or exceeded the specified mastery level.

Regardless of the number of units they had mastered, at the end of the 15-week semester all of the students took a comprehensive final examination. The examination consisted of 53 multiple-choice and short answer items, each of which was related to one of the course objectives.

The grade a student earned in the course

was based upon the number of unit quizzes mastered and the student's score on the final examination. Quiz performance accounted for approximately 60% of the student's grade; final examination performance accounted for approximately 40%.

Procedure

On the first day of class, all the students met in one location, and the course instructor provided them with a brief overview of the course content and procedures. The students also were briefly informed about the study that would be conducted inthe class and, prior to being told the treatment group to which they were assigned, were given the option of not participating in the study. None of the students took this option.

Students were then sent to one of three classrooms, depending upon the group to which they had been randomly assigned. Once they were assembled, each treatment group was informed of the conditions under which it would be operating. In one group, the ascending criteria group, students were told that the mastery criteria they had to attain were 70% correct on the first six unit quizzes, 80% correct on the next three quizzes, and 90% correct on the last six. In a second group, the descending criteria group, students were told that they had to score 90% correct on the first six unit quizzes, 80% on the next three, and 70% on the last six. Students in the third group, the fixed criteria group, were told that they had to score 80% correct on each unit quiz.

Dependent Variables

Six dependent variables were examined in this study. They were (a) the percentage of students who withdrew from the course, (b) student performance on unit quizzes, (c) the number of units the students mastered, (d) the rate at which students progressed through the course, (e) student performance on the final examination, and (0 student attitudes toward the course and the mastery criteria employed. For all statistical tests, alpha was set at .05.

RESULTS

Withdrawals Of the 121 students originally enrolled in

34 ECTJ SPRING1986

the course, 12 withdrew. Four (10.3%) of the 39 students in the ascending criteria group withdrew, as did five (12.2%) of the 41 students in the fixed criteria condition and three (7.3%) of the 41 students in the descending criteria group. A chi-square analysis indicated that the differences in the percentage of students who withdrew from each group were not statistically significant.

Unit Quiz Performance Students' scores on their first attempt on each unit quiz, as well as their scores when they mas te red each unit quiz, were analyzed. Oftentimes, the same quiz score was used in both analyses because in approximately six out of every seven instances students passed a unit quiz on their initial attempt. Table I presents mean percentage scores on initial quiz attempts. Table 2 presents mean percentage scores on unit quizzes mastered. Quiz scores of students who withdrew from the course were not considered.

Analyses of variance indicated that first- attempt quiz performance did not differ among the three groups during the first and second portions of the course, but did differ among the groups during the last portion, F(2,106) = 3.97, p < .05. During that portion, students faced with the 90% criterion scored significantly higher on their first quiz attempts than did students in the other two groups.

Analyses of variance were also used to compare the groups ' quiz performance when units were mastered. As expected, during both the first and third portions of the course, students faced with a 90% mastery criterion scored significantly higher on

the quizzes they mastered than did the students in the other two groups, F(2,106) = 14.96, p <.001; F (2,106) = 20.32, p < .001.

Number of Units Mastered Among the students who did not withdraw from the course, the average number of units mastered differed significantly across the three treatment groups, F(2,106) = 6.19, p < .01. Students in the descending criteria condition mastered an average of 14.9 units, whereas the averages for students in the fixed and ascending criteria conditions were 14.4 and 14.1, respectively: Multiple comparisons using the Newman-Keuls procedure indicated that students in the descending criteria group mastered significantly more units than students in the other two groups.

Rate of Progress Each group's rate of progress was measured by calculating, for each week of the semester, the average cumulative number of units mastered by students in that group. Stu- dents who withdrew from the course were not included in these calculations.

During most of the semester, the rates at which students progressed through the course were very similar across the three treatment groups. Although each group was required to attain a different level of mastery on the quizzes for the first six units, the students in each group took an average of approximately 71/2 weeks to master those units. During the middle portion of the course (units 7-9), when an 80% mastery criterion was in effect for all groups, it took students in each group an average of approximately 2~/2 weeks to master three

TABLE 1 Mean Percentage Scores on Initial Quiz Attempts

Units

Treatment Condition n Units 1-6 Units 7-9 Units 10-15 Overall

Ascending Criteria 35 88.3(4.6) 90.9(5.5) 91.0(4.7) 89.9(3.7)

Constant Criteria 36 87.0(5.6) 89.6(5.2) 88.2(6.1) 88.0(5.1)

Descending Criteria 38 88.3(5.2) 91.3(5.9) 87.5(5.6) 88.6(4.6)

Note. Numbers in parentheses are standard deviations.

MASTERY CRITERIA 35

TABLE 2 Mean Percentage Scores on Unit Quizzes Mastered

Units

Treatment Condition n Units 1-6 Units 7-9 Units 10-15 Overall

Ascending Criteria 35 90.4(3.5) 91.5(3.7) 94.2(2.8) 92.1(2.3)

Constant Criteda 36 90.8(3.2) 90.7(4.6) 90.6(3.0) 90.7(2.8)

Descending Criteria 38 93.9(2.0) 92.7(4.4) 89.2(4.3) 91.8(2.6)

Note. Numbers in parentheses are standard deviations.

units. Student progress through the last portion of the course (units 10-15), however, varied across the three groups. Stu- dents who were faced with a 70% mastery criterion mastered an average of 5.9 units during the last 4 weeks, whereas students faced with the 80% criteria and students faced with the 90% criteria mastered an average of 5.4 and 5.1 units, respectively.

Final Examination Performance The final examination consisted of three parts. The first part covered units 1-6, the units in which mastery performance on quizzes was defined as 70% correct for the ascending criteria group, 80% correct for fixed criteria group, and 90% correct for the descending criteria group. The second part of the final examination covered units 7-9, the units in which mastery performance on quizzes was defined as 80% for all groups. The third part of the exam covered units 10-15, the units in which the ascending criteria group had to score 90%, the fixed criteria group had to score 80%, and the

descending criteria group had to score 70%. Table 3 presents the mean percentage scores for each treatment group on each p o ~ o n of the final examination, as well as each group's mean percentage score on the en- tire exam. Analyses of variance indicated that neither overall performance on the final examination nor performance on any of the three parts of the exam differed significantly among the three treatment groups.

Attitude

At the end of the semester, after the final examination was administered, students were asked to complete a four-item attitude questionnaire. The first three items were of the Likert type and assessed general attitudes toward the course. Student responses to these items indicated that, on the average, students had a moderately favor- able attitude toward the course. Responses to these items did not differ significantly among the three treatment groups.

The fourth item on the attitude questionnaire asked students to identify which of

TABLE 3 Mean Percentage Scores on the Final Examination

PoX'on of Examination =

Treatment Condition n Part 1 Part 2 Part 3 Overall

Ascending Criteda 35 76.0(12.2) 7 6 . 8 ( 1 0 . 0 ) 7 9 . 1 ( 1 0 . 7 ) 78.2(8.4)

Constant Criteria 36 78.1(10.1) 72 .9 (12 .3 ) 77.7(14.6) 76.5(9.0)

Descending Criteda 38 79 .3 (8 .9 ) 7 3 . 1 ( 1 0 . 8 ) 81.4(11.7) 78.7(8.1)

Note. Numbers in parentheses are standard deviations. =Part 1 of the examination covered units 1-6; part 2 covered units 7-9; part 3 covered units 10-15.

36 ECTJ SPRING1986

the three treatment conditions (ascending, descending, or fixed criteria) they preferred. Although each student had been assigned to only one of the conditions, all of the students were aware of the three conditions that had been employed. Overall, 73% of the students expressed a preference for a constant 80% criterion, 22% indicated that they thought the descending criteria condition was best, and 5% stated that they preferred that mastery criteria increase during the semester.

DISCUSSION

The purpose of this study was to examine the effects of various mastery criteria on student performance and attitude. The ex- perimental design employed in the study allowed for two sets of comparisons to be made. One set involved comparisons in student performance when mastery criteria were set at either 70%, 80%, or 90% correct. The second set involved comparisons in student performance and attitude when mastery criteria either increased, decreased, or remained constant during a semester. Results related to these two sets of comparisons will be discussed sepa- rately.

70 %, 80 %, and 90 % Criterion Levels With regard to the comparative effects of the 70%, 80%, and 90% criterion levels, results indicated that some aspects of student quiz performance were differentially affected by these criteria. In both the first and last portions of the semester, those students who were required to score 90% scored higher on the quizzes they mastered than did students in the other two groups. This finding, which is similar to results obtained in previous studies (Carlson & Minke, 1975; Davis, 1975; Johnston & O'Neill, 1973; Parsons & Delaney, 1978; Semb, 1974), is not surprising. Given a rea- sonable standard and adequate time, most students should be able to attain that standard. The higher the required quiz score, the better the score is likely to be at the time of mastery.

While more stringent mastery criteria are likely to affect performance on quizzes that are mastered, are they also likely to affect

initial quiz performance? Not necessarily, according to the results of this study. The criteria employed in this study affected initial quiz performance only during the last portion of the semester. This finding leads us to speculate that for most of the semester the effort most students spent initially studying a unit was not affected by the various criteria employed. However, initial ef- forts at studying a unit apparently were affected by the criteria near the end of the semester, when students were faced with the pressure of having to pass many units in a short period of time and could not afford to fail any quizzes. This conclusion--that high criteria for passing unit quizzes only affect student behavior when students are under some time pressure--supports the view previously expressed by Parsons and Delaney (1978).

Based upon the results of this study and some previous research findings (Hochstet- ter & Caldwell, 1977; Parsons & Delaney, 1978), we also suggest that the use of stringent mastery criteria on unit quizzes may not affect comprehensive examination performance. The additional effort students spend studying for a comprehensive examination may more than offset the small differences in learning during the semester that may result from the various mastery criteria employed.

Whether comprehensive examination performance is differentially affected by the various mastery criteria may be dependent, in part, on how long it has been since students began mastering the units covered on the examination. In those studies in which various mastery criteria did differentially affect comprehensive examination performance (Block, 1972; Parsons & Delaney, 1978; Semb, 1974), the examination was administered no more than 6 weeks after the students began mastering the material. In one instance (Block, 1972), the examination took place only 4 days after the students began to demonstrate mastery. In contrast, in the present study and in one other in which various mastery criteria did not differehtially affect comprehensive examination performance (Hochstetter & Caldwell, 1977), students began mastering unit quizzes as much as 3 months prior to the administration of the comprehensive

MASTERY CRITERIA 37

examination. It may be that, over time, the level of mastery a student is required to attain on unit quizzes is less likely to have an effect on comprehensive examination performance.

It is also interesting to note that in those Studies in which comprehensive examination performance was not differentially affected by various mastery criteria, students faced with low standards performed at a level much higher than that called for by the standards. For example, Parsons and De- laney (1978) found that students faced with a 70% criterion had an average score of 82.4% when they mastered unit quizzes. Similarly, in the study by Hochstetter and Caldwell (1977), although the average criterion level students were required to at- tam was 80.7%, the average mastery score on quizzes was 89.2%. And in the present study, students who were required to sc~ze 70% on quizzes, mastered them with an average score of 89.8%. In each instance, the lack of significant differences in comprehensive examination performance may have been due, in part, to the fact that students faced with fairly low criteria for mastering unit quizzes vastly exceeded those criteria.

Although comprehensive examination performance in this study was not differentially affected by the various mastery cziteria employed, student rate of progress was. During the last part of the semester, students faced with the 70% criterion proceeded through the course more rapidly than did the students in the other two groups. This more rapid pace enabled the students in that group to master significantly more unLts than Me other students Ln the study, but it did not lead to better performance on the final examination.

In summary, although the 90% mastery criterion had a positive effect on some aspects of quiz performance, it had an adverse effect on student rate of progress and the number of units completed. Comprehen- sive examination performance was unaf- fected by variations in the mastery criteria employed. These Findings cast some doubt on the notion that student performance in a mastery-oriented course will be optimized when the criterion for passing unit quizzes is set at a fairly high level.

Ascending, Descending, and Fixed Criterion Levels With regard to whether criteria in a mastery course should increase, decrease, or remain constant during a semester, findings from this study indicate that using the same criterion throughout a semester may be best. Most of the students in this study preferred that strategy, and, in terms of with- drawal rate, overall quiz performance, and final examination performance, the effects of the fixed criterion approach were similar to the effects of the other two approaches.

The descending criteria condition was the second most popular approach among students who participated in this study. This popularity was most likely attributable to the fact that students in this group had to score only 70% on unit quizzes during the last portion of the semester, when most students were tryir~g to compensate for the slow pace they had maintained earlier, In- deed, during the last part of the semester, s tudents in this condit ion proceeded through the course more rapidly than students in the other two groups. As a conse- quence of having maintained this faster pace, students in the descending criteria group completed significantly more units than the other students. Yet, final examination performance among students in this group was not better than among students in the other two groups. In addition, as the mastery criteria for this group decreased, quiz performance decreased, albeit slightly. Thus, the notion that the performance of the students in this group would not slacken was not supported.

As for the benefits previously attributed to the use of ascending criteria (Carlson & Minke, 1975; Johnston & O'Neill, 1973), they were not particularly apparent in this study. Although quiz performance among students in the ascending criteria group did improve as the semester progressed, the improvement was small. Indeed, quiz performance under the 90% criterion was less than one percentage point better among students in th is ~oup thar~ i t was among students in the descending criteria condition, who were required to score 90% on quizzes at the outset of the semester. In addition, students in the ascending criteria group progressed through the course at a

38 ECTJ SP~NG1986

less rapid rate than students in the other two groups.

CONCLUSION

Several years ago, Johnston and O'Neill (1973) suggested that the criteria employed in a mastery-oriented course "should start high and go higher" (p. 268). That statement may be appropriate if an instructor is trying to improve student performance on unit quizzes. However, as the results of this s tudy and some previous research indicate, that statement may not apply if the goal is to improve student performance on a comprehensive examination, particularly an examination covering instruction that span- ned several months. Additional research examining the effects var ious mas te ry criteria have upon comprehensive examination performance appears necessary. Per- haps that research will confirm the tentative conclusion we lean toward as a result of this study: In mastery-oriented courses, it may be best to set mastery criteria at a moderate level and keep them there.

REFERENCES

Block, J. H. (1972). Student learning and the set- ting of mastery standards. Educational Hori- zons, 50, 183-191.

Block, J. H., & Burns, R. B. (1976). Mastery learning. In L.S. Shulman (Ed.), Rezn'ew of Re- search in Education: 4. Itasca, IL: F. E. Peacock.

Bloom, B. S. (1976). Human characteristics and school learning. New York: McGraw-Hill.

Burkman, E., & Brezin, M. (1981). Effects of ex- pectation level on achievement in high school physical science courses employing a quasi- mastery teaching method. Journal of Educational Research, 75, 121-126.

Carlson, J. G., & Minke, K. A. (1975). Fixed and ascending criteria for unit mastery learning. Journal of Educational Psychology, 67, 96-101.

Davis, M. L. (1975). Mastery test proficiency requirement affects mastery test performance. In J. M. Johnston (Ed.), Behavior research and technology in higher education (pp. 185-201). Springfield, IL: Charles C. Thomas.

Dick, W., & Carey, L. (1978). The systematic design of instruction. Glenview, IL: Scott Fores- m a n .

Gagn~, R. M. (1974). Essentials of learning for instruction. Hinsdale, IL: Dryden.

Hochstetter, G. T., & CaldweU, E. C. (1977). The effect of student-selected mastery levels in a PSI course. Journal of Personalized Instruction, 2, 162-164.

Hursh, D. E. (1976). Personalized systems of instruction: What do the data indicate? Journal of Personalized Instruction, 1, 91-105.

Jacobs, R. L. (1983). A review of PSI course features and selected concerns for their im- plementation. Journal of Educational Technology Systems, 11, 335-343.

Johnson, K. R., & Ruskin, R. S. (1977). Behav- ioral instruction: An evaluative review. Washing- ton, DC: American Psychological Association.

Johnston, J. M. & O'Neill, G. (1973). The analysis of performance criteria defining course grades as a determinant of college student academic performance. Journal of Applied Behavior Analysis, 6, 261-268.

Johnston, J. M. & Pennypacker, H. S. (1971). A behavioral approach to college teaching. Amer- ican Psychologist, 26, 219-244.

Keller, F. S. (1968). "Goodbye Teacher . . . " Journal of Applied Behavior Analysis, 1, 79-89.

Parsons, J. A. & Delaney, H. D. (1978). Effect of unit-quiz mastery criteria on student performance. Journal of Personalized Instruction, 3, 225-228.

Semb, G. (1974). The effects of mastery criteria and assignment length on college-student test performance. Journal of Applied Behavior Analysis, 7, 61-69.

Sherman, J. G. (1974). Logistics. in F. S. Keller & J. G. Sherman (Eds.), The Keller Plan Hand- book (pp. 24-49). Menlo Park, CA: W. A. Benjamin.

the effects of various mastery criteria on student performance and attitude in a mastery-oriented...

Documents