1

Core 2009: Second Languages Assessment Study of

Second Language Learning Objective 1.3

May, 2012 This report summarizes the results of the Core assessment study of student learning of the Core Second Language requirement based on courses taught during the Winter and Spring Quarters of 2011. The purpose of the evaluation was to determine the level of mastery of the Second Language Learning Objective 1.3: Demonstrate sensitivity to cultural differences as afforded by the language studied. Student work was evaluated using a rubric in which the work was scored as “Not Meeting,” “Approaching,” “Meeting,” or “Exceeding” the learning objective under review. Success in Core assessment of student learning projects is defined by 80 percent of the students meeting or exceeding the learning objective. Mastery is considered to be an average rating of 3 on the Core Second Language rubric1, which corresponds to an evaluation of “meeting” the learning objective. The assessment relied on Second Language courses from winter and spring terms of the 2010-11 academic year. The second level elementary language course in Second Language was included because some majors (e.g., Business School majors, mathematics, natural sciences) complete their Core requirement at this point, while others are required to complete an Elementary Level 3 course. All students enrolled in the 36 sections of Elementary Level 2 Second Language courses taught in winter term were invited to participate in this Core assessment project, along with all students enrolled in the 27 spring term sections of Elementary Level 3 Second Language courses. Student Participant Selection and Collection of Student Papers Students enrolled in sections of Elementary Second Language were asked to sign a consent form permitting the confidential review of written assignment(s) for purposes of evaluating the student learning in the Second Language Core area. Of all students enrolled, 75% returned a signed consent form for the winter courses of Level 2; 79% of the enrolled students did so for Level 3 courses.2

1 The initial draft of the Second Language Core rubric was developed by the Office of Assessment based on the learning objective (LO 1.3) developed by the Faculty Core Committee. A four-level rubric structure was adopted that ranged from Not meeting (1) to Exceeding (4) and criterion language for the Second Language rubric was drafted. A draft rubric is ordinarily reviewed by the chair of the FCC, the chair(s) of the principal departments in which the Core courses are offered, and members of the Core Curriculum Implementation Team (CCIT), and revisions are made accordingly. 2 These percentages do not take into account non-consenting students who were absent on the day the forms were distributed.

2

Instructors of these courses were asked to identify one or more written assignments from their course that would demonstrate student learning for Second Language Learning Objective 1.3. The selection of the assignment(s) was left to the discretion of the individual faculty member. Faculty members were also asked to submit a written assignment description or instructions that were provided to students as they completed each selected assignment (e.g., set of exam questions). Most faculty members elected to submit final examination responses. Student names, faculty names, and course names/numbers were removed from all written materials prior to the scoring session. Student work was submitted to the Office of Assessment by the faculty members for 30 of the 36 sections of Level 2 and all 27 sections of Level 3. Scoring Procedures – The “Rubric Scoring Party” Faculty met for approximately three hours on a single occasion in the spring to score the submitted Elementary Level 2 assignments. During this scoring session, the group began by reading and discussing the Second Language scoring rubric to be sure that it was clear and understandable to all present. Then the group scored a common paper and compared scores to achieve inter-rater reliability. A 55% agreement in scores on the first paper was achieved. Discrepancies in scoring were discussed until consensus was reached on the language and interpretation of the rubric. A second paper was then scored by the whole group. The degree of agreement was 73% after this second paper. A final brief discussion of scoring ensued, and then the full set of student assignments were distributed among the scorers for review. Approximately 60% of the student assignments in this evaluation were read by a single reader. In the case where there was a score discrepancy the assessment received a 3rd reading. Only one of the assignments (2%) had to be read by a third reader. All scorers received a small stipend for their participation in the scoring party. The Level 2 rubric scorers were invited to score the Elementary Level 3 assignments. Because these student assignments were collected at the end of the academic year, the scoring took place during the summer and fall. Faculty who accepted the invitation to score Level 3 assignments were given the option of receiving the assignments in hard copy or electronically. Scorers were asked to apply the same Second Language scoring rubric and determine the appropriate score for each student’s work. These Level 3 scores were communicated to the Assessment Office when the scorers had completed all evaluations. Scorers received a small stipend for their Core assessment efforts. Scoring Results and Interpretation Elementary Level 2 Results The Second Language Learning Objective 1.3 was evaluated at the end of Elementary Level 2 courses in winter of 2011. Of the 111 sets of assignments from Level 2 scored, 5 percent of student papers received a rubric score of “4 – Exceeding”, 46 percent received a score of “3 - Meeting”, 41 percent received a score of “2 – Approaching” and 8 percent received a score of “1 – Did Not Meet.” Fifty-one percent of the student work was evaluated as “Meeting” or “Exceeding” the learning standard. The average rubric score for this learning objective was a 2.49. These results show that our students’ mastery of this learning objective falls between approaching and meeting.

3

To satisfy the Core Curriculum, students majoring in Business, Mathematics or Natural Science are required to complete the second level of a second language. All other majors in the College of Arts and Sciences are required to complete a third level elementary course. Therefore, this next analysis summarizes the results for the Level 2 Second Language assessment for just those students who can satisfy their Core requirement at this point in the sequence. Of the 58 sets of Level 2 assignments from these students, 5 percent of student papers received a rubric score of “4 – Exceeding”, 41 percent received a score of “3 - Meeting”, 47 percent received a score of “2 – Approaching” and 7 percent received a score of “1 – Did Not Meet.” Forty-six percent scored “Meeting” or “Exceeding.” The average rubric score for this learning objective was a 2.45. These results for students in majors requiring only two courses are quite close to the Level 2 course results for all students.

Elementary Level 3 Results Level 3 student work was scored using the same rubric as Level 2. Of the 103 sets of assignments from Level 3 classes, 8 percent of student papers received a rubric score of “4 – Exceeding”, 44 percent received a score of “3 - Meeting”, 33 percent received a score of “2 – Approaching” and 15 percent received a score of “1 – Did Not

4

Meet.” Fifty-two percent scored “Meeting” or “Exceeding.” The average rubric score for this learning objective was a 2.44. A larger number of students scored a “1—Did Not Meet” in this level than in Level 2.

Conclusion The results show that most students are “approaching” or “meeting” the learning objective, but are not reaching the benchmark set for student learning in the Core. Generally across Core assessments, we have found that achieving our ambitious goal of 80 percent of students meeting or exceeding a particular learning objective is challenging in part due to the complexity of the learning objectives that underpin the Core and the associated need to design assignments or exam questions that ask students to address the range of learning expected. In this case, for students to meet the learning objective, according to the rubric they would be expected to describe cultural differences in the L2 countries or communities, recognize differences in cultural viewpoints represented in the products, practices and perspectives of L2 countries or communities, and demonstrate awareness of values of another cultural group. Discussion of these assessment results by the faculty can yield important insights about how to further improve student learning and assessment strategies. The following questions are offered as potential conversation starters for faculty teaching in the Second Language Core area:

• Are course assignments or exam questions effectively capturing all of the components of the learning objective under review?3

• Do faculty teaching in this area of the Core share a common understanding of the desired learning expressed in the learning objective statement?

• Have the faculty participated in reviewing the Second Language Core rubric and the criteria that will be applied when this Core area is reviewed?

• Do the faculty have recommendations to improve the Core assessment process in this area?

3 Mapping assignments to the Core learning objectives provides a clear context for students to demonstrate what they have learned. Following some previous Core investigations faculty have shared ideas about developing assignments that map more closely to the Core learning objectives. In assessments of student learning conducted after discussions of assignment design, significantly greater learning is evident.

5

Report prepared by the Office of Assessment and the Core Curriculum Implementation Team

• Christine Bachen, Director, Office of Assessment • Carol Ann Gittens, Former Director, Office of Assessment • Phyllis Brown, Associate Provost for Undergraduate Studies and Director of the Undergraduate Core

Curriculum • Eileen Elrod, Associate Vice Provost for Faculty Development • Phil Kesten, Associate Vice Provost for Undergraduate Studies • Barbara Molony, Member, Core Curriculum Implementation Team • Kathleen Villarruel Schneider, Senior Assistant Dean for the College of Arts & Sciences

A special thank you is extended to the faculty who taught the Second Language courses in 2010-11 and to those who participated in the Rubric Scoring Parties. The staff of the Office of Assessment is also acknowledged for their assistance with the preparation of this report.

SCU Core Curriculum Second Language Evaluation Rubric Work Sample #: ______

1)Points to keep in mind when using this rubric:

This is a HOLISTIC rubric so there is only one score. Determine the overall evaluative score (1-4) for each objective based on the level that best characterize the performance demonstrated in the assignment. SCORES MUST BE WHOLE NUMBERS– no decimal points. Write the score in the box provided.

2) Every paper will have one (1) score for each objective being evaluated. 3) Work samples were collected from different courses with differing assignments (varied formats, requirements, lengths, etc). Because of the variation,

work samples should not be compared to one another – each should be independently evaluated using this rubric. The scores for any given work sample should not be influenced by other papers already reviewed.

4) When applying this rubric to students’ work samples, it is important to realize that some of the descriptors may not be addressed because they were not elicited by the course assignment. You should determine the rubric score based on what the student has written. Do NOT adjust your score as a result of reading the assignment description!

5) Remember you are NOT grading the papers like one would for a class. You are evaluating the paper based solely on the criteria that appear on this rubric.

6) Please feel free to write notes or marks on the rubric or work sample. Should it be necessary to discuss a score with a colleague, these notes will aid in your recollection for why you selected a particular overall evaluative score.

Objective 1.3 Demonstrate sensitivity to cultural differences as afforded by the language studied.

Exceeding (4) Meeting (3) Approaching (2) Not Meeting (1)

Articulates a complex understanding of cultural differences in the L2 countries or communities. Analyzes differences in cultural viewpoints in terms of how they are symbolized by the products, practices and perspectives of those L2 countries or communities. Demonstrates keen awareness of values of another cultural group.

Describes cultural differences in the L2 countries or communities. Recognizes differences in cultural viewpoints represented in the products, practices and perspectives of those L2 countries or communities. Demonstrates awareness of values of another cultural group.

Identifies some cultural differences in the L2 countries or communities. Shows emergent recognition of how cultural viewpoints are represented in the products, practices and perspectives of those L2 countries or communities. Demonstrates a developing awareness of values of another cultural group.

Has a minimal understanding of cultural differences in the L2 countries or communities. Demonstrates little or no recognition of how cultural viewpoints are represented in the products, practices and perspectives of those L2 countries or communities. Little evidence of an awareness of values of another cultural group.

WRITE OVERALL SCORE FOR OBJECTIVE 1.3 IN THIS BOX:

Notes: