Developing a PortfolioAssessment:

Pacesetter® Spanish

ANDREA FERCSEY, CARMEN LUNA,EVA PONTE, and PABLO ALIAGA

College Board Report No. 99-2

College Entrance Examination Board, New York, 1999

College Board Report No. 99-2

Developing a PortfolioAssessment:

Pacesetter® Spanish

ANDREA FERCSEY, CARMEN LUNA,EVA PONTE, and PABLO ALIAGA

Acknowledgments

We would like to acknowledge the six Pacesetter®

Spanish teachers who participated in Phase I of thisresearch project: Eva Chapman and Ruth Ann Kerkoc,Leland High School, San Jose, California; GildaNissenberg, North Miami Beach High School, Florida;Dianne Michel Stenroos, Rutland High School,Vermont; Carol Thorp, East Mecklenburg High School,Charlotte, North Carolina; and María Elena Villalba,Miami Palmetto Senior High School, Florida.

We would also like to acknowledge the followingPacesetter Spanish teachers who participated in Phase IIof this research project: Esperanza Cobián-Bashara,Silver Creek High School, San Jose, California; SharonDyrland, Surrattsville High School, Clinton, Maryland;Rosalba García, Bellaire High School, Bellaire, Texas;Ruth Ann Kerkoc, Leland High School, San Jose,California; Dennis Lavoie, Fairport High School,Fairport, New York; Gilda Nissenberg, North MiamiBeach High School, Miami, Florida; Nereida SamudaZimic, William B. Travis High School, Austin, Texas;Carol Thorp, East Mecklenburg High School, Char-lotte, North Carolina; and María Elena Villalba, MiamiPalmetto Senior High School, Miami, Florida.

The contributions and efforts of these individualswere essential throughout all stages of the project.

The authors would like to thank Carol Myford, RickMorgan, and Gita Wilder for their help in interpretingand analyzing data, as well as for their review of anearlier draft of this paper.

Andrea Fercsey is an assessment specialist at ETS. Carmen Luna is a group leader at ETS.Eva Ponte was a 1996 summer intern at ETS and agraduate student at the University of California atBerkeley.Pablo Aliaga was a 1997 summer intern at ETS and isa graduate student at the University of Michigan.

Researchers are encouraged to freely express their pro-fessional judgment. Therefore, points of view oropinions stated in College Board Reports do not neces-sarily represent official College Board position or policy.

Founded in 1900, the College Board is a not-for-profiteducational association that supports academicpreparation and transition to higher education forstudents around the world through the ongoing collab-oration of its member schools, colleges, universities,educational systems and organizations.

In all of its activities, the Board promotes equitythrough universal access to high standards of teachingand learning and sufficient financial resources so thatevery student has the opportunity to succeed in collegeand work.

The College Board champions—by means of superiorresearch; curricular development; assessment; guidance,placement, and admission information; professionaldevelopment; forums; policy analysis; and publicoutreach—educational excellence for all students.

Additional copies of this report (item #200272) may beobtained from College Board Publications, Box 886,New York, New York 10101-0886, (800) 323-7155.The price is $15. Please include $4 for postage andhandling.

Copyright © 1999 by College Entrance ExaminationBoard and Educational Testing Service. All rightsreserved. College Board, Advanced PlacementProgram, Pacesetter, SAT, and the acorn logo areregistered trademarks of the College EntranceExamination Board.

Printed in the United States of America.

ContentsAbstract...............................................................1

PHASE I REPORT....................................................1

Introduction ........................................................1

Background .........................................................1The Pacesetter® Program ........................................1Pacesetter Spanish ...................................................2Portfolio Assessment ...............................................2

Pacesetter Spanish Portfolio Research Project (Phase I) .................................4

Method ...................................................................4Portfolio Analyses...................................................7Description of Elements..........................................7Distribution of Ratings ...........................................7Time Used to Read Portfolios.................................8Culminating Assessment .........................................9Portfolio Assessment Matrix Correlations ............10Portfolios Read Twice: Reliability Study...............11Third Rater Analysis .............................................15Selection of Benchmarks .......................................15

Summary of Findings.........................................16

Recommendations .............................................16

References .........................................................16

Phase I Appendix ..............................................37

Tables1. Distribution of Ratings ..................................72. Distribution of Ratings by Aspects

and Dimensions .............................................83. Time Used to Read Each Portfolio .................93A. Summary........................................................94. Average Time Employed to Read Portfolios

From Different Teachers ................................95. Average Time Employed by Each Rater .......106. Time Employed to Read Portfolios

at Different Sessions.....................................107. Culminating Assessment and

Dimension 2 Portfolio Ratings.....................108. Portfolio Assessment

Matrix Correlations .....................................119. Summary of Portfolio

Score Discrepancies ......................................1310. Summary of Portfolio Score Discrepancies

for Each Aspect in Each Dimension .............1311. FACETS OUTPUT (7.1.1) Students

Measurement Report (Arranged by FN) ......1412. Distributions of Raters’ Judgments ..............1513. Portfolios Read Three Times........................15

Figure1. FACETS map for the whole scale

(21 portfolios, two readers) ..............................12

PHASE II REPORT.................................................17

Rationale and Expected Outcomes....................17

Pacesetter Spanish Portfolio Research Project (Phase II)..............................18

Method .................................................................18Summary of Ratings..............................................19Time Used to Read Portfolios...............................21Distribution of Ratings for Each Rater .................21Portfolios Read Twice...........................................22Correlation Among the Ratings of Portfolios .......27Reliability Study....................................................27Analyses ................................................................29

Students ............................................................29Raters ...............................................................33Aspects and Dimensions ...................................33

Summary of Findings.........................................34

Recommendations .............................................34

References .........................................................35

Phase II Appendix.............................................67

Tables1. Frequency of Ratings in

Each Rating Category.....................................192. Frequency of Ratings in Each

Category for the Aspects of Dimension 1 .......203. Frequency of Ratings in Each

Category for the Aspects of Dimension 2 .......214. Time Employed by Each Rater in Minutes .....225. Total Count and Relative Frequency by

Category and Rater ........................................236. Comparison of Portfolios and Raters .............247. Discrepancies Between Raters.........................268. Specific Discrepancies Between Raters ............269. Bivariate Correlation Matrix ..........................27

10. Rater Severity.................................................2911. Percentage in Each Scale Category .................33

Figures1. FACETS: Partial credit model for raters .......282. FACETS: Partial credit model for aspects.....303. FACETS: Rating scale model........................313A. FACETS: Student adjusted scores .................32

1

AbstractPortfolios are one of the assessment tools used inPacesetter® Spanish. In Phase I of this study, conducted in1995–96, a first attempt was made to develop a stan-dardized portfolio assessment system. As part of thissystem, a set of guidelines and an assessment matrix wereprepared. The assessment matrix was piloted during aportfolio reading in which six teachers participated. Atthe end of the reading, some portfolios were identifiedas benchmarks to be used for training purposes.

After the reading, results were compiled and analyzed.Data were collected to investigate several aspects of theportfolio assessment system (i.e., scores, the reliability ofthe raters, correlation with the Culminating Assessment,and time needed to read portfolios). Our purpose was todetermine which components of the portfolio assessmentsystem were working as intended and which might needto be changed. To this end, we explored ways ofgathering preliminary evidence of the reliability andvalidity of the portfolio assessment system. Our findingssuggested that refinement of the matrix was needed aswell as clearer guidelines for the selection of artifacts toinclude in the portfolios.

In Phase II of our research, in consultation withparticipating teachers, a revised set of guidelines and arevised assessment matrix were distributed to allschools implementing the Pacesetter Spanish course in1997. Seven teachers (three of whom had also been apart of Phase I of this study) participated in a portfolioreading in 1997, at which both the guidelines and thematrix were further refined for immediate dissemina-tion to the schools. A new set of benchmark portfolioswas identified, and a set of sample portfolios was pre-pared for use in subsequent professional developmentsessions for Pacesetter Spanish teachers.

In general, the results of our findings are encour-aging. Initiating national and/or regional portfolioscoring sessions would greatly enhance the validity andreliability of the portfolio component as an integral partof the Pacesetter Spanish program.

PHASE I REPORT

IntroductionThe Pacesetter® program developed by the College Boardand Educational Testing Service (ETS) is an integrated setof learning outcomes, course materials, and instructionalexperiences, including various approaches to assess-ment, and is currently being offered in three subjectareas: English, Spanish, and Mathematics.

The Pacesetter Spanish program has provisions forstudent self-assessment, peer assessment, local assess-ments, and end-of-course standardized assessment, aswell as portfolio assessment. All Pacesetter Spanish stu-dents are expected to assemble portfolios with productsfrom their work throughout the course in order to showevidence of what they have learned.

From March to April 1996, the Pacesetter Spanishprogram conducted a research project on the assessmentportfolio component, with the following goals: (a) toprepare a working definition of Pacesetter Spanish port-folios as assessment tools, (b) to develop a standardizedportfolio assessment system, and (c) to utilize theresearch findings for upcoming professional develop-ment sessions for teachers. This paper describes thelatest developments in these three areas.

Background

The Pacesetter® ProgramThe Pacesetter program is part of the College Board’sefforts to promote achievement of educational excellencefor all students. The College Board defines Pacesetter asa school-based instructional assessment program thatintegrates high academic standards for all students,teacher preparation to help achieve those standards, anda broad range of assessments measuring student achieve-ment and facilitating instructional decisions.

The Pacesetter program is structured around threeessential components. First, it includes course frame-works specifying content curriculum, which detail theknowledge and skills students are expected to master.Teachers and students are encouraged to have a sharedunderstanding of what content areas are to be coveredthroughout the course. Second, it provides professionaldevelopment sessions for teachers, which include avaried range of activities. Third, the program incor-porates several types of assessment, ranging fromembedded assessments to standardized assessments.

The College Board will conduct research toexamine the program outcomes: how the programprepares students for future careers as well as how itprepares students for other College Board assess-ments, namely SAT® I: Reasoning Test, SAT II:Subject Tests, and Advanced Placement Program®

courses and examinations. As part of programimplementation, research will focus on how teachersadopt Pacesetter classroom strategies. Finally, thewhole assessment process will be monitored toprovide feedback for further revisions to instructionalmaterials and methodology.

2

Pacesetter SpanishThe aim of Pacesetter Spanish is to provide third-levelhigh school students with a contextualized approachto language mastery. The program integrates learningoutcomes, course materials and learning experiences,and various types of assessment.

Because the program embraces a contextualizedapproach to learning, course objectives go beyond thoseestablished by traditional instruction. The three mainexpected course outcomes are: (a) to use the Spanish lan-guage to acquire knowledge, (b) to understand Hispaniccultures, and (c) to use Spanish effectively to communicatewith others. These outcomes support the five goals of theNational Standards for Foreign Language Learning. Theyfocus as much on the process of acquiring new informa-tion and cultural skills as they do on acquiring specificknowledge in specific domains of the language.

Pacesetter Spanish covers a broad range of content.Students learn about the contributions of variousHispanic figures in different fields. The curriculummaterials facilitate students’ ability to make connectionsbetween different aspects of Spanish-speaking cultures,as well as to gain interdisciplinary knowledge. Theprogram has a holistic approach to language skills.

The materials of the course framework are themati-cally linked. Students read authentic texts, engage indiscussions, develop research projects, and experienceaesthetic works (music, arts, etc.). Teachers may chooseto supplement course materials in different ways, and touse additional sources of information, the resultingproducts presented as evidence of students’ learning.

The role of students in the program varies somewhatfrom their role in traditional courses. Students areencouraged to work both independently and in groups,to conduct research projects using different resources,and to approach learning from an interdisciplinaryperspective (i.e., they use knowledge and inquiry skillsfrom other areas of the curriculum to help them learnSpanish). They also take an active role in the evaluationprocess by monitoring and evaluating their own perfor-mance. Pacesetter Spanish empowers students to beautonomous, self-directed language learners.

Finally, assessment aims at both the processes andthe products of instruction. Assessment is integratedinto the course, taking alternative approaches beyondtraditional tests. Students are active participants in theprocess, conducting self-assessment through learninglogs and journal activities, and through peer assess-ments. Local assessments are designed by teachers toevaluate student performance and teacher effectiveness.The Culminating Assessment, a standardized com-ponent provided at the end of the course, may serveaccountability purposes. Students are expected to main-

tain a portfolio with samples of their work to showevidence of achievement in the various aspects of thedimensions assessed through this component.

These new approaches to instruction demand notonly a change in the curricula and the role of students,but also in the role of teachers. Teachers are expected(1) to teach a language course not centered in thelanguage itself, (2) to teach an integrated and inter-disciplinary course with a holistic approach, (3) toimplement cooperative learning approaches in theirclassrooms, (4) to focus on process as much as onproducts, (5) to be able to provide different sources ofinformation, (6) to be able to search for differentsources of evidence of students’ performance, (7) toteach students different learning strategies, (8) to use thescoring rubrics appropriately, (9) to adequately manageportfolios, and (10) to promote student self-assessmentand self-learning.

To attain this competency, participating teachershelp define (a) the knowledge and skills necessary forstudents’ success and (b) ways to find evidence ofstudents’ learning. These teachers also help createscoring rubrics. To teach Pacesetter Spanish, all partici-pating teachers must attend professional developmentsessions before the beginning of the academic year andare also expected to attend midyear meetings. Bothare designed to give ample opportunities for the par-ticipants to exchange suggestions on pedagogicalapproaches and techniques, as well as to give feedbackon the current approach to the content of coursematerials. It is to be noted that the leaders and trainersat these sessions are themselves teachers implementingthe Pacesetter Spanish program in their classrooms.

In the 1996–97 school year, the Pacesetter Spanishprogram was taught in approximately 25 districts, 60 schools, by 80 teachers, to 4,000 students.

Portfolio AssessmentThe United States educational system has faced severalproblems in the past. Resnick and Resnick (1992) havepointed out the link between testing efforts andeducational reforms in this country. This link hasfocused attention on school assessment practices andits problems. Some assessment alternatives have beenformulated by the educational community in an attemptto overcome these problems, including performance-based assessment.

Performance-based assessment involves both instruc-tional processes and products. One of its main featuresis that students create or construct complex responsesand products. Answers can no longer be considered assimply right or wrong. Alternative answers given by

3

students and teachers use evidence from different sourcesand media. Because this process generates complexresponses, human judgment is required to evaluatestudents’ responses along a continuum of achievement.Performance-based assessment can be linked to and em-bedded in instructional practice, and it is expected thatthis link will allow assessment to inform and guideinstructional practice.

Portfolio assessment is a special type of performance-based assessment. A portfolio is a selected collectionof students’ work over time evaluated by one or moreraters using clear criteria. In some cases, portfoliosmay also include students’ reflections about their ownwork and progress. Portfolios, like most types of perfor-mance-based assessments, are considered tools to guideinstructional practice. If a portfolio assessment system iscorrectly implemented, instruction should be expectedto: (1) pay more attention to the process of learning, (2)lead teachers to think, evaluate, and modify their ownpractices, (3) change students’ traditional role from apassive one to one in which they take charge of theirown learning and the evaluation of that learning, (4) beguided by current theories of development and theprocess of learning, and (5) make explicit what contentis valuable for both teachers and students.

To promote this change, portfolios should be morethan a pile of schoolwork. Gitomer and Duschl (1995)indicate that good portfolio practice requires funda-mental changes in conceptions of (scientific) knowledge,teaching, learning, and assessment. They explain indetail what these changes mean in the area of scienceeducation. Some of their claims can be applied to othereducational domains. First, curricula should reflect thenature of the discipline being taught; students shoulduse procedures, methods, and practices similar to thoseused by professionals in the discipline. The dynamicsand interactions found in the field should also be anintegral part of instruction. Second, it is necessary totake a constructivist approach to learning: to haveconceptual change as the goal of learning. Third, assess-ment should serve the needs of students and teachers aswell as others who have a stake in what happens in theclassroom. Criteria used to consider student perfor-mance should be explicit, and assessment must meetthree conditions: (a) it should focus on knowledge andskills considered important within the discipline, (b) itshould contribute to instruction and learning, and (c)it should serve accountability purposes.

Arter and Spandel (1992) state that portfolios willonly have the desired effects if they are carefullyplanned. They give a definition of portfolios thatincludes multiple elements and lists requisites for a goodportfolio system. First, it is necessary for users to have aclear idea of the purpose of the portfolio, since different

objectives will lead to quite different portfolios. Second,students should participate in the selection of theportfolio content. This selection process demands self-reflection from students, forcing them to analyze theirown work and what elements in that work best serveto demonstrate their knowledge and skills. To do this,students should have clear guidelines of what artifacts toselect as part of their portfolios. Third, assessmentcriteria must be fully and carefully defined and open toall. Finally, assessment and instruction should form anintegrated whole. Portfolios can monitor students’accomplishments as well as provide for new and betterways for students to improve on their achievements.

Portfolios define the teacher’s role in a “new” way:Teachers are considered professional researchers studyingtheir own practice. (This view of teachers was alreadyproposed by Elliot [1985].) This new definition of teachingrequires teachers to revise and change their beliefs andapproaches to instruction, learning, and assessment, andto rethink instructional goals. Without such changes itwould be difficult to attain the goals set by portfolioassessment. Sheingold, Heller, and Paulukonis (1995)identified the following five categories in which teachersreported changes: (1) using new sources of evidence, (2)sharing responsibility for learning and assessment, (3)changing goals of instruction, (4) using new ways ofevaluating evidence, and (5) changing the general viewof the relationship between assessment and instruction.

With respect to students, portfolios are beneficial inseveral ways. First, students tend to find them moti-vating and engaging. Second, through portfoliosstudents become more aware of and responsible fortheir own learning and are able to make more informedchoices about their work and how to approach it.Third, these students—more than students in traditionalassessment settings—tend to collaborate with otherstudents and with teachers. Fourth, different ap-proaches and levels of learning are allowed within theportfolio framework; students are given a broader rangeof opportunities to learn and to demonstrate theirlearning. Also, because portfolios pay more attention tohow student work compares to established standardsinstead of to the work of other students, a less compet-itive environment is created. Finally, because portfoliocontents tend to focus on what is important to learn,and for what purposes, students are better prepared forwork life and life in general.

Despite their promising outlook, portfolios offer nopanacea. Several problems may preclude their being asbeneficial for education as could be expected, the mostimportant problem one of implementation. In thisrespect, there is a big gap between theory and practice.Teachers, the principal catalysts of portfolios, must beprepared to perform their new assigned role. They must

4

be able to (a) clearly define the purpose of instruction andundergo all the above-mentioned changes, (b) clarifycomplex tasks and provide students with additionalsources and media of information, (c) be able to findevidence of students’ learning in different ways, makingcontinuous and accurate judgments, and (d) continuouslyrevise and adapt the instructional process. Some teachersfind this an enormous responsibility in terms of invest-ment of classroom time, professional development, andscoring. This may explain the reluctance of some teachersto implement portfolio assessment in their classrooms.

However, teachers are not the only ones who mayfind it difficult to pursue a portfolio assessment model.The general public is also concerned about how reliable,valid, and fair portfolio scores are for accountabilitypurposes. These issues may be even more critical whenportfolios are used in large-scale assessments thanwhen they are used as an assessment tool in the class-room. Portfolio scoring becomes an essential part of theportfolio system in terms of reliability, validity, andfairness. The first step in portfolio scoring is to developrubrics and construct an assessment matrix. The rubricsshould include both the content and the processesstudents are expected to master. Rubrics should alsodescribe what kind of evidence we should expect for thedifferent levels on the performance scale. Ideally,teachers should be involved in the process of definingand refining the rubrics. Some studies have been con-ducted to study how consistent and accurate readers arein their judgment (see Bridgeman, Chittenden, andCline (1995); Myford and Mislevy (1995); and Koretz(1992, 1994)). Reliability has usually been reportedin terms of inter-rater agreement (i.e., the correlationbetween the scores two different raters give to the samestudent work). When discrepancies between these ratersare large (the criteria for “large” depends on the scaleused), usually a third reading is done. The use of a thirdrater reduces the impact of rater severity on a student’sscores (i.e., how strict a rater is when evaluating astudent’s work), but this procedure does not resolve theproblem of rater interchangeability. For example, ifboth raters are extremely severe or lenient, thisprocedure will not be able to detect that, and the finalscore given to the student may be erroneous. Myford,Marr, and Linacre (1996) propose to calibrate ratersaccurately and precisely, adjusting examinees’ scores forrater severity differences, hence improving reliabilitycalculations by removing the errors in those scoresassociated with rater severity.

Statistics can point out the problems; however, tofind out the nature of these problems, a naturalisticstudy is needed. Mislevy and Myford (1995) conductedboth a statistical and a naturalistic analysis. They identi-fied some challenges that raters faced, as well as specific

types of portfolios that were difficult to evaluate.Among the challenges faced by raters, Mislevy andMyford included: (1) the “empathy mode,” a tendencyto evaluate students in terms of potentials; (2) the“bounce effect,” the effect the last portfolio read has onthe reading of the next one (i.e., comparison to anotherstudent is used instead of analyzing the student’swork by defined rubrics); (3) the overuse of irrelevantbackground knowledge for decision making; (4) the useof the middle levels of the scale, which the authors call“play it safe”; and (5) having many or few experienceswith the medium or style in which the student isworking. With respect to portfolios, challenging port-folios are those in which there is a lack of consistency(i.e., good ideas but not enough technique to developthem), different levels of abilities, different sources ofevidence, and unique or very special portfolios.

Pacesetter Spanish PortfolioResearch Project (PHASE I)MethodPortfolios are one among several assessment modesincluded in Pacesetter Spanish, and they provide teachersand students with a tool of great instructional andmotivational value. In Pacesetter Spanish, a portfolio isa collection of significant samples of student work overtime, accompanied by clearly stated evaluation criteriaand students’ reflections on their own learning progress.Most of the activities and final projects of the units inthe Pacesetter Spanish course materials are consideredacceptable choices for inclusion in students’ portfolios.Because of the variety of these activities, portfoliosmay contain a rich array of samples, including videosof student reports and role playing, audio excerpts ofinformal and formal interviews and presentations, draftsand final versions of written work, collages, and otherdescriptions of projects.

Portfolios can yield valuable data about students’progress and lead them effectively through theirlanguage learning. They function as guides by allowingstudents to make choices, helping them to both under-stand and demonstrate how they reason, create, and usestrategies. Also, portfolios promote students’ reflectionon their work and learning. Thus, portfolios helpstudents offer evidence of their progress toward meetingestablished outcomes, and enable them to take on theresponsibility for their own learning process.

The Pacesetter Spanish portfolio research projectgoals are: (1) to prepare a working definition of Pace-

5

setter Spanish portfolios as assessment tools, (2) todevelop a standardized portfolio assessment system,and (3) to use the project findings for training teachersin upcoming professional development sessions.

To achieve these goals, several issues were addressed:(1) the development of an assessment matrix to takeinto account course outcomes, performance indicators,and scoring rubrics for the Culminating Assessment, (2)piloting and refining the resulting assessment matrix,and (3) identifying benchmarks and developing a setof sample portfolios to be used for training purposes.

Project personnel included two ETS project staffmembers from the Pacesetter Spanish program and asummer intern. Andrea Fercsey and Carmen Luna of theAssessment Division coordinated this phase of theresearch with the help of Eva Ponte, graduate student atthe University of California at Berkeley, who was thesummer intern assigned to work on this project. SixPacesetter Spanish teachers participated in the researchproject, representing four different districts in which theprogram is being implemented. It should be noted thatthese participating teachers had a very short period oftime (two months) to devote to the implementation ofportfolios in their classes. Such a limitation posedvarious constraints, and it is expected that a full schoolyear devoted to portfolios would ease these extraneousproblems.

The first meeting was held in April 1996. Colleagueswho had worked on the Pacesetter English portfolioproject presented their experience with their program.The goals, issues, and design of the Pacesetter SpanishPortfolio Research Project were discussed and revisedas needed.

The role of portfolios as an assessment tool was clari-fied: Pacesetter Spanish portfolios would be used toevaluate classroom products and not processes. However,since portfolios are a collection of products over time, theinformation gathered through their evaluation may beused to enlighten instructional materials and methods.

Through discussions with participants, it becameapparent that teachers needed much guidance tobe better prepared to help their students select samplesfor portfolios. It was emphasized that the materialsneeded to be extremely teacher friendly, and issues ofportfolio management (especially logistics and storage)were also discussed. It was determined that the contentof Spanish portfolios for this phase of the researchproject should be two writing samples, two speakingsamples, one reading comprehension sample, onelistening comprehension sample, and the final projectsfor the units. It was made clear that this list of artifactswas only a suggestion open to revision.

A consensus was reached to work with the sameterminology used in the five strands of the Pacesetter

Spanish Culminating Assessment and end-of-yearreport. These five strands are: “Beginning,” “Develop-ing,” “Promising,” “Accomplished,” and “Advanced.”Using course outcomes as the basis for the elaborationof the assessment matrix, two dimensions were defined.The first dimension was called “Demonstrating Knowl-edge of Hispanic cultures,” and the second dimensionwas named “Using Spanish to Communicate Effec-tively.” The three aspects of Dimension 1 are notevaluated explicitly in the Culminating Assessment. Adecision was made to include Dimension 1 in portfolioassessment to better address the course outcomes. Apreliminary assessment matrix was then developed, andseveral aspects were included within each dimension. Apreliminary description of each of the five strands of thescale was developed. A cover sheet and basic guidelinesfor students were reviewed and approved. An exampleof these forms can be found in Appendix 1.

The assessment matrix included three aspects underDimension 1, “Demonstrating Knowledge of HispanicCultures”: Showing awareness of the diversity of Hispaniccultures, Identifying contributions of Hispanic figures, andMaking connections with other disciplines and ownculture(s). The second dimension, “Using Spanish to Com-municate Effectively,” included two aspects: Derivingmeaning from texts and personal interactions (receptiveskills: listening and reading) and Expressing meaning inoral and written form (productive skills: speaking andwriting). This version of the assessment matrix is includedin Appendix 2. Concerning the use of the assessmentmatrix, it was noted that some artifacts may be used toevaluate different dimensions—and even different aspectswithin those dimensions—so raters have to be prepared todo multiple readings of the same work as needed (e.g., avideotape may need to be seen twice: the first time toevaluate the student’s knowledge of Hispanic cultures,and the second time to evaluate the student’s ability tocommunicate in Spanish). For all aspects of bothdimensions, the matrix also included a category called“Not Enough Evidence to Judge.” The raters were to usethis category when they felt they were unable to give arating because of a lack of sufficient artifacts.

During April and May 1996, participating teacherscarried out a more concerted implementation of port-folio assessment in their own classrooms. Students wereinformed of the draft assessment matrix, cover sheet,and guidelines. It was pointed out to both students andtheir parents that this was a research project, and thatthe findings would facilitate new approaches andrevisions to the Pacesetter Spanish program. Prior to thesecond meeting, a scoring sheet was also developed.(This form is shown in Appendix 3.)

At the second meeting in June 1996, the assessmentmatrix was piloted, and sample portfolios were read

6

and scored. Participating teachers were asked to provide25 portfolios each. It should be noted that one partici-pant was unable to attend. However, she did send thecorresponding portfolios for her students, and thesewere integrated as part of the pool to evaluate. Duringthe reading sessions, raters were asked to identify theartifacts and contents of all portfolios in the scoringsheets. The transcription of comments written in thescoring sheets is included in Appendix 5.

This second meeting started with a session devoted tofeedback. Participants discussed the implementation ofportfolios in their classrooms and noted problems theyencountered. First, they mentioned they had too littletime to give their students a clear understanding of howportfolios work and enable the students to gather thenecessary information. These difficulties seem inherent inthis research project. The particular time constraints ofthe project should not be problematic when portfolios areimplemented in Pacesetter Spanish at the beginning of theschool year. Second, participants related some problemswith portfolio management throughout the school year.ETS project staff suggested the following stages to helpmanage portfolios. During the first marking period ofthe academic year, students should concentrate on theorganizational aspect of portfolios. During the secondmarking period, students should become fully cognizantof the rubrics. During the third marking period, a confer-ence should be held with individual students to ensurethat they can fully justify each artifact chosen at thatpoint and that they become aware of any aspects not dulyaddressed. Then, students should finish gathering allnecessary evidence for their portfolios. At the end of theprocess, there should be no glaring omissions or lack ofclarity. A final conference should then be conducted.

Some participants also indicated that several studentsviewed portfolios just as a collection of work andhad problems understanding the assessment matrix. Toaddress this issue, it was suggested that in the futurestudents could be presented with several portfoliosshowing evidence of performance for each level in thescale (i.e., from Beginning to Advanced).

Participants also mentioned some positive aspects ofimplementing portfolios. Less able students felt they“had a shot at this.” Students with average abilities saidthey felt very comfortable with this approach to learning.Also, as students were selecting pieces for their port-folios, they were given the opportunity to improve theirproducts, motivating them to go over their work. Thisprovided extremely beneficial effects in terms of boththeir learning experience and outcomes. More important,students felt they were learning to teach themselves andwere becoming more aware of their own processes.

Both teachers and ETS project staff were extremelypleased with the quantity and quality of the portfolios.

Initially some of the participants had doubted thepossibility of using a portfolio system in their class-room. While they faced difficulties during the initialstage of implementation, they felt a great sense ofaccomplishment at the end of the project. ETS projectstaff were delighted to see that both teachers andstudents were truly becoming fully involved in thedevelopment and implementation of portfolios in thePacesetter Spanish program.

In the next step of the meeting, the group as a wholeselected portfolios that were thought to be representativeof different levels of the scale. These portfolios werediscussed with the guidance of ETS project staff. Duringthese discussions, some teachers reported problems theyfaced when portfolios contained pieces at different levelsof performance. ETS project staff indicated that since itis usual for students’ learning to register different “peaksand valleys” throughout the school year, their artifactswould consequently show evidence of such differences.

During this part of the meeting, teachers alsoreported that poorly organized portfolios were verydifficult for teachers to read. ETS project staff indicatedthat students should be strongly encouraged to properlyorganize their portfolios. (For instance, when studentsworking as a group submit a video with a presentationor a tape with an interview, they should at all timesidentify themselves. Otherwise, raters who are not theirteachers will find it impossible to identify students inorder to evaluate their individual work.)

The next planned activity at the meeting was thereading and scoring of portfolios. Because of their lackof experience in similar endeavors, participants decidedto start by working in pairs. Participants and ETSproject staff then started by reading two portfolios inpairs, afterward moving on to read and score portfoliosindividually. Twenty-one portfolios were selected to beread twice. ETS project staff were used as third raters incase of large discrepancies. Originally, it was expectedthat each rater would read six portfolios; however,because some raters evaluated portfolios faster thanother raters, two raters judged five portfolios, two ratersread the expected number of portfolios (six), andanother two raters read seven portfolios.

In all, a total of 30 portfolios were read: 8 portfolioswere read in pairs, 21 portfolios were read individuallyby two different raters (double reading), and 1 was readby only one rater. The flow of portfolios for this readingwas organized so that teachers did not read their ownstudents’ portfolios. Raters rated each aspect and thengave a total score for each dimension.

During the final part of the meeting, the assessmentmatrix was discussed, taking into account knowledgegained from the reading sessions. A discussion tookplace on whether Aspect 1 (Showing awareness of the

TABLE 1

Distribution of RatingsCategory Number Category Name Frequency Percentage

1 Beginning 12 2.91

2 Developing 23 5.57

3 Promising 158 38.25

4 Accomplished 114 27.60

5 Advanced 10 2.42

0 No Evidence 96 23.24

TOTAL 413 100.00

Distribution N = 317 Mean: 3.27 SD: .80

7

diversity of Hispanic cultures) and Aspect 2 (Identifyingcontributions of Hispanic figures) could really beconsidered different. The conclusion was that they wereindeed different, but that teachers did not have a clearenough idea of what they were looking for in a port-folio. Consequently, they were not very effective incommunicating this aspect to their students. Also, ratersoften indicated that there was “not enough evidence” toevaluate students’ work on Aspect 3 of Dimension 1(Making connections with other disciplines and ownculture[s]). This may have occurred because this aspectwas inadvertently omitted from the cover sheet thatstudents were told to use to write their justification foreach selected piece. It was also noted that in most casesthere was little or no evidence to judge Dimension 1;there seemed to be a problem with both the quantityand quality of artifacts used to show evidence oflearning for Aspect 1. There was usually plenty ofevidence to judge Dimension 2, and this may be due toboth students and teachers having experience evaluatingproductive language skills in their courses.

At the end of the process, the parameters initiallyindicated for selecting the required number of pieces in astudent’s portfolio were found to be inadequate, resultingin portfolios that seemed to be lacking in the necessaryevidence to judge all aspects of both dimensions. Adecision was made: There would no longer be a setnumber of pieces to be included, and more emphasiswould be placed on the process of selecting enoughartifacts to show evidence of a student’s accomplishments.

Portfolio AnalysesAfter the second meeting, all data and information wereanalyzed. Certain limitations of this portfolio assess-ment should be considered. First, guidelines to selectmaterials were just tentative. Second, the assessmentmatrix was treated as a pilot. Third, certain aspectswere quite new to both students and teachers, and thetime constraints prevented them from becoming familiarwith those new aspects.

The group of raters was composed of five PacesetterSpanish teachers and two ETS project staff members.None of the raters had any prior experience scoring aSpanish portfolio, and only one ETS staff had priorexperience scoring portfolios—but in another contentarea (English as a Second Language).

Training was done first as a whole group, then in pairs.While reading portfolios as a group, raters were asked tohave the assessment matrix present at all times, andto think of ways in which they were using it to makedecisions. Raters were also asked to reflect on the use ofspecific evidence to justify the ratings they were assigning.

Finally, they were asked to note all the problematicaspects they found when scoring any portfolio. Whenraters were reading in pairs, the two of them discussedand decided on the ratings for each aspect and holisticallyfor the two dimensions. Next, the group as a whole dis-cussed and shared their experiences, and then the ratersstarted reading and scoring portfolios individually.

Description of ElementsAs mentioned earlier, 30 portfolios were read. Therewas a total of 59 portfolio readings: Eight portfolioswere read in pairs, 21 were read individually with tworaters per portfolio (42 readings), 8 portfolios were readby a third rater when discrepancies of 2 points or morewere found between the first two raters, and 1 portfoliowas read by only one rater.

Distribution of Ratings To analyze the distribution of ratings, we constructedtwo tables, one showing the frequencies of ratings givenacross portfolio aspects and the other showing thedistribution of ratings for each portfolio aspect anddimension. The rating scale was composed of fivecategories: “Beginning,” “Developing,” “Promising,”“Accomplished,” and “Advanced.” There is anothercategory in the analysis, named “Not Enough Evi-dence,” not included in the rating scale. (See page 8.)

Table 1 shows that the categories with the highestfrequencies are 3 and 4 (i.e., “Promising” and “Accom-plished”), which, when combined, accounted for abouttwo-thirds of all ratings given. Overall, students’ perfor-mances tend to be somewhat above average (i.e., 68percent of the ratings were 3 or higher while fewer than9 percent were 2 or lower), with more studentsperforming in the middle and upper half of the scale.The high percentage of judgments falling in the category“Not Enough Evidence” is also noticeable.

8

To study student performance and the incidence ofthe category “Not Enough Evidence” in more detail,Table 2 displays the distribution of frequencies in eachrating category for each aspect and dimension.

As expected, the rating category “Promising” has thehighest frequencies. There are only two aspects in whichthis does not occur: Aspect 3, in Dimension 1, and Aspect1, in Dimension 2. In the first case, Making connectionswith other disciplines and own culture(s), the categorywith the highest frequency is “Not Enough Evidence”(71 percent). The reason for this may be that this aspectwas not included in the “cover sheet” form given tostudents to select their work and justify their selection.In the second case, Deriving meaning from texts andpersonal interactions, the rating category with the highestfrequency is “Accomplished.” Not surprisingly, students’performance in this aspect (receptive skills) tended to be

better than in other aspects. This conforms to knownpatterns of language acquisition. In general, teachers hadmore experience evaluating both aspects of Dimension 2throughout the Pacesetter Spanish course. Also, studentshad less difficulty identifying artifacts to show evidenceof their achievements in Dimension 2. (They tend toshow their command of a language in terms of theirlistening, speaking, reading, and writing abilities.)

Included in Table 2 are the means and standarddeviation of the score distributions for each aspect andfor the total in each dimension. The statistics describingthese distributions were calculated for each aspect anddimension. When we calculated the means and standarddeviations, we did not include the category “Not EnoughEvidence.” The means ranged from 2.8 (total Dimension1) to 3.54 (Aspect 1, Dimension 2). The overall mean,without making distinctions between aspects and dimen-sions, is 3.27. This indicates, as we have seen before, thatoverall students’ performance is medium-high. When wereview the means for the aspects, we see that students’performance tends to be similar across the differentaspects and dimensions. Variability within dimension,measured by the standard deviation, is also similar acrossdifferent aspects and dimensions: It ranges from .61(Aspect 1, Dimension 2) to .94 (Aspect 3, Dimension 1).

In general, we see that most ratings fell in themedium-high categories of the scale (“Promising” to“Accomplished”). A much smaller number fell in theother categories (“Beginning,” “Developing,” and“Advanced”). Aspect 1, Dimension 1 (Showing aware-ness of the diversity of Hispanic cultures), wasthe aspect having the highest percentages in categories“Beginning” and “Developing.”

Time Used to Read PortfoliosBecause this was the first time that portfolios were readand scored, we were interested in the amount of timeused to read them (as shown in Table 3). Raters wereasked to keep track of this. Some forgot, hence themissing data in Table 3.

As indicated in Table 3A, the average time employedto read a portfolio was 46.74 minutes. The shortestamount of time needed was 20 minutes while thelongest was 120 minutes (Table 3). Several tables wereconstructed to examine whether the amount of timeneeded to read a portfolio was different for portfolioscoming from different Pacesetter Spanish teachers, fordifferent raters, and for readings in different sessions.

As indicated in Table 4, there is some variation in termsof the time raters needed to rate portfolios from differentteachers. On average, portfolios from the class of Teacher2 took the shortest amount of time to read (41.25 minutes

TABLE 2

Distribution of Ratings by Aspects and Dimensions

Dimension 1

Category Category Aspect 1 Aspect 2 Aspect 3 Total Name Freq. % Freq. % Freq. % Freq. %

1 Beginning 3 5 2 3 1 2 4 7

2 Developing 8 14 4 7 0 0 4 7

3 Promising 26 44 24 41 9 15 25 42

4 Accomplished 11 19 12 20 5 9 9 15

5 Advanced 2 3 1 2 2 3 2 3

0 No evidence 9 15 16 27 42 71 15 25

TOTAL 59 100 59 100 59 100 59 100

Aspect 1 Aspect 2 Aspect 3 Total

N 50 43 17 44

Mean 3.02 3.14 3.4 2.80

SD 0.89 0.80 0.94 0.84

Note: These statistics have been calculated without taking intoaccount the category “No Evidence.”

Dimension 2

Category Category Aspect 1 Aspect 2 Total Name Freq. % Freq. % Freq. %

1 Beginning 0 0 0 0 2 3

2 Developing 2 3 2 3 3 5

3 Promising 20 34 28 48 26 44

4 Accomplished 27 46 27 46 23 39

5 Advanced 1 2 1 2 1 2

0 No evidence 9 15 1 2 4 7

TOTAL 59 100 59 100 59 100

Aspect 1 Aspect 2 Total

N 50 58 55

Mean 3.54 3.24 3.33

SD 0.61 0.86 0.77

Note: These statistics have been calculated without taking intoaccount the category “No Evidence.”

9

TABLE 3

Time Used to Read Each PortfolioStudent Student’s Teacher Rater Date Minutes

1 3 7 St/Ev 20

1 3 6 Su/M 35

2 1 6 St/Ev 20

2 1 4 St/Ev 30

3 6 4 St/Ev 20

3 6 3 Su/M 55

4 3 8 St/Ev 65

4 3 2 Su/M 40

5 3 1 St/Ev 60

5 3 7 St/Ev 35

6 4 1 St/Ev 30

6 4 7 St/Ev 30

7 4 2 St/Ev

7 4 8 Su/M 50

8 1 4 St/Ev

8 1 3 Su/M 35

9 2 7 St/Ev 20

9 2 6 St/Ev 25

10 6 3 St/Ev 50

10 6 1 Su/M 20

11 5 3 St/Ev

11 5 1 Su/M 40

12 1 7 St/Ev 40

12 1 6 Su/M 45

13 2 6 St/Ev 45

13 2 4 St/Ev 35

14 2 4 St/Ev 40

14 2 6 40

15 5 1 St/Ev 30

15 5 7 Su/M 40

16 5 2 St/Ev 60

17 5 8 St/Ev 70

17 5 4 Su/M 55

18 3 6 St/Ev 40

18 3 4 Su/M 30

19 6 2 St/Ev 45

20 4 3 St/Ev 60

20 4 1 Su/M 35

21 6 8 Su/M 50

21 6 2

22 1 2 & 3 St/M

23 1 8 St/M 90

24 2 4 & 7 St/M 95

25 2 7 Su/M 30

26 3 4 & 7 St/M-Ev

27 3 4 & 7 St/M 55

28 6 2 & 3 St/M 120

29 5 1 & 6 St/M 100

30 4 1 & 6 St/M 80

TABLE 3A

SummaryMean time used to read portfolios: 46.74 minutes

Standard Deviation: 22.83

TABLE 4

Average Time Employed to Read Portfolios FromDifferent TeachersTeacher N Mean St. Dev. Minimum Maximum

1 6 43.33 24.43 20 90

2 8 41.25 23.26 20 95

3 9 42.22 14.81 20 65

4 6 47.50 19.94 30 80

5 7 56.42 23.58 30 100

6 7 51.42 33.50 20 20

N = Number of portfolios readMinimum = Minimum time neededMaximum = Maximum time needed

per portfolio); portfolios from the class of Teacher 5took the longest to read (56.42 minutes per portfolio).

Table 5 displays the average time each rater neededto read a portfolio. Raters worked at different paces;consequently, we combined different pairs of raters, sothat all 21 portfolios were read twice. In Table 5, Raters1 through 7 are individual raters, number 8 refers toRaters 4 and 6, number 9 to Raters 2 and 3, and num-ber 10 to Raters 1 and 5. Although the sample is small,it took more time to read portfolios in pairs thanindividually. This was expected because discussion tookplace during the reading. Within individual raters, Rater6 was the fastest rater, and Rater 7 was the slowest.

Table 6 shows the average time needed to read portfo-lios in each session. Session 1 ran from Saturday morningthrough the afternoon; Session 2 took place the afternoonand early evening of Saturday; and Session 3 took placeSunday morning. Raters took more time on average to rateportfolios during the first session. This was expected sinceit was the first Pacesetter Spanish portfolio evaluation, andbecause the readings in this first session were done mostlyin pairs. There is almost no difference in the average timeneeded to rate portfolios during Sessions 2 and 3.

Culminating AssessmentAs previously indicated, Pacesetter Spanish includes astandardized assessment component called the Culmin-ating Assessment. This assessment was developed forthe areas of reading, speaking, listening, and writing.Reading and speaking were locally scored by individualteachers, and listening and writing were centrally scored

10

at a national reading. Dimension 2 of the portfolio scor-ing sheets measures language, and it is divided into twoaspects: receptive skills (listening and reading), and pro-ductive skills (writing and speaking). Although we cannot directly compare these aspects with the areas of theCulminating Assessment, we wanted to examine whetherthe patterns of these two modes of assessments weresimilar (i.e., if a student got a high score in the Culmin-ating Assessment, we expected him or her to get a highscore in the respective aspects of the portfolio). Both theCulminating Assessment and portfolio scales contain fivestrands, and there was an attempt to maintain parallelismin the descriptors for each strand, especially for Dimen-sion 2 (where the constructs are similar).

Table 7 displays the scores obtained by students bothin the Culminating Assessment and in the portfolio. Theinformation displayed in this table indicates thatin general portfolio scores tended to be lower thanCulminating Assessment scores. This may have happenedbecause those skills not included in the CulminatingAssessment are usually harder for students to masterand for teachers to assess.

The overall correlation coefficients between theCulminating Assessment and the portfolio scores werelow (r = .15 between Culminating Assessment listening

grade and portfolio Dimension 2, Aspect 1 Derivingmeaning from texts and personal interactions; and r = .07between Culminating Assessment writing grade andportfolio Dimension 2, Aspect 2, Expressing meaning inoral and written forms). These results suggest that theCulminating Assessment and the portfolio assessmentmay be tapping different sets of knowledge and skills,even though the constructs were assumed to be similar.

Portfolio Assessment MatrixCorrelationsThe portfolio’s assessment matrix included sevenratings: one rating for each of the three aspects of

TABLE 6

Time Employed to Read Portfolios at Different SessionsSession N Mean St. Dev. Minimum Maximum

1 6 90.00 21.68 55 120

2 22 39.54 15.65 20 70

3 14 40.00 10.19 20 55

N = Number of portfolios readMinimum = Minimum time neededMaximum = Maximum time needed

TABLE 7

Culminating Assessment and Dimension 2 PortfolioRatingsStudent Culminat. Portfolio Culminat. Portfolio

Assess. Dim. 2 Assess. Dim. 2Listening Aspect 1 Writing Aspect 2 Grade Rating Grade Rating

1 3 3 3 3

2 4 2.5 3 3

3 3 2.5 3 4

4 2 4 3 3.5

5 5 4 5 3

6 3 4 3 3

7 4 4 3 3.5

8 5 3.5 4 4

9 5 4 5 4

10 5 4 4 3

11 4 4 5 4

12 5 3.5 5 3

13 4 2.5 4 3

14 5 3 4 3

15 4 3.5 3 3.5

16 5 4 4 4

17 5 4 4 5

18 5 3.5 5 3

19 5 2.5 3 3.5

20 4 4 4 3.5

21 5 4 5 4

22 5 4 4 4

23 5 3 5 3

24 4 3 5 3

25 5 3 5 3

26 4 4 4 4

27 4 3 3 3

28 5 3 4 4

29 2 3 2 3

30 5 4 4 4

Mean 4.30 3.47 3.93 3.48

TABLE 5

Average Time Employed by Each RaterRater N Mean St. Dev. Minimum Maximum

1 6 35.83 13.57 20 60

2 3 48.33 10.41 40 60

3 4 50.00 10.80 35 60

4 6 35.00 11.83 20 55

5 7 35.71 9.76 20 45

6 7 30.71 8.38 20 40

7 5 65.00 16.58 50 90

8 2 75.00 28.28 55 95

9 1 120.00 120 120

10 2 90.00 14.14 80 100

N = Number of portfolios readMinimum = Minimum time neededMaximum = Maximum time needed

11

each facet. We have three facets: students, readers, andportfolio aspects and dimensions. Some students weremore proficient than others, some readers rated moreseverely than others, and some portfolio aspects wereharder to get high ratings on than others. This variationamong students, raters, and aspects is expected; how-ever, unexpected variations may signal attention tounresolved or unnoticed problems; i.e., those parts of thedata that seem not to follow usual patterns of variability.

FACETS produces a map in which all facets of theanalysis are shown in one figure, providing the readerwith general information for each facet. We include inFigure 1 a reproduction of the FACETS map.

FACETS calibrates all facets of the analysis so thatthey can be positioned on the same equal interval scale.This scale is shown in the first column of the map.

Students, the first facet of our study, occupies thesecond column of the FACETS map. This column dis-plays the estimates of students’ proficiency on theportfolio assessment. Student measures are orderedwith more proficient students at the top of the columnand less proficient students at the bottom of the column.Student proficiency measures range from −3.2 logits to3.2 logits.

The third column in Figure 1 represents raters interms of the harshness or leniency they exercised whenrating portfolios. Harsher raters appear at the top of thecolumn, and more lenient raters appear at the bottom.Rater 6 was the harshest, and Rater 3 was the mostlenient. Rater harshness measures range from −2.3 logitsto 2.3 logits. As expected, this variation is smaller thanthe variation found in student measures.

The fourth column of the map displays the portfolioin terms of its aspects. The hardest aspects are at thetop of the column and the easiest at the bottom. Thisindicates that Aspect 3, Dimension 1 was the hardest toget high ratings on, and Aspect 2, Dimension 1 was theeasiest one to get high ratings on.

Columns 5 through 11 show the most probablerating for a student at a given level on the logit scale, as

Dimension 1, one rating for each of the two aspectsof Dimension 2, and two ratings as totals for bothdimensions. We include in Table 8 the correlationsamong all these ratings.

All correlations between aspects and dimensions aresignificant at the .06 level (except for the correlationbetween Aspect 1, Dimension 2 and Aspect 2,Dimension 1), and range from moderate (.25) to high(.72). Correlations among the three aspects underDimension 1 range from .30 to .51, while the corre-lation between the two aspects of dimension 2 is .29.Two correlations are moderately high: the correlationbetween Aspect 2, Dimension 1 and Aspect 2, dimen-sion 2 (.46), and the correlation between Aspect 3,Dimension 1 and Aspect 2, Dimension 2 (.54). Thecorrelations of Aspects 1 and 2 of Dimension 2 withthe total of Dimension 2 are .61 and .52.

Portfolios Read Twice:Reliability StudyPortfolio assessment is based on personal judgments.One common approach to studying the generalizabilityand fairness of the judgments is to ask a second personto read and score the portfolios. Due to the nature ofthe project and the stage in which the Pacesetter Spanishprogram was at that moment, and also due to timelimitations, it was only possible to obtain a smallsample of portfolios read twice.

FACETS (Linacre, 1989), a computer program basedon Rasch partial credit models, was used to analyze thedata. FACETS analyzes data from assessments that haveseveral facets (e.g., raters, students, tasks) by assigningparameters to each facet in the model. FACETS has theform of a log-linear model for main effects and estimatesthose effects in logits (logits are the logarithmic odds ofa given rating compared to the next lower one). Linearmeasures are then constructed from ordered qualitativedata, allowing the separation of the contribution of

TABLE 8

Portfolio Assessment Matrix Correlations Dimension 1 Dimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2 Dimension 2Aspect 1 Aspect 2 Aspect 3 Total Aspect 1 Aspect 2 Total

Dimension 1 Aspect 1

Dimension 1 Aspect 2 .37

Dimension 1 Aspect 3 .51 .30

Total Dimension 1 .72 .69 .68

Dimension 2 Aspect 1 .26 .13 .36 .35

Dimension 2 Aspect 2 .30 .46 .54 .43 .29

Total Dimension 2 .34 .25 .34 .43 .61 .52

12

expected from a rater with average harshness, in each ofthe portfolio aspects and dimensions. The horizontallines across a column indicate the point at which thelikelihood of getting the next higher rating begins toexceed the likelihood of getting the next lower rating.For instance, if we look at scale 1 (Aspect 1, Dimension1), we see that students with measures from −2.1 to 1.5logits are more likely to receive a 3 (“Promising”) thanany other rating on that aspect.

In an effort to study how reliable raters’ judgmentswere, and also with the aim of identifying problematicportfolios, we analyzed the discrepancies in ratingsbetween raters of the same portfolio. Table 9 includesthe number and percentage of scoring discrepancies foreach aspect of the portfolio, and in total.

As we have indicated before, the scale covers fivelevels: beginning, developing, promising, accomplished,and advanced. We also had a category for cases where

Figure 1. FACETS map for the whole scale (21 portfolios, two readers).

13

there was not enough evidence to judge. In the last row ofthe table, we have included those situations in which onerater said there was no evidence, and the other consid-ered there was enough evidence and gave a judgment.

The number of agreements is moderate; it constitutes40 percent of all paired judgments. (However, it shouldbe noted that cases where both raters considered therewas no evidence to judge have been included here as“agreements.”) The number of portfolios having only a1-point discrepancy is also moderate (33 percent of allpaired judgments). Finally, the number of discrepanciesof 2 points was small (3 percent).

An important finding is that the number of discrep-ancies between raters who judged student work and

raters who said there was not enough evidence to judgewas considerable (23 percent of all paired responses).Most discrepancies of this type occurred in Dimension 1,especially in Aspect 3.

We were also interested in seeing at which specificlevels of the scale these discrepancies occurred. Table 10displays those discrepancies for each portfolio aspectand for the portfolio as a whole.

The type of discrepancy with the highest occurrence(41 percent) is that in which one rater considered astudent’s performance promising while the other con-sidered it accomplished (discrepancy 3-4). Another typeof discrepancy with a high percentage of occurrence (22percent) was that in which one rater thought there was

TABLE 9

Summary of Portfolio Score DiscrepanciesDimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2

Aspect 1 Aspect 2 Aspect 3 Aspect 1 Aspect 2 D 1 D2 Total Percentage

Same Score 8 7 14 4 11 6 8 59 40

1–Point Disc. 6 7 10 10 7 9 49 33

2–Point Disc. 2 1 1 1 5 3

Evid. vs. Noevidence 5 6 7 6 7 3 34 23

TABLE 10

Summary of Portfolio Score Discrepancies for Each Aspect in Each DimensionDimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2 Dimension 1: Dimension 2: Total

Discrepancy Aspect 1 Aspect 2 Aspect 3 Aspect 1 Aspect 2 Total Total Frequency Percentage

No Evid.–1 1 1 1

No Evid.–2 1 2 2 5 6

No Evid.–3 3 4 3 3 4 2 19 22

No Evid.–4 3 2 1 1 7 8

No Evid.–5 1 1 2 2

1–2 1 1 1

1–3 1 1 1

2–3 2 1 1 2 2 2 10 11

2–4 1 1 2 2

3–4 3 6 9 7 5 6 36 41

3–5 1 1 2 2

4–5 1 1 2 2

TOTAL 13 14 7 17 10 15 12 88

(Discr.)

Agreement

(full scores) 8 6 2 4 11 5 9 45

Agreement

(No Evid., 0-0) 1 12 1 14

TOTAL

(Agreement) 8 7 14 4 11 6 9 60

14

no evidence to judge, and the other considered thework showed evidence of being promising (difference“No Evidence” – 3). This latter type of discrepancy ismore important and indicates that raters’ internalizationof the assessment matrix and search of evidence wasproblematic. One possible explanation is that someraters judged only those things students had indicated asevidence of their learning in a specific area, and otherraters used materials not marked by the student for thatspecific area. The only aspect in which there are nodiscrepancies of this type is Aspect 2, Dimension 2.

Aspect 3, Dimension 1 registered the fewest discrep-ancies. It should be noted, though, that most agreementswere of the type “No Evidence”–“No Evidence,”meaning that both raters judged there was not enoughevidence to evaluate a student’s performance.

Usually a 1-point discrepancy is considered smallenough to categorize the rating as an acceptablejudgment. However, problems may arise when thereis a cut point. FACETS provides a “fair” measure, whichindicates the score a student would have received had

his or her ratings been adjusted for rater effects. Evensmall adjustments can be important for those studentswhose scores lie in critical cut-score regions. Table 11shows each student’s observed average score and hisor her “fair” average score (i.e., adjusted for severity orleniency of the raters providing the ratings for thatstudent).

The reported scores of six students would havechanged after results were adjusted for rater severity orleniency. Four students (students 5, 6, 15, and 12)would have changed their reported score from “Promis-ing” to “Accomplished,” and two students (students 20and 8) would have changed their scores from“Accomplished” to “Promising.”

We also studied the behavior of each rater separately.Table 12 indicates how many times each rater usedthe judgment “No Evidence” and the other levels of thescale.

Raters 4 and 6 used the category “Not EnoughEvidence to Judge” very frequently. In each case over 35percent of their judgments were “Not Enough Evidence.”

TABLE 11

FACETS OUTPUT (7.1.1) Students Measurement Report (Arranged by FN).

15

As we can see, these raters also read more portfolios thanthe others. In future analysis, it may be interesting tostudy the relationship between time used to read portfo-lios, search for evidence, and use of the “No Evidence”category.

We can intuitively see why FACETS indicated rater 3as the most lenient: 69 percent of the ratings that thisrater gave were 4’s. No other rater gave that high apercentage of 4’s. Rater 6 was the harshest; only 8percent of the ratings this rater gave were 4’s, and therater gave no 5’s.

Third Rater AnalysisUsing only the portfolio ratings for total Dimension 1and total Dimension 2, we selected some portfolios withextreme discrepancies. We defined “extreme” as port-folios with dimension ratings 2 or more points apart, andalso portfolios with ratings of evidence (any evidence)as opposed to “Not Enough Evidence to Judge.” Dueto time constraints, not all portfolios were read twicewhen this selection was made. Eight portfolios wereselected for a third reading.

We analyzed the ratings each rater gave to eachportfolio aspect (see Appendix 5). Table 13 summarizesthe results of this analysis.

As we can see, portfolios 14 and 17 produced a lot ofdiscrepant ratings, indicating that these were problem-atic portfolios. Discrepancies mostly occurred in Aspect1, Dimension 1 (4 portfolios) and Aspect 1, Dimension2 (3 portfolios).

Selection of BenchmarksAfter the June 1996 meeting in Miami, ETS project staffselected some portfolios to be used as benchmarks. Stafftook into account the discrepancies between raters aswell as other factors such as content and variety ofartifacts when selecting the benchmarks. The selection

was made using a table in which portfolio scoreswere presented by matched pairs of raters (this table isincluded in Appendix 4; those portfolios selected asbenchmarks are indicated in bold). Besides the portfo-lios selected using the information given in thementioned table, another portfolio that was read by agroup was also selected. Thus, a packet of portfoliomaterials was prepared and used at the three professionaldevelopment sessions carried out in summer 1996 fornew Pacesetter Spanish teachers. The midyear meetingsat the end of the fall semester of 1996 also includedsessions on portfolios, and the samples from the trainingpackets served to disseminate the Pacesetter Spanishportfolio model.

Also, as an ancillary result, the rubrics for the fourlinguistic skills were revised, and a decision was made thatPacesetter Spanish would have a unified set of rubrics tobe used throughout the course. It is expected that bothteachers and students will become thoroughly familiarwith these rubrics and should encounter little difficultyapplying them to their work during the school year.

In order to analyze the information gathered fromthe portfolios, and to provide some useful feedback toteachers, all scoring sheets with all written commentswere transcribed. A copy of these transcriptions can befound in Appendix 5.

TABLE 12

Distributions of Raters’ JudgmentsBeginning Developing Promising Accomplished Advanced Not Enough Total Number

Rater 1 2 3 4 5 Evidence of Judgments

1 2 ( 5%) 23 (55%) 13 (31%) 4 ( 9%) 42

2 5 (14%) 11 (31%) 15 (43%) 4 (11%) 35

3 1 ( 3%) 1 ( 3%) 7 (20%) 24 (69%) 2 ( 6%) 35

4 1 ( 2%) 15 (31%) 4 ( 8%) 6 23 (47%) 49

5 1 ( 2%) 8 (19%) 17 (40%) 10 (24%) 6 (14%) 42

6 27 (55%) 4 ( 8%) 18 (37%) 49

7 1 ( 2%) 1 ( 2%) 18 (43%) 17 (40%) 5 (12%) 42

TABLE 13

Portfolios Read Three TimesPortfolio Ratings in Agreement Ratings 2 points apart

3 6 1

5 5 2

8 5 2

12 5 2

13 6 1

14 3 3

17 1 6

18 7 0

16

Summary of Findings• This research showed that some elements of the

delineated Pacesetter Spanish system of portfolioassessment worked as intended, while others needfurther refinement.

• Pacesetter Spanish portfolio raters very often used themedium category of the scale (“Promising”), and notthe extreme categories (“Developing” or “Advanced”).This may have been due in part to the small sample ofportfolios read. In addition, students in a level threelanguage class tend to cluster in the middle of the scale,which confirms actual classroom experience.

• Often the length of time needed to read a PacesetterSpanish portfolio depended on the variety and com-plexity of artifacts included (i.e., a portfolio with manydifferent types of media took longer to read than aportfolio with only paper-based samples). Althoughsuch variety and complexity are encouraged throughoutthe course framework, this element of time needs to betaken into account when preparing to read portfolios.

• The correlation between Dimension 2, “UsingSpanish to Communicate Effectively,” and thePacesetter Spanish Culminating Assessment was low.These findings suggest that both assessment instru-ments serve to evaluate different sets of knowledgeand skills, although the construct may appear similar.Whereas the Culminating Assessment evaluates fourseparate linguistic skills (reading, speaking, listening,and writing), Dimension 2 of the Pacesetter Spanishportfolio assessment system groups them in twoaspects: Deriving meaning from texts and personalinteractions (receptive skills) and Expressing meaningin oral and written form (productive skills).

• All correlations between aspects and dimensions of thePacesetter Spanish portfolio assessment system werefound to be significant, except for the correlation be-tween Aspect 1, Dimension 2 and Aspect 2, Dimension 1.

• The number of rating agreements was moderate.However, there was a considerable number of dis-crepancies between raters regarding the presence orlack of enough artifacts to judge students’ perfor-mance within the parameters of the PacesetterSpanish portfolio assessment system.

Recommendations• As shown by Phase I results, further research is

needed to refine the assessment matrix and produce afinal version to implement as part of the formalcomponents of the Pacesetter Spanish program.

• A set of more comprehensive and clearer instructionsshould be given to teachers and students aboutthe Pacesetter Spanish portfolio goals, processes, andlogistics. Some suggestions included requesting studentsto correctly copy and identify all samples they presentfor evidence of individual work, as well as reducing alllarge artifacts such as posters to a manageable size.

• It is necessary to discuss in depth the process ofselection of students’ work (i.e., the number of productsnecessary to demonstrate a certain level of perfor-mance). Guidelines for the selection of material forPacesetter Spanish portfolios need further clarification.

• More support should be given to teachers to betterunderstand and teach Dimension 1 as defined in thePacesetter Spanish portfolio assessment system. It isnecessary to define what constitutes evidence ofachievement in this area, how and where to findevidence, and how it can be collected by their students.

• Further discussion is warranted regarding theappropriateness of grouping the four linguistic skillsin the two aspects of Dimension 2 of the PacesetterSpanish portfolio assessment system.

• In order to apply the new revised version of thePacesetter Spanish portfolio assessment matrix, anew research project should include an expandedportfolio reading with more raters and a largernumber of portfolios. It would be beneficial for thefive original teachers who participated in the Phase Ireading to be incorporated in this expanded readingas experienced raters to help train the new teachers.

• The expanded portfolio reading should be carried outfollowing the model delineated in the correspondingsection of the Pacesetter Scoring Handbook.

• In addition to quantitative data, other qualitative datapertaining to discussion and debriefing sessions withparticipants should be collected during the PacesetterSpanish portfolio reading.

• The new assessment matrix and guidelines for port-folios should be disseminated to all schools imple-menting the Pacesetter Spanish program so thatteachers and students can use them.

• New benchmark portfolios from the expandedPacesetter Spanish portfolio reading should be identi-fied, and a set of sample portfolios should beprepared for use in future training sessions.

ReferencesArter, J.A. and Spandel, V. (1992). Using portfolios of student

work in instruction and assessment. Educationalmeasurement: Issues and practice, Spring 92, 36–44.

17

Bridgeman, B., Chittenden, E., and Cline, F. (1995). Char-acteristics of a portfolio scale for rating early literacy.Princeton, NJ: Educational Testing Service.

Educational Testing Service. (1995). Capturing the power ofclassroom assessment. Focus 28. Princeton, NJ.

Elliot, J. (1985). Facilitating action research in schools: Somedilemmas. In R. Burgess (Ed.), Field methods in the studyof education (pp. 235–242). Lewes, England: Falmer Press.

ETS Trustees’ Colloquy. (1995). Performance Assessment:Different needs, difficult answers. Princeton, NJ: Edu-cational Testing Service.

Hardy, R.A. (1995). Examining the costs of performance assess-ment. Applied measurement in education, 8(2), 121–134.

Gifford, B.R. and O’Connor, M.C. (Eds.) (1992). Changingassessments. Alternative views of aptitude, achievementand instruction. Boston: Kluwer Academic Publishers.

Gitomer, D.H. and Duschl, R.A. (1995). Moving toward aportfolio culture in science education. Princeton, NJ:Educational Testing Service.

Koretz, Daniel. (1994). The evolution of a portfolio program:The impact and quality of the Vermont portfolio pro-gram in its second year (1992–93). Los Angeles: NationalCenter for Research on Evaluation, Standards, andStudent Testing.

Koretz, Daniel. (1994). Lessons from an evolving system.Interim report: The reliability of Vermont portfolioscores in the 1992–93 school year. Los Angeles: NationalCenter for Research on Evaluation, Standards, and Stu-dent Testing.

Koretz, Daniel. (1992). Can portfolios assess student perfor-mance and influence instruction? The 1991–92 Vermontexperience. Los Angeles: National Center for Research onEvaluation, Standards, and Student Testing.

Koretz, Daniel. (1992). The reliability of scores from the 1992Vermont portfolio assessment program. Los Angeles:National Center for Research on Evaluation, Standards,and Student Testing.

Mislevy, R.J. (1995). On inferential issues arising in the California learning assessment system. Princeton, NJ:Educational Testing Service.

Myford and Mislevy. (1995). Monitoring and improving aportfolio assessment system. Princeton, NJ: EducationalTesting Service.

Myford, C.M., Marr, D.B., and Linacre, M. (1996). Readercalibration and its potential role in equating for the testof written English. Princeton, NJ: Educational TestingService.

Resnick, L.B. and Resnick, D.P. (1992). Assessing the thinkingcurriculum: New tools for educational reform. In B.R.Gifford, and M.C. O’Connor (Eds.), Changing assess-ments. Alternative views of aptitude, achievement andinstruction. Boston: Kluwer Academic Publishers.

Sheingold, K. and Fredericksen, J. (1995). Linking assessmentwith reform: Technologies that support conversationsabout student work. Princeton, NJ: Educational TestingService.

Sheingold, K., Heller, J.I., and Paulukonis, S.T. (1995): Activelyseeking evidence: Teacher change through assessmentdevelopment. Princeton, NJ: Educational Testing Service.

Shepard, L. (1989) Why we need better assessments. Educa-tional Leadership, (46 7), 4–9.

Wolf, D., Bixby, J., Glenn, J., and Gardner, H. (1991). To usetheir minds well: Investigating new forms of studentassessment. In G. Grant (Ed.), Review of EducationalResearch 17, 31–74. Washington, DC: American Educa-tional Research Association.

PHASE II REPORT

Rationale and Expected OutcomesPacesetter Spanish is a new and challenging third-levelhigh school Spanish course developed by the CollegeBoard and Educational Testing Service (ETS) thatcombines a carefully integrated set of learning outcomes,course materials, and instructional experiences, includingvarious approaches to assessment. Throughout thecourse, there is a provision for student self-assessmentthrough learning logs and related journal activities, forpeer assessment through work in pairs and small groups,for ongoing teacher-developed assessments, and for astandardized end-of-course assessment, as well as forstudents to assemble portfolios with products from theirwork during the year.

A portfolio is a collection of significant samples ofstudent work over time, accompanied by clear criteriafor evaluation, as well as the student’s own reflectionson his or her progress. Portfolios, although not replacingother types of assessment, give teachers and studentsanother tool of great instructional and motivationalvalue. The portfolios for Pacesetter Spanish may containa rich array of samples, including videos of studentreports and role playing, audio excerpts of informal andformal interviews, drafts and final versions of writtenwork, and collages and descriptions of projects. In fact,many of the activities and final projects for the units areacceptable choices for inclusion in student portfolios.

In the case of Pacesetter Spanish, portfolios are tobe used for assessment purposes, not to look at progressbut to evaluate results, thus also serving to inform andgive feedback on instructional materials and methods.Portfolios yield valuable data about students’ achieve-ment and effectively lead students through the processof acquiring a new language. They serve as a guide forstudents as they make choices and demonstrate howthey reason, create, strategize, and reflect. Ultimately,portfolios allow students to take on responsibility fortheir own learning process and offer evidence of their

18

progress toward meeting the course outcomes. Theseportfolios help teachers and students be reflective oftheir work, assess their effectiveness as teachers andlearners, and reshape their approach to the coursematerials. Sessions on assessing these student portfolioshave become an integral part of the professional devel-opment institutes for teachers taking place each summerin different areas of the country.

Research is needed to ensure that it is appropriate toinclude portfolio assessment as an important part of theintegrated instruction-based assessment component. Thisstudy is designed to facilitate the practice of developingand assessing portfolios and to ensure that the proceduresbeing taught at teacher institutes will lead to accurate, fairassessment of portfolios. The specific goals are (1) toprepare a working definition of Pacesetter Spanish portfo-lios as assessment tools; (2) to develop a standardizedportfolio assessment system; and (3) to use project resultsas a training tool for future professional developmentsessions for teachers. In summary, the ultimate goal is toenhance classroom practice and assessment.

In Phase I of the Pacesetter Spanish Portfolio ResearchProject, carried out in 1995–96, the following issues wereconsidered: (a) the role of the portfolio as an assessmenttool, its working definition and its appropriateness withinthe course framework; (b) the development of scoringrubrics and an assessment matrix, taking into accountcourse outcomes, performance indicators, and scoringrubrics for the four language proficiency skills (reading,writing, listening, and speaking); (c) the student’s per-spective (cover sheet and guidelines); (d) discussion andrevision of the assessment matrix; (e) reading portfolios;(f) debriefing session; and (g) preparation of trainingpackets for 1996 Professional Development SummerInstitutes for Teachers.

Both the 1996 Professional Development SummerInstitutes for Teachers and midyear meetings includedsessions on portfolios, and the samples chosen for thetraining packets served to disseminate the PacesetterSpanish portfolio model.

The final draft of the corresponding report was pro-duced, and the assessment matrix needed to be refinedaccordingly.

For Phase II of the Pacesetter Spanish PortfolioResearch project the following tasks were planned:

• In spring 1997, in consultation with the teacherswho participated in Phase I of this project, and takinginto account Phase I results, the assessment matrixwas refined.

• In summer 1997, during the three-day refresher sessionat the Delaware Summer Institute for Teachers, a port-folio reading was held. The new matrix was piloted bythe participating teachers. Since it is essential that readers

develop a shared understanding of the scoring criteriaand learn to apply the criteria consistently, benchmarkportfolios from Phase I were selected for training pur-poses. After the initial training, the reading was carriedout following the model delineated in the portfoliosection of the Pacesetter Scoring Handbook (seeAppendix A). In addition to tabulation of scores, otherdata (for example, notes from discussion with teachers,minutes from the debriefing after reading, etc.) werecollected during the reading. The final version of therevised portfolio scoring matrix and guidelines wasrefined for immediate dissemination to participatingschools in order for Pacesetter Spanish teachers to startapplying them in the 1997–98 school year.

• In fall 1997, the compilation of the results of thisreading was analyzed. Benchmark portfolios fromPhase II were identified, and a set of sample portfolioswas developed to be used in the 1997 midyear pro-fessional development training sessions for teachers.

After Phase I, initiated in 1995–96, a new assessmentmatrix was prepared, ready to be further refined. AfterPhase II, this new and revised matrix was applied anddisseminated for further use by Pacesetter Spanish class-room teachers.

Through discussions with participants to determinewhether the portfolios are a valid reflection of studentachievement, and by calculating how reliably the port-folios can be scored, we expect to have a basis forexpanding the use of portfolio assessment as an integralpart of the Pacesetter Spanish program.

Pacesetter Spanish PortfolioResearch Project (PHASE II)MethodAndrea Fercsey and Carmen Luna, of the AssessmentDivision, coordinated this phase of the research withthe help of Pablo Aliaga, a graduate student at the Uni-versity of Michigan, who was the ETS summer internassigned to work on this project. David Baum of theAssessment Division also assisted at the portfolio reading.

Portfolios were evaluated on the campus of theUniversity of Delaware from July 10 through 13, 1997.Six schools sent in work by students enrolled inPacesetter Spanish during the 1996–97 school year.Seven teachers participated in assessing these portfolios;two of them had prior experience scoring PacesetterSpanish portfolios. Not every school that sent in portfo-lios had teachers participating in the assessment of theseportfolios. Three others assisted in the assessment of the

19

portfolios. All three worked for ETS; two with priorexperience scoring Pacesetter Spanish portfolios.

Ninety-five portfolios were designated to be assessed atthe beginning of the grading session. Of these, a total of82 were read. Thirteen portfolios were rendered unread-able due to missing data (i.e., failure to incorporate coversheets). Cover sheets (see Appendix B) are essential to theevaluation of portfolios because they contain a rationaleof the choice of a particular document as evidence of aspecific dimension and aspect. Of the 82 portfolios thatwere read, 51 were read once, and 31 were reread by adifferent rater. Hence, a total of 113 readings were com-pleted. Ratings were provided for the three aspects ofDimension 1, “Demonstrating Knowledge of HispanicCultures” (Showing awareness of the diversity ofHispanic cultures, Identifying contributions of Hispanicfigures, and Making connections with other disciplinesand own culture(s)) and the two aspects of Dimension 2,“Using Spanish to Communicate Effectively” (Derivingmeaning from texts and personal interactions; i.e.,receptive skills such as reading and listening, andExpressing meaning in oral and written form, i.e.,productive skills such as speaking and writing). Ratersused the Pacesetter proficiency categories (“Beginning,”“Developing,” “Promising,” “Accomplished,” and “Ad-vanced”) as well as “Not Enough Evidence to Judge” and“Missing” to rate the portfolios. Appendix C provides thescoring rubrics for each aspect. Appendix D providesrater comments about rubrics and portfolios.

Summary of RatingsTable 1 shows the frequency of ratings in each category.Of the 882 ratings, 31.4 percent were “Promising.” Themajority of ratings (58.2 percent) were either “Devel-

oping” or “Promising.” Last year 66 percent of theratings fell within the “Promising” and “Accomplished”categories and 2.42 percent were “Advanced.” However,this year less than 1 percent of the students received arating of “Advanced.” Finally, 5.4 percent of the ratingsassigned were “No Evidence” due to lack of artifacts.

On average, this year’s students received lowerratings than last year’s students. In 1996, 68 percent ofthe ratings were 3 or higher. In 1997, only 45 percentof the ratings were 3 or higher. This year’s students mayhave been less able compared to last year’s students, theraters could have been more severe in their grading, orthe raters may have been better instructed in scoringthis year because of the previous year’s study.

Also, there was a significant decrease in the use of the“No Evidence” category. This suggests improvementin teacher and student understanding of what is neededto evaluate the portfolios: In a portfolio system like thisone, the selection process is complicated and revealing.Students need to develop a deep understanding aboutthe nature of quality in their work and how the selectedpieces constitute evidence for a particular dimension oraspect. Teachers also need to be fully cognizant of allpossible ways to show evidence of achievement.Consequently, the fact that portfolios in 1997 containedmore complete evidence of students’ work could beinterpreted as a sign of better implementation in theactual classrooms.

Table 2 provides the frequency of ratings for eachaspect of Dimension 1, “Demonstrating Knowledge ofHispanic Cultures.”

Aspects 1 and 3 have similar rating distributions.Further, the proportion of “No Evidence” ratings in thethree aspects of Dimension 1 are more uniform thisyear than last year. Last year “No Evidence” had quite

TABLE 1

Frequency of Ratings in Each Rating Category*

*Data include 13 portfolios that were not evaluated (reported as “Missing”).

20

different response rates. In 1996, over 70 percent of theratings of Aspect 3 were “No Evidence”; by compari-son, only 6 percent of the ratings of Aspect 3 were “NoEvidence” this year.

Table 3 shows the frequency distribution of ratingsfor Dimension 2, “Using Spanish to CommunicateEffectively.”

Aspect 1 has a higher percentage of “No Evidence”and “Beginning” ratings, and a lower percentage of “De-

veloping” ratings than Aspect 2 or the general dimensionratings. The distribution of ratings for “No Evidence” inthe three categories is quite similar to that of last year.

A comparison of Dimension 1 and 2 ratings shows thatsimilar percentages (57 percent to 60 percent) of theratings were “Developing” or “Promising.” For the“Beginning” rating, Dimension 1 has approximately 16percent compared to 8 percent for Dimension 2.For the “Accomplished” rating, Dimension 1 has approx-

TABLE 2

Frequency of Ratings in Each Category for the Aspects of Dimension 1*

*Data include 13 portfolios that were not evaluated (reported as “Missing”).Aspect 1: Showing awareness of the diversity of Hispanic culturesAspect 2: Identifying contributions of Hispanic figuresAspect 3: Making connections with other disciplines and own culture(s)

21

TABLE 3

Frequency of Ratings in Each Category for Aspects of Dimension 2*

imately 10 percent compared to 16 percent for Dimension2. More “Advanced” ratings were given for Dimension 2than for Dimension 1. Finally, more “No Evidence”ratings were given for Dimension 1 than for Dimension 2.

Time Used to Read PortfoliosTable 4 is a summary of the time spent on portfolios byeach rater. The table provides the number of portfolioseach rater read and the total time, average time,standard deviation, minimum, and maximum time ittook the raters to evaluate portfolios.

On average, raters read 11 portfolios, each portfoliotaking about 45 minutes to evaluate. Average time spenton portfolios ranged from 34 to 60 minutes, the rangeof time for each individual portfolio being 17 to 150minutes. Average time spent per portfolio was similar tolast year’s average of 47 minutes.

Distribution of Ratings for Each RaterTable 5 shows the distribution of ratings for each rater.Only two raters gave the “Advanced” proficiencyrating. This may be an indication of disparate use of the

*Data include 13 portfolios that were not evaluated (reported as “Missing”).Aspect 1: Deriving meaning from printed materials and oral discourseAspect 2: Expressing meaning in oral and written form

rubrics by raters. Rater 7 gave more than one-fifth ofthe portfolios he/she read a “No Evidence” rating. Noother rater gave more than one-sixth of the portfoliosthey read a rating of “No Evidence.” Rater 7 never gavea rating higher than “Promising.” Five raters gaveapproximately 50 percent of the portfolios they read a“Promising” rating. Eighty-five percent of the ratingsRater 5 gave were either “Promising” or “Developing.”Consequently, it appears that rater 5 tends to rate moreleniently than the other raters.

Portfolios Read TwiceTable 6 shows that of the 31 portfolios that weredouble read, 8 were given the same two total scores forboth dimensions. After evaluating all aspects of a partic-ular dimension, raters had to give a holistic rating to thewhole dimension. Of the remaining portfolios, approx-imately 39 percent had a score that differed by only onepoint on either Dimension 1 or 2 or both. Only oneportfolio had a difference in score of 3 points for a givendimension. The average absolute difference in time to

score the portfolios was 15 minutes. However, thisnumber is exaggerated by an apparent outlier of 105minutes. The average absolute difference in time afterremoving the apparent outlier is 12 minutes.

Table 7 examines the discrepancies between raterswithin each aspect of both dimensions. Overall, 39 per-cent of ratings were identical. Another 40 percent oftheir ratings differed by a single point. Thirteen percentof the ratings differed by 2 or more points. Finally, in10 percent of the cases, one rater gave a “No Evidence”rating and the other rater believed there was sufficientevidence to score the portfolio on that aspect. This isfewer than last year.

Table 8 shows where specific discrepancies betweenraters occurred. Thirty-two percent of the discrepanciesinvolved a rating of “Developing” and a rating of“Promising.” A year ago the greater number of discrep-ancies involved a rating of “Promising” and a rating of“Accomplished.” Nineteen percent of the discrepanciesinvolved ratings of “Beginning” and “Developing.” Only6 percent of the discrepancies involved a rating of “Begin-ning” and a “No Evidence” rating. Aspect 2 ofDimension 2 registered the fewest cases of discrepancies.

22

TABLE 4

Time Employed by Each Rater in Minutes

23

TABLE 5

Total Count and Relative Frequency By Category and Rater

24

TABLE 6

Comparison of Portfolios and Raters

25

*Absolute Difference = Number indicates the difference between the two ratings given by separate raters to the same portfolio.

**Undefined = Difference in rating could not be calculated because one rater considered that there was not enough evidence to judge a portfolioand the other rater gave the portfolio a rating (usually a rating of 2 or 3).

TABLE 6 (continued)

26

TABLE 8

Specific Discrepancies Between Raters

Dimension 1, Demonstrating Knowledge of Hispanic CulturesAspect 1: Showing awareness of the diversity of Hispanic culturesAspect 2: Identifying contributions of Hispanic figuresAspect 3: Making connections with other disciplines and own culture(s)Dimension 2, Using Spanish to Communicate EffectivelyAspect 1: Deriving meaning from printed materials and oral discourseAspect 2: Expressing meaning in oral and written form

TABLE 7

Discrepancies Between Raters

Dimension 1, Demonstrating Knowledge of Hispanic CulturesAspect 1: Showing awareness of the diversity of Hispanic culturesAspect 2: Identifying contributions of Hispanic figuresAspect 3: Making connections with other disciplines and own culture(s)Dimension 2, Using Spanish to Communicate EffectivelyAspect 1: Deriving meaning from printed materials and oral discourseAspect 2: Expressing meaning in oral and written form

27

Correlation Among the Ratings of Portfolios Each portfolio received seven ratings: one rating for eachof the three aspects of Dimension 1, one for each of thetwo aspects of Dimension 2, and two total ratings. Table 9summarizes the correlations among the various ratings.Correlations among the three aspects in Dimension 1range from .555 to .611, and the correlation between thetwo aspects of Dimension 2 is .615. The correlationbetween the three aspects of Dimension 1 and the totalfor Dimension 1 range from .702 to .830. The correla-tion between the two aspects of Dimension 2 and thetotal for Dimension 2 are .842 and .835, respectively.The correlation between the two dimensions ismoderately high at .653. With the exception of thecorrelations between aspect ratings and total dimensionratings, the correlations are all higher than last year.

Reliability StudyAs was done last year, FACETS (Linacre, 1989) wasused to summarize the data. FACETS constructs linearmeasures from qualitatively ordered counts by means offaceted Rasch analysis. Each observation is the outcomeof an interaction between elements of facets (e.g., a“student’s” portfolio evaluated by a “rater” on several“aspects”). Each observation provides FACETS withinformation about the elements that interact to con-struct it. From this information, FACETS estimates aquantitative measure for each element of each facet(e.g., each student, each rater, each portfolio aspect).

The measures for the elements obtained from oneanalysis are all in the same linear frame of reference onone common interval scale.

For this study, two partial credit models and a ratingscale model were used. The partial credit models wereconstrained with regard to rater and aspects of theportfolio. The partial credit model for raters allows usto compare the raters. The partial credit model foraspects allows us to compare the rating scales forthe aspects. Finally, the rating scale model uses all theinformation together to compare students, raters, andaspects of the portfolio.

The output in FACETS for these models is a set ofrulers that provides a graphical description of thevariable. A “+” or “−” before the facet name indicateswhether the facet measures are positively or negativelyoriented. The vertical axis provides a linear definition ofthe variable. Each element name is positioned accordingto its measure. Elements with extreme scores are notpositioned by measure but are placed at the extreme topor bottom of the column of their facet.

Figure 1 shows the results obtained from using thepartial credit model for raters. The figure reveals that notevery rater is rating students in the same way. It appearsthat Rater 7 (R.7) is using a scale that ranges from 1 to3. Most others are using a scale that ranges from 1 to 4.Finally, Raters 1 and 10 appear to be using the full rangeof ratings offered, 1 to 5. Figure 1 also indicates thatRaters 11 and 12 are grading most leniently, and Rater 4is grading most harshly. Figure 1 indicates that studentstend to receive higher ratings on aspects from Dimension1, “Demonstrating Knowledge of Hispanic Cultures,”than those from Dimension 2, “Using Spanish to

TABLE 9

Bivariate Correlation MatrixDimension 1 Dimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2 Dimension 2Aspect 1 Aspect 2 Aspect 3 Total Aspect 1 Aspect 2 Total

Dimension 1 Aspect 1 1.000 .568 .555 .702 .398 .264 .429

Dimension 1 Aspect 2 1.000 .611 .830 .606 .511 .576

Dimension 1 Aspect 3 1.000 .795 .573 .359 .525

Total Dimension 1 1.000 .642 .543 .653

Dimension 2 Aspect 1 1.000 .615 .842

Dimension 2 Aspect 2 1.000 .835

Total Dimension 2 1.000

Dimension 1, Demonstrating Knowledge of Hispanic Cultures

Aspect 1: Showing awareness of the diversity of Hispanic cultures

Aspect 2: Identifying contributions of Hispanic figures

Aspect 3: Making connections with other disciplines and own culture(s)

Dimension 2, Using Spanish to Communicate Effectively

Aspect 1: Deriving meaning from printed materials and oral discourse

Aspect 2: Expressing meaning in oral and written form

Figure 1. FACETS: Partial credit model for raters.

28

not use the scoring rubrics in the same manner; sometend to rate more leniently than others.

The second model used was the partial credit modelfor aspects. The results obtained from using that modelare summarized in Figure 2. Figure 2 indicates that therating scale for one aspect functions somewhat differ-ently from another. The aspects generally fall into tworating scale groups. Group 1 consists of Aspects 1, 2, and3 for Dimension 1, and total for Dimension 1 as well asAspect 1 for Dimension 2. These aspects appear to berated on a 1 to 4 scale. All these aspects refer either toreceptive language skills (Deriving meaning from printed

Communicate Effectively.” This may be explained bythe fact that teachers have more experience ratinglinguistic skills than the concepts included in the aspectsfor “Demonstrating Knowledge of Hispanic Cultures.”

Table 10 shows each rater’s observed average (i.e.,average of the raw score rating given) and his or her“fair” average (the observed average adjusted for thedeviation of the portfolio in each rater’s sample fromthe overall mean of the portfolios across all raters). Therater separator index of 5.24 indicates that the tenraters can be separated into five statistically distinctseverity strata. These results suggest that the raters do

Dimension 1, Demonstrating Knowledge of Hispanic Cultures

Aspect 1: Showing awareness of the diversity of Hispanic cultures

Aspect 2: Identifying contributions of Hispanic figures

Aspect 3: Making connections with other disciplines and own culture(s)

Dimension 2, Using Spanish to Communicate Effectively

Aspect 1: Deriving meaning from printed materials and oral discourse

Aspect 2: Expressing meaning in oral and written form

29

TABLE 10

Rater Severity

44, 48, 52, 57, 70, 73, and 79) would have had their totalscores changed if we had adjusted their scores for raterseverity. This rate of 20 is less than the 29 rate of changelast year. It should be noted that the 14 students withasterisks in the observed score column have unusualpatterns of ratings (i.e., more than typical variations intheir ratings). In each case, there are one or more unex-pected ratings—aberrant ratings that don’t seem to “fit”with the others. A mean-square infit or outfit valuegreater than 1.5 indicates a student’s portfolio mightneed another review before a score report is issued,particularly if the student’s total score is near a criticaldecision-making point in the score distribution.

AnalysesWe ran two FACETS analyses: one on the ratings of the21 portfolios read twice in the June 1996 portfolioreading, and one on the ratings of the 31 portfoliosread twice in the June 1997 portfolio reading. We thenexamined selected pieces of output from the two anal-yses to see how they compared.

StudentsEach of the portfolios received 14 ratings (i.e., sevenratings each from two raters). For each portfolio,FACETS averaged the ratings to compute a total score.We then compared the range of total scores acrossyears. The range of portfolio scores was somewhatwider in 1997 than in 1996. The total scores for the 21portfolios rated in 1996 ranged from 2.4 to 4.3. Bycomparison, the total scores for the 31 portfolios readin 1997 ranged from 1.5 to 4.0.

materials and oral discourse) or a conceptual introspec-tion of different elements of the Spanish-speaking worldand cultures (“Demonstrating Knowledge of HispanicCultures”). In both cases, it is harder to apply judgementor show evidence for the type of documents or behaviorthat demonstrate evidence of achievement of these.Group 2 consists of Aspect 2 for Dimension 2 (Expres-sing meaning in oral and written form) and total forDimension 2, “Using Spanish to Communicate Effec-tively.” These aspects appear to be rated on a 1 to 5 scale,although a rating of 5 is difficult to achieve. These aspectsinclude productive skills, which are usually measuredand evaluated in language classrooms. Students andteachers have more experience with the kind of evidencethat is necessary to show achievement in these aspects.

The final model is the rating scale model. This modeldoes not constrain or condition any facet. This modelwould be valid if the partial-credit models showed thatthe raters used the aspect rating scales in a similarfashion, and that the aspect scales functioned similarly.In our case they don’t; consequently, the rating scalemodel is used with caution. The results obtained usingthis model are shown in Figure 3. Figure 3 indicates thatstudents tend to get higher ratings on Dimension 1 thanDimension 2. It appears that there were three distinctstrata of rater severity. Raters 4, 7, 8, and 9, who werethe most harsh, form the first stratum. Raters 2, 5, 10,and 1 form a second stratum. Raters 11 and 12, themost lenient, form a third stratum. The distribution ofstudents appears skewed, with some outliers at the top.

Figure 3A shows that differences between the students’observed averages and fair averages indicate that 20 stu-dents (12, 17, 18, 20, 23, 24, 25, 26, 30, 32, 36, 37, 40,

30

On average, students did better in 1996 than in1997. The average total score for the 21 portfolios ratedin 1996 was 3.3 (SD = 0.4); the average total score forthe 31 portfolios rated in 1997 was 2.5 (SD = 0.6).

When FACETS calculates “fair averages” for students,it adjusts each student’s total score for the severity orleniency of the two raters who rated that particular

student. On average, the amount of difference betweenthe “observed average” (i.e., the unadjusted raw scoreaverage of the raters’ ratings of a given student) and the“fair average” (i.e., the observed average adjusted forrater severity or leniency differences) was 0.2 in both1996 and 1997. The largest severity or leniency adjust-ment for any portfolio was 0.7 in 1997 and 0.5 in 1997.

Figure 2. FACETS: Partial credit model for aspects.

Dimension 1, Demonstrating Knowledge of Hispanic Cultures

Aspect 1: Showing awareness of the diversity of Hispanic cultures

Aspect 2: Identifying contributions of Hispanic figures

Aspect 3: Making connections with other disciplines and own culture(s)

Dimension 2, Using Spanish to Communicate Effectively

Aspect 1: Deriving meaning from printed materials and oral discourse

Aspect 2: Expressing meaning in oral and written form

31

Figure 3. FACETS: Rating scale model.

Dimension 1, Demonstrating Knowledge of Hispanic Cultures

Aspect 1: Showing awareness of the diversity of Hispanic cultures

Aspect 2: Identifying contributions of Hispanic figures

Aspect 3: Making connections with other disciplines and own culture(s)

Dimension 2, Using Spanish to Communicate Effectively

Aspect 1: Deriving meaning from printed materials and oral discourse

Aspect 2: Expressing meaning in oral and written form

32

Figure 3A. FACETS: Student adjusted scores.

33

RatersIn 1996, in Phase I of the portfolio study, each of theseven raters rated five portfolios, but not all raters ratedthe same set of five portfolios. For each rater, FACETScalculated the average of the ratings that rater gave thefive portfolios she or he rated. The average of theratings each rater gave ranged from 3.0 for Rater 5 to3.6 for Raters 3 and 4.

In 1997, in Phase II of the portfolio study, each of the10 raters rated from two to nine portfolios. The average

of the ratings each rater gave ranged from 1.8 for Rater4 to 3.1 for Rater 11. On average, raters tended to givelower ratings to the portfolios scored in 1997 than tothose scored in 1996.

Aspects and DimensionsWhen raters used the scoring rubrics for the variousaspects and dimensions, there were differences in 1996and 1997 in the percentages of ratings falling in eachscale category. Those differences are shown in Table 11.

Figure 3A (continued).

TABLE 11

Percentage in Each Scale Category

34

Dimension 1, “Demonstrating Knowledge of HispanicCultures”

Aspect 1, Showing awareness of the diversity ofHispanic cultures

• In 1996, raters used all five scale categories; in 1997,raters used only four scale categories (i.e., 1 to 4).

• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.

Aspect 2, Identifying contributions of Hispanic figures

• In 1996 and 1997, raters used four of the five scalecategories (i.e., 1 to 4).

• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.

Aspect 3, Making connections with other disciplinesand own culture(s)

• In 1996, raters used only three scale categories (i.e.,3 to 5); in 1997, raters used four scale categories (i.e.,1 to 4).

• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.

Dimension 1

• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used four scale categories (i.e., 1 to 4).

• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.

Dimension 2, “Using Spanish to CommunicateEffectively”

Aspect 1, Deriving meaning from texts and personalinteractions

• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used four scale categories (i.e., 1 to 4).

• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.

Aspect 2, Expressing meaning in oral and written form

• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used all five scale categories.

• In 1996, the majority of ratings were 3; in 1997, themajority of ratings were 2 or 3.

Dimension 2

• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used all five scale categories.

• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.

Summary of Findings• This research showed that the aspects of the de-

lineated Pacesetter Spanish system of portfolioassessment worked better in Phase II than in Phase I.Students and teachers had a better understanding ofwhat constitutes evidence of achievement as well asof the selection process and organization of artifactsincluded in the portfolios.

• Pacesetter Spanish portfolio raters used the categoriesof “Promising” and “Developing” for the majority ofratings, but less than in 1996, and not the extremecategories (“Accomplished” or “Advanced”).

• The length of time to rate the portfolios in 1996 and1997 was an average of 45 to 50 minutes. Althoughvariety and complexity are encouraged throughout,this element of time needs to be taken into accountwhen preparing to read portfolios.

• Although higher than in 1996, the number of ratingagreements was moderate. However, there were aconsiderable number of discrepancies between ratersregarding the presence or lack of enough artifacts tojudge students’ performance within the parameters ofthe Pacesetter Spanish portfolio assessment system.

• The proportion of ratings of “No Evidence” forDimension 1, Aspect 3 was dramatically reduced.

• Raters differ in level of severity exercised when rat-ing portfolios. Some grade significantly more harshlythan others. About one-fourth of the students wouldhave received different total scores if their scores hadbeen adjusted for differences in rater severity.

• The rating scales for the individual aspects functionsomewhat differently from one another. Raters usedall five-scale categories for some of the rating scalesbut only used four-scale ratings for other scales.

• The rating patterns for about 20 percent of theportfolios contained one or more surprising or un-expected ratings (i.e., ratings that don’t seem to “fit”with the other ratings given to that portfolio). Whensuch variability is present in a set of ratings, it issuggested that those in charge of monitoring qualitycontrol in the scoring session give those portfolios asecond look to determine which ratings are “out ofsync” with the other ratings, and whether thoseunexpected ratings shall stand “as is” or be changed.

Recommendations• A final version of the matrix should be incorporated

as part of the formal components of the PacesetterSpanish program.

35

• A set of guidelines should be included in theTeacher’s and Student’s Edition of the PacesetterSpanish curriculum materials. The guidelines shouldinclude clear instructions of the selection process ofwork samples for the portfolio, cover sheets, rubrics,expectations, and any other pertinent information.

• Activities in the Pacesetter Spanish curriculummaterials should be designed to reflect all aspects anddimensions of the assessment portfolios so studentshave ample opportunity to demonstrate their achieve-ment of the course outcomes.

• Dissemination of the portfolio component of thePacesetter Spanish program should be carried outat the different professional development sessions.Training sessions should include an overview of thesteps to follow when implementing the system ofportfolio assessment as well as an explanation onhow to apply the matrix when evaluating portfolios.

• New benchmark portfolios from the 1997 PacesetterSpanish reading should be identified, and a set ofsample portfolios should be prepared for use intraining sessions.

• Efforts should be made to organize scoring sessionsat the national or regional level for Pacesetter Spanishportfolios so teachers can have an opportunity toapply the assessment matrix and have discussions onwhat constitutes evidence of achievement for eachaspect and dimension.

• Further research is needed to ensure that the evalua-tion of portfolios in the Pacesetter Spanish program isdone effectively and that all quality control measuresare taken.

• Additional research is necessary to evaluate the dif-ferent elements that comprise the Pacesetter Spanishportfolio assessment system. Results could guidefurther refinement of this component and clarify itsconnection to the goals of the overall program.

• Evaluative studies should be conducted to measurethe effectiveness and impact of the Pacesetter Spanishprogram in general. It is particularly important tofind out how each component contributes to thesuccess of the program and its vision of integratingstandards, professional development, instruction, andassessment.

ReferencesFercsey, A., Luna, C. and Ponte, E. (1997). Pacesetter Spanish

Portfolio Research Project: Phase I Report. [Unpublishedmonograph]. Princeton, NJ: Educational Testing Service.

Linacre, J.M. and Wright, B.D. (1994). A User’s Guide toFACETS: Rasch Measurement Computer Program.[Computer program manual]. Chicago: MESA Press.

Sheingold, K., Storms, B., Thomas, W. and Heller, J. (1997).Pacesetter English Portfolio Assessment: 1995 Report.Center for Performance Assessment. Princeton, NJ:Educational Testing Service.

Phase I Appendix

39

Appendix 1: DRAFT COVER SHEET

PACESETTER SPANISH PORTFOLIO ASSESSMENT

NAME:

Selection: Date work was done:

Why did you select this piece of work?

What does it show or tell about you?

What did you learn from doing this assignment?

Please check the dimension(s) or aspects for which you are showingevidence with this assignment:

• Dimension 1: Demonstrating Knowledge of Hispanic Cultures

Showing Awareness of the Diversity of Hispanic Cultures

Identifying Contributions of Hispanic Figures

• Dimension 2: Using Spanish to Communicate Effectively

Deriving Meaning from Texts and Personal Interactions

Expressing Meaning in Oral and Written Form

40

Appendix 1: DRAFT COVER SHEET (Continuation)

PACESETTTER SPANISH PORTFOLIO ASSESSMENT

NOMBRE:

Título: Fecha de ejecución:

¿Por qué seleccionaste esta obra?

¿Qué muestra o dice esta obra sobre ti?

¿Qué aprendiste de esta tarea?

He seleccionado esta obra para indicar evidencia en las siguientesdimensiones:

• Dimensión 1: Demostrar conocimiento de las culturas hispanas

Demostrar conciencia de la diversidad de las culturas hispanas

Identificar contribuciones de figuras hispanas

• Dimensión 2: Usar el español para comunicaciones eficientes

Comprensión de textos e interacciones personales

Expresión oral y escrita

41

Ap

pen

dix

2: A

ssessm

en

t M

atrix

PA

CE

SE

TT

ER

S

PA

NIS

H

DIM

EN

SIO

N: D

EM

ON

ST

RA

TIN

G K

NO

WL

ED

GE

O

F H

IS

PA

NIC

C

UL

TU

RE

S

Po

rtfo

lio

A

ssessm

en

t: E

valu

atin

g A

sp

ects o

f th

e D

im

en

sio

n

Beg

in

nin

g

�

Dev

elo

pin

g

�

Pro

misin

g

�

Acco

mp

lish

ed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

id

en

ce to

ju

dg

e�

Sh

ow

in

g

Aw

aren

ess o

f

th

e D

iv

ersity

of H

isp

an

ic

Cu

ltu

res

Vag

ue in

dicatio

n

frag

men

tary

, su

perficial

Lim

ited

-in

co

mp

lete

-a

few

sp

ecific

ex

am

ples

-u

ne

ve

n

-p

artia

l u

nd

ev

elo

ped

So

me ex

am

ples

-g

en

eral in

sig

hts

-m

id

dle ran

ge o

f

in

sig

hts/

ex

am

ples

Am

ple/

fu

ll, w

id

e

ran

ge o

f ex

am

ples,

deep

er/

mo

re

pro

fo

un

d

Co

nsisten

tly

in

sig

htfu

l,

va

ried

, ex

ten

ded

Beg

in

nin

g

�

Dev

elo

pin

g

�

Pro

misin

g

�

Acco

mp

lish

ed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

id

en

ce to

ju

dg

e�

Id

en

tify

in

g

Co

ntrib

utio

ns

of H

isp

an

ic

Fig

ures

List a

few

n

am

es a

nd

co

ntrib

utio

ns

brief, sk

etch

y

descrip

tio

n

So

me d

escrip

tio

n o

f

co

ntrib

utio

ns

-lim

ited

n

otio

ns o

f

co

ntrib

utio

n

So

me in

feren

ces ab

ou

t

pla

ce a

nd

im

pa

ct in

histo

ry

Reco

gn

itio

n o

f im

pact

in

h

isto

ry

-a

mp

le in

feren

ces

ab

ou

t p

lace in

h

isto

ry

Th

oro

ug

h in

feren

ces

-co

nsisten

tly

in

sig

htfu

l

an

aly

sis o

f co

ntrib

utio

ns

an

d im

pact/

ro

le in

histo

ry

Beg

in

nin

g

�

Dev

elo

pin

g

�

Pro

misin

g

�

Acco

mp

lish

ed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

id

en

ce to

ju

dg

e�

Mak

in

g

co

nn

ectio

ns

with

o

th

er

discip

lin

es

an

d o

wn

cu

ltu

re(s)

Iso

lated

-n

o lo

gical co

nn

ectio

ns

-few

, irrelev

an

t,

clu

eless, su

perficial

co

nn

ectio

ns

Lim

ited

-p

artial

-n

ot alw

ay

s relev

an

t

-u

nev

en

co

nn

ectio

ns

So

me g

en

eral n

otio

ns

with

ou

t m

uch

dep

th

/in

sig

ht

Am

ple ex

am

ples o

f

in

sig

htfu

l an

d w

id

e

ran

ge o

f co

nn

ectio

ns

-so

me in

feren

ces

Th

oro

ug

h an

d in

sig

htfu

l

co

nn

ectio

ns

-n

ew

-b

ey

on

d w

ha

t th

ey

'v

e

seen

in

co

urse

-creativ

ity

-o

rig

in

ality

-o

rig

in

al co

nn

ectio

ns

42

Ap

pen

dix

2

: A

ssessm

en

t M

atrix

(C

on

tin

ua

tio

n)

PA

CE

SE

TT

ER

S

PA

NIS

H

DIM

EN

SIO

N: U

SIN

G S

PA

NIS

H T

O C

OM

MU

NIC

AT

E E

FF

EC

TIV

EL

Y

Po

rtfo

lio

A

ssessm

en

t: E

valu

atin

g A

sp

ects o

f th

e D

im

en

sio

n

Beg

in

nin

g

�

Dev

elo

pin

g

�

Pro

misin

g

�

Acco

mp

lish

ed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

id

en

ce to

ju

dg

e�

Deriv

in

g

mean

in

g

fro

m tex

ts

an

d p

erso

nal

in

teractio

ns

(Sk

ills:

Listen

in

g

an

d

Read

in

g)

Un

dersta

nd

s few

gen

era

l id

ea

s

-co

ntin

uo

us

misu

nd

ersta

nd

in

gs

Un

ev

en

co

mp

reh

en

sio

n,

freq

uen

t/

nu

mero

us

misu

nd

erstan

din

gs

-u

nd

ersta

nd

s so

me

main

id

eas o

n fam

iliar

to

pics

Un

dersta

nd

s m

ain

id

ea

s

an

d so

me d

eta

ils o

n

so

mew

ha

t fa

milia

r to

pics

-m

ay

m

ak

e so

me

in

feren

ces d

ep

en

din

g o

n

co

mp

lex

ity

o

f tex

t

Un

dersta

nd

s m

ain

id

ea

s

an

d m

ost d

eta

ils ev

en

on

u

nfa

milia

r to

pics

-ca

n m

ak

e in

feren

ces

ap

pro

pria

te to

th

e ta

sk

Th

oro

ug

h

un

dersta

nd

in

g,

ex

tra

po

la

tes

in

fo

rm

atio

n in

n

ew

way

s, in

clu

din

g so

me

so

cia

l a

nd

cu

ltu

ra

l

referen

ces a

nd

affectiv

e o

verto

nes

Beg

in

nin

g

�

Dev

elo

pin

g

�

Pro

misin

g

�

Acco

mp

lish

ed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

id

en

ce to

ju

dg

e�

Ex

pressin

g

mean

in

g in

oral an

d

written

fo

rm

(Sk

ills:

S

peak

in

g

an

d W

ritin

g)

Co

mm

un

ica

tio

n

im

ped

ed

b

y

in

terferen

ce fro

m

an

oth

er la

ng

ua

ge

-lack

o

f flu

en

cy

-in

ad

eq

ua

te v

oca

bu

la

ry

an

d/

or p

ro

nu

ncia

tio

n

-fra

gm

en

ted

-rep

etitio

us m

essa

ge

fo

rces in

terp

reta

tio

n,

-p

ro

nu

ncia

tio

n

ham

pers

co

mm

un

ica

tio

n

Lim

ited

flu

en

cy

-co

nv

ey

s sim

ple

messag

es

-u

nco

nn

ected

d

isco

urse

or sen

ten

ces

-lim

ited

ra

ng

e o

f

vo

ca

bu

la

ry

-so

me in

terferen

ce fro

m

an

oth

er la

ng

ua

ge

-p

ro

nu

ncia

tio

n m

ay

in

terfere w

ith

co

mm

un

ica

tio

n

So

me u

nd

erly

in

g

org

an

iz

atio

n o

f th

ou

gh

ts/

id

eas/

disco

urse

-em

erg

in

g n

otio

n o

f

ad

eq

ua

te reg

ister

-a

deq

ua

te ra

ng

e o

f

vo

ca

bu

la

ry

-self-co

rrectio

ns

-p

ro

nu

ncia

tio

n m

ay

b

e

aw

kw

ard

b

ut d

oesn

't

in

terfere w

ith

co

mm

un

ica

tio

n

-lev

el o

f flu

en

cy

d

oesn

't

in

terfere w

ith

co

mm

un

ica

tio

n

Messa

ge is a

rticu

la

ted

(o

rg

an

iz

ed

ap

pro

pria

tely

) -p

ro

per

use o

f circu

mlo

cu

tio

n

-a

t tim

es crea

tiv

e id

ea

s,

-co

heren

t u

se o

f

vo

ca

bu

la

ry

-m

in

im

al in

terferen

ce

fro

m a

no

th

er la

ng

ua

ge

-m

ostly

a

deq

ua

te u

se o

f

reg

ister

Hig

hly

crea

tiv

e,

-th

oro

ug

h, m

ea

nin

gfu

l,

-ev

id

en

tly

o

rg

an

iz

ed

,

-w

ell a

rticu

la

ted

a

nd

org

an

ized

m

essag

e,

-ap

pro

pria

te u

se o

f

reg

ister

-w

id

e ra

ng

e o

f

vo

ca

bu

la

ry

-little o

r n

o in

terferen

ce

fro

m a

no

th

er la

gu

ag

e,

-h

ig

h lev

el o

f flu

en

cy

43

Appen

dix

3S

tuden

t N

um

ber

:R

ater

Nu

mb

er:

A

rtif

acts

Use

d

Dim

ensi

on

1:

Dem

on

stra

tin

g K

no

wle

dg

e o

f H

isp

anic

Cu

ltu

res

Dim

ensi

on

2:

Usi

ng

Sp

anis

h t

o C

om

mu

nic

ate

N

ote

s/

C

om

men

ts

Div

ersi

ty o

f H

isp

anic

Cu

ltu

res

Co

ntr

ibu

tio

ns

of

His

pan

ic F

igu

res

Co

nn

ecti

on

s w

ith

oth

er d

isci

pli

nes

Der

ivin

g m

ean

ing

fro

m t

exts

(L

&R

)E

xpre

ss. m

eanin

gfr

om

tex

ts (

S&

W)

S

core

Scale

: B

eg

inn

ing

, D

ev

elo

pin

g,

Pro

mis

ing

, A

cco

mp

lish

ed

, A

dv

an

ced

, N

ot

en

ou

gh

ev

iden

ce t

o j

ud

ge.

Oth

er c

om

men

ts/s

ugges

tions:

44

Appen

dix

4

Po

rt.

# R

ater

Dat

eT

ime

Use

dD

imen

sio

n 1

Asp

ect

1D

imen

sio

n 1

Asp

ect

2D

imen

sio

n 1

Asp

ect

3D

imen

sio

n 2

Asp

ect

1D

imen

sion 2

Asp

ect

2D

1D

2

11

6S

t/E

v

2

0m

Pro

mis

ing

Pro

mis

ing

No e

vid

ence

No e

vid

ence

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

15

Su

/M

3

5m

Acc

om

pli

shed

Pro

mis

ing

NA

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

25

St/

Ev

20

mD

evel

op

ing

Dev

elo

pin

gN

AP

rom

isin

gD

evel

opin

gD

evel

opin

gD

evel

opin

g

24

St/

Ev

30

mP

rom

isin

gN

o e

vid

ence

No e

vid

ence

No e

vid

ence

Pro

mis

ing

No e

vid

ence

Pro

mis

ing

34

St/

Ev

20

mN

o e

vid

ence

Pro

mis

ing

No e

vid

ence

Dev

elo

pin

gP

rom

isin

gN

o e

vid

ence

Pro

mis

ing

33

Su

/M

5

5m

Dev

elo

pin

gP

rom

isin

gA

cco

mp

lish

edA

cco

mp

lish

edA

ccom

pli

shed

Pro

mis

ing

Acc

om

pli

shed

47

St/

Ev

1h

5m

Pro

mis

ing

Beg

inn

ing

Pro

mis

ing

Acc

om

pli

shed

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

42

Su

/M

4

0m

Pro

mis

ing

Pro

mis

ing

No e

vid

ence

Pro

mis

ing

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

51

St/

Ev

1h

Pro

mis

ing

Pro

mis

ing

NA

Acc

om

pli

shed

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

56

St/

Ev

35

mP

rom

isin

gN

o e

vid

ence

No e

vid

ence

Pro

mis

ing

Pro

mis

ing

No e

vid

ence

Pro

mis

ing

61

St/

Ev

30

mP

rom

isin

gA

cco

mp

lish

edN

AA

cco

mp

lish

edA

ccom

pli

shed

Pro

mis

ing

Acc

om

pli

shed

66

St/

Ev

30

mP

rom

isin

gP

rom

isin

gN

o e

vid

ence

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

72

St/

Ev

Acc

om

pli

shed

Acc

om

pli

shed

No e

vid

ence

Acc

om

pli

shed

Acc

om

pli

shed

Acc

om

pli

shed

Acc

om

pli

shed

77

Su

/M

5

0m

Dev

elo

pin

gA

cco

mp

lish

edN

o e

vid

ence

Acc

om

pli

shed

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

84

St/

Ev

No e

vid

ence

No e

vid

ence

No e

vid

ence

Pro

mis

ing

Acc

om

pli

shed

No e

vid

ence

Acc

om

pli

shed

83

Su

/M

3

5m

Pro

mis

ing

Pro

mis

ing

Acc

om

pli

shed

Acc

om

pli

shed

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

1 P

ort

foli

o n

um

ber

s in

bo

ld i

nd

icat

e th

e p

ort

foli

o w

as s

elec

ted

as

a b

ench

mar

k.

45

Ap

pen

dix

4 -

Co

nti

nu

atio

n

Po

rt.

# R

ater

Dat

eT

ime

Use

dD

imen

sio

n 1

Asp

ect

1D

imen

sio

n 1

Asp

ect

2D

imen

sio

n 1

Asp

ect

3D

imen

sio

n 2

Asp

ect

1D

imen

sion 2

Asp

ect

2D

1D

2

96

St/

Ev

20

mP

rom

isin

gA

cco

mp

lish

edP

rom

isin

gN

o e

vid

ence

Acc

om

pli

shed

Pro

mis

ing

No e

vid

ence

95

St/

Ev

25

mA

cco

mp

lish

edA

cco

mp

lish

edN

AA

cco

mp

lish

edA

ccom

pli

shed

Acc

om

pli

shed

Acc

om

pli

shed

10

3S

t/E

v

5

0m

Beg

inn

ing

Acc

om

pli

shed

NA

Acc

om

pli

shed

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

10

1S

u/M

20

mD

evel

op

ing

Pro

mis

ing

NA

Pro

mis

ing

Pro

mis

ing

Dev

elopin

gP

rom

isin

g

11

3S

t/E

vA

cco

mp

lish

edA

cco

mp

lish

edN

AA

cco

mp

lish

edA

cco

mpli

shed

Acc

om

pli

shed

Acc

om

pli

shed

11

1S

u/M

40

mA

cco

mp

lish

edP

rom

isin

gP

rom

isin

gA

cco

mp

lish

edA

cco

mpli

shed

Pro

mis

ing

Acc

om

pli

shed

12

6S

t/E

v

4

0m

Pro

mis

ing

No e

vid

ence

No e

vid

ence

Acc

om

pli

shed

Pro

mis

ing

No e

vid

ence

Pro

mis

ing

12

5S

u/M

45

mA

cco

mp

lish

edP

rom

isin

gN

AP

rom

isin

gP

rom

isin

gA

ccom

pli

shed

Pro

mis

ing

13

5S

t/E

v

4

5m

Pro

mis

ing

Dev

elo

pin

gN

o e

vid

ence

Dev

elo

pin

gP

rom

isin

gD

evel

opin

gP

rom

isin

g

13

4S

t/E

v

3

5m

No e

vid

ence

No e

vid

ence

No e

vid

ence

Pro

mis

ing

Pro

mis

ing

No e

vid

ence

Pro

mis

ing

14

6--

--

No e

vid

ence

No e

vid

ence

No e

vid

ence

No e

vid

ence

Pro

mis

ing

No e

vid

ence

No e

vid

ence

14

4S

t/E

v

4

0m

Pro

mis

ing

No e

vid

ence

No e

vid

ence

Pro

mis

ing

Pro

mis

ing

No e

vid

ence

Pro

mis

ing

15

1S

t/E

v

3

0m

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

Acc

om

pli

shed

Pro

mis

ing

Pro

mis

ing

15

6S

u/M

40

mP

rom

isin

gN

o e

vid

ence

Pro

mis

ing

Acc

om

pli

shed

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

16

2S

t/E

v 1

hA

cco

mp

lish

edP

rom

isin

gP

rom

isin

gA

cco

mp

lish

edA

ccom

pli

shed

Pro

mis

ing

Acc

om

pli

shed

16

7N

A 1

hA

cco

mp

lish

edP

rom

isin

gP

rom

isin

gA

cco

mp

lish

edA

ccom

pli

shed

Acc

om

pli

shed

Acc

om

pli

shed

17

7S

t/E

v 1

h 1

0m

Pro

mis

ing

Pro

mis

ing

No e

vid

ence

No e

vid

ence

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

17

4S

u/M

55

mA

dvan

ced

Acc

om

pli

shed

Advan

ced

Advan

ced

Advan

ced

Advan

ced

Advan

ced

46

Ap

pen

dix

4-C

on

tin

uat

ion

Po

rt.

# R

ater

Dat

eT

ime

Use

dD

imen

sio

n 1

Asp

ect

1D

imen

sio

n 1

Asp

ect

2D

imen

sio

n 1

Asp

ect

3D

imen

sio

n 2

Asp

ect

1D

imen

sion 2

Asp

ect

2D

1D

2

18

5S

t/E

v

4

0 m

Beg

innin

gP

rom

isin

gN

AA

cco

mp

lish

edP

rom

isin

gP

rom

isin

gP

rom

isin

g

18

4S

u/M

30

mN

o e

vid

ence

Pro

mis

ing

No e

vid

ence

No e

vid

ence

Acc

om

pli

shed

No e

vid

ence

No e

vid

ence

19

2S

t/E

v

4

5m

Dev

elop

ing

Dev

elo

pin

gN

o e

vid

ence

Pro

mis

ing

Dev

elopin

gD

evel

opin

gD

evel

opin

g

19

7N

A 1

hP

rom

isin

gP

rom

isin

gN

o e

vid

ence

Acc

om

pli

shed

Pro

mis

ing

Pro

mis

ing

Pro

mis

ing

20

3S

t/E

v 1

hP

rom

isin

gA

cco

mp

lish

edA

cco

mp

lish

edA

cco

mp

lish

edA

ccom

pli

shed

Acc

om

pli

shed

Acc

om

pli

shed

20

1S

u/M

35

mP

rom

isin

gP

rom

isin

gN

AP

rom

isin

gA

cco

mpli

shed

Pro

mis

ing

Pro

mis

ing

21

7S

u/M

50

mP

rom

isin

gA

cco

mp

lish

edN

o e

vid

ence

Acc

om

pli

shed

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

21

2S

u/M

Pro

mis

ing

Pro

mis

ing

NA

Acc

om

pli

shed

Acc

om

pli

shed

Pro

mis

ing

Acc

om

pli

shed

47

Appen

dix

5

PO

RT

FO

LIO

S R

AT

ED

BY

TW

O D

IF

FE

RE

NT

PE

RS

ON

S∗

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

31938

OK

-Perf

ect

Bio

gra

fía-

tape

(D1&

2)∗

∗S

pee

ch (

D2)

Work

Shee

t (D

1&

2)

Guía

Mex

ico (

D1&

2)

Work

Shee

t (D

1&

2)

Dim

ensi

on 2

: P

rom

isin

g;

how

ever

, not

enough e

vid

ence

to j

udge

rece

pti

ve

skil

ls.

Pro

ject

(D

2)

Vid

eo (

D2)

Tap

e (D

1&

2)

Itin

erar

io (

D1&

2)

Rea

din

g (

D1&

2--

1N

E)

[?]

on f

ood (

D1&

2)

Dim

ensi

on 1

: P

rom

isin

g (

had

ther

e bee

n m

ore

evid

ence

it

mig

ht

hav

e bee

nA

ccom

pli

shed

).

Dim

ensi

on 2

: P

rom

isin

g (

not

enough e

vid

ence

!)

∗ T

his

is

a t

ran

scri

pt

of

the o

rig

inal

rati

ng

sh

eets

. In

try

ing

to

be f

ait

hfu

l to

th

e o

rig

inals

, w

e h

av

e w

rite

n b

etw

een

bra

ck

ets

(i.

e.,

[])

tho

se w

ord

s th

at

were

dif

ficu

lt t

o r

ead

fro

m t

he h

an

dw

riti

ng

.

∗∗ D

1:

Inst

rum

en

t w

as

mark

ed

un

der

on

e o

r m

ore

asp

ect

of

dim

en

sio

n 1

.

D2

: In

stru

men

t w

as

mark

ed

un

der

on

e o

r m

ore

asp

ect

of

dim

en

sio

n 2

.

D1

&2

: In

stru

men

t w

as

mark

ed

un

der

on

e o

r m

ore

asp

ect

of

each

dim

en

sio

n.

D

1--

NE

or

D1

--N

E:

Inst

rum

en

t w

as

mark

ed

un

der

the r

esp

ecti

ve d

imen

sio

n b

ut

there

was

no

t en

ou

gh

ev

iden

ce t

o j

ud

ge i

ts c

on

trib

uti

on

to

th

e r

esp

ecti

ve d

imen

sio

n.

D

1&

D2

--1

NE

or

D1

&D

2--

2N

E:

Inst

rum

en

t w

as

mark

ed

un

der

on

e o

r m

ore

asp

ect

of

each

dim

en

sio

n,

bu

t n

ot

en

ou

gh

ev

iden

ce w

as

fou

nd

fo

r o

ne o

f th

e

dim

en

sio

ns.

48

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30129

Dim

ensi

on 1

:O

K/N

ED

imen

sion 2

: O

K

Char

t (D

2)

Wo

rksh

eet

(D2

)E

ssay

(D

2)

Par

agra

ph (

D1&

2)

Pam

phle

t (D

1&

2)

Post

er (

D2)

Ess

ay (

D1&

D2)

Ess

ay (

D2)

Foto

(D

1--

NE

)T

ape

(D1&

2--

2N

E,

read

tex

t)T

ape

(D1&

2--

2N

E)

Invit

ació

n (

D2)

Foto

s L

abel

ed (

D2

--N

E)

Dim

ensi

on 1

: D

iver

sity

: pro

mis

ing/

No e

vid

ence

Dim

ensi

on 2

: P

rom

isin

g

Rea

din

g:

Popo (

D2)

Pre

senta

tion:

Invit

atio

ns

(D2)

May

as P

roje

ct (

D1

&2

)P

age:

Dia

rio (

D2)

Coll

age

(D2)

Itin

erar

io:

Méx

ico

(D

1)

Coll

age:

Cer

van

tes(

D1

)In

vit

acio

nes

/expli

c. (

D2

)D

ebat

e E

coló

gic

o (

D1

&2

)P

royec

to:

Exti

nci

ón

(D

2)

Coll

age:

La

moda

(D2

--N

E)

Dim

ensi

on 1

: D

evel

opin

gD

imen

sion 2

: D

evel

opin

g

Lit

tle

evid

ence

show

n i

n s

om

e ar

eas

yet

ther

e is

A L

OT

in t

he

port

foli

o!

but

it d

oes

n't

serv

e th

e p

urp

ose

.

49

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30831

Dim

ensi

on 1

:O

K/N

E

Dim

ensi

on 2

:

Work

Shee

t (D

2)

Work

Shee

t (D

1&

2--

1N

E)

Ess

ay C

har

t (D

1&

2)

Tap

e (D

1&

2)

Para

gra

ph

(D

2)

Post

er (

D2)

Ess

ay (

D1&

2)

Para

gra

ph

(D

2)

Pam

phle

t (D

1&

2)

Invit

ació

n (

D2)

Ess

ay (

D2)

Ess

ay (

D2)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

/ P

rom

isin

g h

ispan

ic f

igure

s

Dim

ensi

on 2

: P

rom

isin

g

Note

: C

over

shee

ts n

ot

incl

uded

.

Los

Chiv

elos[

?] (

D1

)F

usi

ón (

D1)

--ev

iden

ce s

ho

wn

on

ly o

n c

ov

er s

hee

tco

mm

ents

[?]

(D2

)Is

la P

ascu

a (D

2)

Fra

nco

(D

1)

Mar

tí/J

uár

ez (

D1)

Bio

gra

fía

(D2)

Pet

e S

ampra

s (?

)C

art

a [

?]

(D2

)P

asc

ua [

?]

(D2

)B

lasó

n (

D2)

Via

je [

?]

(D2

)P

intu

ras

(D1&

2)

Dim

ensi

on 1

: P

rom

isin

gD

imen

sion 2

: A

ccom

pli

shed

50

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

31

92

9O

K-P

erfe

ctQ

uiz

(D

1&

2)

Pla

n d

e V

uel

o (

D1

&2

)F

oll

eto

Mex

ico

(D

1&

D2

)B

anan

a S

pli

t (D

2)

Vid

eo (

D2

)P

oem

a "L

a M

ura

lla"

(D

1&

D2)

Tap

e-her

o (

D1&

D2--

1N

E)

Tap

e &

Co

llag

e (D

2)

Tap

e (

?) (

D2)

Dim

en

sio

n 1

: P

rom

isin

g

Dim

en

sio

n 2

: A

cco

mp

lish

ed

Est

a es

su

vid

a (D

2)

Mex

ico

(D

1&

D2)

Ban

ana

Sp

lit

(D1

&D

2--

2?)

Her

o o

f Y

ear

(D1&

D2--

1?)

Vid

eo -

-No

t in

clu

ded

Bo

lív

ar ?

(D

1&

D2

)H

isp

anic

s in

US

A (

D1

&D

2)

La

Mura

lla

(NE

)

Dim

ensi

on

1:

Pro

mis

ing

. H

ow

ever

, st

ud

ent

cou

ld h

ave

incl

ud

ed m

ore

pro

du

cts

.

Dim

2.:

Acc

om

pli

shed

. S

tud

ent

incl

ud

es m

ore

ev

iden

ce o

f d

imen

sio

n 2

than

dim

en

sio

n 1

.N

ote

: S

he

has

mo

re a

rtif

acts

to

pro

ve

evid

ence

of

wri

tin

g s

kil

ls.

Sh

ed

idn

't i

nclu

de a

ny

ev

iden

ce o

f li

sten

ing

co

mp

reh

en

sio

n.

31296

Dim

ensi

on 1

:N

O/N

E

Dim

ensi

on 2

: O

K

Car

tel:

Logo d

e P

aca

(D2

)C

arta

(D

2)

Tap

e: E

spañ

a (D

1&

2)

Tap

e: M

adre

Ter

esa

(D1

&2

)L

ette

r: I

sla

(D1&

2--

1N

E)

Work

Shee

ts:

Coló

n, C

ort

és, Is

abel

, D

. M

arin

a (D

1)

Wri

ting:

Auto

bio

gra

fía

(D2)

Wri

ting:

Food (

D1&

2)

Sum

mar

y:

(D1&

2--

1N

E)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

Note

: C

hoic

e of

dim

ensi

on d

idn't

alw

ays

show

evid

ence

for

mak

ing a

ppro

pri

ate

judgem

ent.

S

tuden

t's w

riti

ng a

nd s

pea

kin

g d

emonst

rate

d w

ell

arti

cula

ted m

essa

ges

.

Post

er (

D2)

Car

ta (

D2)

Car

ta/l

ectu

ra (

D1&

2)

Work

shee

t (D

1&

2)

Tap

e (D

2)

Bio

gra

fía

(D2)

Receta

(D

2)

Res

earc

h (

D1

&2

)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

Dim

ensi

on 2

: P

rom

isin

g

51

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30351

Dim

ensi

on 1

: O

K-

Per

fect

Dim

ensi

on 2

:O

K

Res

um

en:

Caj

as (

D2

)T

ape:

Mar

tha

Ander

son

(D

1&

2)

Wri

ting:

Goya/

Pic

asso

(D

1)

Wri

ting:

Popo/I

xtl

a (D

2)

Wri

ting:

Guer

ra C

ivil

(D

1&

2)

Wri

ting:

Ben

it./

Cháv

ez (

D1&

2)

Post

er:

A l

a der

iva

(D2

)P

ost

er:

La

selv

a (D

2)

Post

er:

Imm

igra

tion (

D2)

Tap

e: M

alin

che/

Isab

el (

D1

&2

)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

Post

er w

ith p

rese

nta

tion (

D2)

A l

a d

eriv

a (D

2--

NE

)T

ape

(D1)

Tap

e (D

1&

D2)

Res

um

en (

D2)

Work

Shee

t (D

2)

Ess

ay (

D1)

Ess

ay (

D2)

Ess

ay (

D1&

D2)

Ess

ay (

D1&

D2)

Dim

ensi

on 1

: P

rom

isin

gD

imen

sion 2

: P

rom

isin

g

30358

Dim

ensi

on 1

: O

K

Dim

ensi

on 2

: O

K

Com

p.:

Juár

ez/M

artí

(D

1&

2)

Isab

el/M

alic

he

(D1

&2

)B

enit

o J

uár

ez (

D1&

2)

El

Gre

co (

D1

&2

)Je

rry

Gar

cía

(D1

&2

)C

oll

age:

Popo/I

xtl

a (D

1&

2)

Auto

bio

gra

fía

(D2)

Dim

ensi

on 1

: A

ccom

pli

shed

Dim

ensi

on 2

: A

ccom

pli

shed

Post

er:

Quie

n y

o ...

(D

2)

A l

a D

eriv

a (D

2)

Ixtl

a/P

opo (

D1&

2)

Jerr

y G

arcí

a (D

2)

Isab

el/L

a M

alic

he

(D1

&D

2)

Ben

ito J

uár

ez (

D1&

2)

Culm

inat

ing A

ss. (D

2)

Esp

aña/

Gre

co (

D1)

--st

uden

t did

n't

indic

ated

that

she

wan

ted t

o u

se C

.A.

as e

vid

ence

.L

os

may

as (

D1)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: P

rom

isin

g

Note

: S

ize

of

som

e of

the

pro

duct

s w

as v

ery l

arge

and d

iffi

cult

to h

andle

. "I

nte

racc

iones

per

sonal

es"

- S

tuden

t under

stood i

t as

cooper

atio

n i

nst

ead o

fsp

eak

ing

.

52

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30144

Dim

1:

NO

-NE

Dim

ensi

on 2

OK

-Perf

ect

Ess

ay (

D1&

D2)

Ess

ay/P

ost

Car

d (

D2)

Wo

rk S

hee

t-p

arag

rap

h (

D2

)P

ost

er (

D2)

Pic

ture

(D

2)

Dra

win

g (

D1&

2)

Esc

udo (

D2)

Men

u (

D2)

Párr

afo

(D

2)

Post

er (

D1&

2--

NE

)P

ost

er (

D2)

Par

agra

ph (

D1&

D2

)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

Dim

ensi

on 2

: P

rom

isin

g.

El

[? -

"ile

gib

le"]

(D

1&

D2)

Bis

(D

2)

Moai

s (D

1&

D2)

Chic

os

de

hoy (

D2

)[?

-"il

egib

le"]

(D

2)

Bio

gra

fía

(D2)

Com

ida

(D1&

D2)

[? -

"il

egib

le"]

(D

2)

Men

u (

D2)

Exti

nci

ón (

D2)

Cum

bre

(D

1&

D2--

1N

E)

Mo

da

(D2

--N

E)

Ch

ang

e[?]

(D

1-D

2)

Dim

ensi

on 1

: P

rom

isin

gD

imen

sion 2

: A

ccom

pli

shed

30460

Dim

ensi

on 1

: O

K

Dim

ensi

on 2

: O

K-

Per

fect

Ess

ay:

Guer

ra C

ivil

Esp

aña

(D1

)P

ascu

a post

card

(D

1)

C.A

. (D

2)

Deb

ate

(D1

&2

)E

ssay

(D

2)

Wo

rksh

eet

(D2

--N

E)

Esc

udo (

D1)

Cháv

ez

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

in s

pea

kin

g. N

ot

enough e

vid

ence

to f

ull

y j

udge

oth

eras

pec

ts o

f th

is d

imen

sion.

No

te:

Hav

e u

sed

to

sco

re a

spec

ts n

ot

mar

ked

by

stu

den

t.

Com

p.:

Guer

ra C

ivil

, G

oya

(D1)

Tarj

eta

(D

1)

Tap

e: O

rale

s (D

2)

Com

p.:

Per

sona

adm

iro (

D2)

Can

ción:

Chic

as d

e hoy (

D2)

Pro

yec

tos

Dim

ensi

on 1

: A

ccom

pli

shed

Dim

ensi

on 2

: A

ccom

pli

shed

Note

: pro

ject

s did

n't

hav

e co

ver

shee

ts, but

ther

e is

ple

nty

of

evid

ence

in a

ll f

ield

s.

53

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30816

Dim

ensi

on 1

: O

K

Dim

ensi

on 2

: O

K

Com

p.:

Mar

tí,

Juár

ez,

Cháv

ez (

D2

)S

um

mar

y:

A l

a der

iva

(D2

)W

ork

shee

t: B

olí

var

/Was

h.

(D2

)S

um

mar

y (

D2)

Outl

ine:

Goya/

Pic

asso

(D

2)

Work

shee

t: H

ispan

os

(D1

)S

um

mar

y:

(D1)

Cas

sett

e: C

.A. (D

2)

Dim

ensi

on 1

: D

evel

opin

g

Dim

ensi

on 2

: P

rom

isin

g

Note

: S

tuden

t's p

ort

foli

o n

eeds

more

evid

ence

in d

imen

sion 1

. L

ack o

f pro

duct

s re

sult

in a

n i

nab

ilit

y t

o a

rriv

e at

an a

sses

smen

t oth

er t

han

dev

. D

im 2

pro

vid

es m

ore

evid

ence

and r

esult

s in

ass

essm

. of

pro

mis

ing.

Anci

anos

(D2)

Com

p.:

Mar

tí/C

háv

ez (

D1&

2)

Com

p.:

A l

a der

iva

(D2

)B

olí

var

(D

2)

[?]

(D2

)G

oya

y P

icas

so (

D1

)[?

] H

ispan

ic (

D1&

2)

Mar

tí/J

uár

ez (

D1)

Mia

mi

(did

n't

indic

ate

dim

ensi

on)

Ass

ess.

(did

n't

indic

ated

imen

sion)

Mi

her

man

a (d

idn't

indic

ate

dim

ensi

on)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

30435

Dim

ensi

on1:

OK

Dim

2:

OK

- P

erfe

ct

Pam

phle

t: a

uto

bio

gra

phy (

D1&

2)

Map

: M

exic

o (

D1

&d

2)

Outl

ine:

Popo/I

xtl

a (D

1&

2)

Cas

sett

e: A

la

Der

iva(

D1

&2

--1

NE

)S

um

mar

y:

El

maí

z (D

1&

2--

1N

E)

Let

ter:

Isl

a de

Pas

cua

(D1

&2

)S

um

mar

y:

El

Gre

co (

D1

&2

)P

oem

: B

lasó

n (

D1&

D2)

Pam

phle

t: E

spañ

a (D

1&

D2

)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

Note

: V

ery g

ood p

ort

foli

o e

spec

iall

y w

riti

ng, sp

eakin

g a

nd c

om

pre

hen

sion a

nd

read

ing. S

tuden

t does

not

show

evid

ence

of

contr

ibuti

on o

f H

ispan

ic f

igure

s but

chec

ks

the

spac

e n

ever

thel

ess.

N

eed

more

in

form

atio

n.

Juli

a (D

1&

2--

NE

)D

eriv

a (D

1&

2)

Map

a M

exic

o (

D1

&D

2--

2N

E)

Popo (

D1&

2)

Maí

z (D

1&

2--

1N

E)

Car

ta (

D1&

2)

Vam

isa?

(D

1&

2)

Pin

tura

Gre

co (

D1

&2

--2

NE

)B

lasó

n (

D1&

2)

Esp

aña

(D1&

2)

Dim

ensi

on1:

Acc

om

pli

shed

Dim

ensi

on 2

: A

ccom

pli

shed

54

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Ag

reem

ent

Art

ifac

tsC

om

men

ts

30124

Dim

ensi

on1 :

NO

-N

E

Dim

ensi

on 2

:O

K-P

erf

ect

Bio

gra

fía

(D2--

NE

)C

en

a (

D2

--N

E)

Cer

van

tes

(D1&

2)

Itin

erar

io M

exic

o (

D1

&2

)D

ebat

e E

coló

gic

o (

D1

&2

)A

nim

ales

pel

igro

ex

t. (

D2

--N

E)

La

moda

(D2)

Deri

va (

D2

)Is

la d

e P

ascu

a (D

2)

Chic

as d

e hoy (

D2)

Gru

pos

étnic

os

(D1

)D

iari

o (

D2)

Dim

ensi

on 1

: A

ccom

pli

shed

(S

hould

hav

e ex

pla

ined

more

, but

pro

ved

more

than

pro

mis

ing)

Dim

ensi

on 2

: P

rom

isin

g (

close

to a

ccom

pli

shed

but

no q

uit

e th

ere:

adeq

uat

e re

gis

ter

and

flu

ency

)

Post

er (

D1&

2--

no

t en

ou

gh

)P

ost

er (

D1&

2--

no

t en

ou

gh

)P

ost

er (

D2)

Guid

e (D

1&

2)

Post

er (

D1&

2--

no

t en

ou

gh

)C

um

bre

(D

1&

2--

s n

ot

enough)

La

Moda

(D2--

not

eno

ug

h)

A l

a der

iva

(D2)

Post

al (

D2)

Song (

D2)

Res

earc

h (

D1

&2

)D

iari

o (

D2)

Dim

ensi

on 1

: N

ot

enough t

o e

val

uat

e w

hole

dim

ensi

on (

on t

he

firs

t as

pec

t: p

rom

isin

g)

Dim

ensi

on 2

: P

rom

isin

g

55

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30476

Dim

ensi

on 1

: N

O-

NE

Ess

ay (

D2)

Ess

ay (

D2)

Lis

t (D

1&

2)

Invit

acio

nes

(D

2)

Lis

ta C

om

ida

(D1)

Ess

ay (

D2)

Post

er (

D2)

Coll

age

(D2)

Wo

rk-s

heet

(D2

)D

raw

ing (

D1&

2--

NE

)E

xpla

nat

ion-s

ket

ch (

D1

&2

--1

NE

)E

ssay

(D

1&

2)

Tap

e (D

2)

Tap

e (D

2&

2)

Ess

ay (

D1&

2)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

Dim

ensi

on 2

: P

rom

isin

g

Com

p.:

Art

ista

Lo

ca (

D2

)T

ape:

Invit

ació

n b

anq

uet

e (D

2)

Tap

e: D

ebat

e ec

oló

gic

o (

D2

)C

om

p.:

Goya/

Pic

asso

(D

1&

2--

1?)

Str

ip s

tory

: P

opo (

D1&

2--

1?)

Dib

ujo

: A

la

der

iva

(D1

&2

--1

?)C

oll

age:

Auto

bio

gra

fía

(D2)

Nam

e P

roje

ct:

(D2

)C

om

p.:

Bio

gra

fía

(D2)

Lis

ta c

om

ida:

(D

1)

Invit

ació

n (

D2)

Esl

abón (

D1&

2)

Dim

ensi

on 1

: D

evel

opin

gD

imen

sion 2

: P

rom

isin

g

Note

: T

oo m

uch

of

sam

e kin

ds

of

mat

eria

ls. D

idn't

know

how

to c

hoose

bes

t w

ork

s.N

ot

eno

ug

h e

vid

ence

in

so

me

case

s.

56

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30457

Dim

ensi

on 1

:O

K/N

O-N

E

Dim

ensi

on 2

: N

O-

NE

Tap

e (D

2)

Post

er/E

ssay

(D

1&

2)

Car

ta (

D2)

[bla

nco

en d

oc.

] (D

1&

2)

Work

Shee

t (D

2)

Post

er (

D2)

Esc

ud

o (

D2

--N

E)

Ess

ay (

D1&

2)

Invit

acio

nes

(D

2)

Vid

eo (

D1)

Post

er (

D1&

2--

2N

E)

Sk

etch

(D

2--

NE

)

Dim

ensi

on 1

: D

iver

sity

=pro

mis

ing/N

ot

enough e

vid

ence

.

Dim

ensi

on 2

: P

rom

isin

g, but

my s

ense

is

that

wit

h m

ore

evid

ence

it

would

be

acco

mp

lish

ed.

Tap

e (C

.A.)

(D

2)

Pic

ture

seq

. (D

1&

2--

1N

E)

Car

ta (

D2)

I. P

ascu

a (D

2)

C.

de

hoy (

D2)

Coll

age

(D2--

NE

)E

scu

do

(D

2--

NE

)It

iner

ario

de

Via

je (

D2

--N

E)

Lis

ta c

om

idas

(D

2--

NE

)C

ena

de

honor

(D2)

Leo

par

do

(lo

go

) (D

2--

NE

)C

oll

age

(D2--

NE

)V

ideo

(D

1--

NE

)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

Dim

ensi

on 2

: V

ery l

ittl

e ev

iden

ce, ca

n't

judge

over

all

dim

ensi

on.

30394

OK

-Perf

ect

Pla

nes

de

Vuel

o -

-no

in

clu

ye

las

ref

lexio

nes

(D

1)

Wri

ting:

Tar

jeta

Post

al (

D2)

Clo

ze (

D1&

2--

1N

E)

Pam

phle

t: A

uto

bio

gra

fía

(D2)

Pam

phle

t: V

iaje

(D

1)

Wri

ting:

Tole

do (

D2

)T

ape:

La

Sel

va

(D2

)W

riti

ng:

La

Guer

ra (

D1

&D

2)

Over

all

evid

ence

in t

his

port

foli

o i

ndic

ates

stu

den

t has

pro

gre

ssed

to t

he

level

that

show

s his

abil

ity t

o h

ave

gen

eral

insi

ght

and u

nder

stan

din

g a

nd d

ispla

y e

xam

ple

s of

dif

f.as

pec

ts o

f th

e cu

ltu

re.

Tap

e (D

1&

2)

# 1

& T

ape

# 2

(D

1&

2)

Vid

eo (

NE

)-re

ad!

Bio

gra

fía

(D2)

Bro

chure

(D

1&

D2

)R

esum

en (

D2)

# 1

& R

esum

en # 2

(D

2)

Bla

són (

D2--

NE

)P

lan d

e vuel

o (

D1

&D

2)

Dim

ensi

on 1

: P

rom

isin

g f

or

only

2 o

f th

e as

pec

ts;

not

enough e

vid

ence

to j

udge

dim

ensi

on a

s a

whole

, si

nce

contr

ibuti

ons

are

mis

sing.

Dim

ensi

on 2

: P

rom

isin

g (

gre

at d

iffe

rence

s bet

wee

n r

ecep

tive

skil

ls a

nd p

roduct

ive

skil

ls!)

57

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Ag

reem

ent

Art

ifac

tsC

om

men

ts

30396

Bla

són (

D1&

2)

Esp

aña

(D1&

2)

Los

May

as (

D1&

2)

Los

Azt

ecas

y l

os

May

as (

D1

&2

)

Pla

n d

e vuel

o (

D1

&2

)

Goya

& P

icas

so (

D2

)

Com

posi

ción:

Yo s

oy (

D2)

Com

posi

ción:

A l

a D

eriv

a (D

2)

Tap

e: D

eriv

a (D

2)

Pic

asso

& G

oya

(D1

&2

)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

No

te:

To

o m

uch

mat

eria

l.

Cas

sett

e:

A l

a D

eriv

a (D

2)

Cas

sett

e: G

oya

y P

icas

so (

D1&

D2)

Yo s

oy (

D2)

Ess

ay:

Ixtl

a y P

opo (

D1&

D2)

Let

ter:

Quer

ido A

leja

nd

ro (

D1

&D

2)

Vea

las

sel

vas

tro

pic

ales

(D

2)

Pla

n d

e vuel

o:

Los

may

as (

D1&

D2)

Coll

age:

Ess

ay:

Axte

cas,

may

as y

tolt

ecas

(D

1&

D2)

Ess

ay:

Tre

s obra

s d

e lo

s m

ayas

(D

1&

D2

)

Una

Esp

aña:

Much

as c

ult

ura

s (D

1&

D2

)

Bla

són y

cognad

os

del

bla

són (

D1&

D2)

Dim

ensi

on 1

: A

ccom

pli

shed

Dim

ensi

on 2

: A

ccom

pli

shed

Note

: Y

o s

oy -

- I

could

n’t

judge

the

pro

duct

bec

ause

stu

den

t did

n’t

incl

ude

soru

ce.

T

res

obra

s de

los

may

as -

- I

could

n’t

fin

d t

his

docu

men

t in

the

port

foli

o.

G

ener

al:

Stu

den

t in

cluded

man

y d

ocu

men

ts a

nd p

roje

cts

to s

how

evid

ence

of

pro

gre

ss i

n D

imen

sion 1

. It

show

s th

at s

he

work

ed o

n t

his

dim

ensi

on t

hro

ughout

the

cou

rse.

58

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30390

Dim

ensi

on 1

: N

O

Dim

ensi

on 2

: O

K

Vid

eo (

D2)

Ess

ay (

D1&

2)

Ess

ay (

D1&

2)

Work

Shee

t (D

2)

Ess

ay (

D2)

Vid

eo (

D2)

3 T

apes

(D

1&

2)

Ess

ay, P

amph. (D

1&

2)

Dim

ensi

on

1:

Ad

van

ced

Dim

ensi

on

2:

Ad

van

ced

No

te:

cov

er s

hee

ts w

ere

exce

llen

t so

urc

e o

f w

riti

ng

an

d c

om

pre

hen

sio

n

Esp

aña

May

asG

uer

nic

a/3 M

ayo

Méx

ico

Bla

són

Vid

eo:

Me

llam

o S

onia

Vid

eo:

A l

a D

eriv

a

Dim

ensi

on 1

: P

rom

isin

gN

ote

: A

lthough t

he

studen

t did

n't

hav

e m

any a

rtif

acts

, I

could

judge

that

he/

she

was

at

leas

t P

rom

isin

g i

n d

emonst

rati

ng k

now

ledge

of

his

pan

ic c

ult

ure

s.N

ote

: S

tuden

t had

one

pro

duct

to s

how

evid

ence

of

the

div

ersi

ty o

f his

p. cu

lture

s.D

imen

sion 2

: A

ccom

pli

shed

Note

: B

ased

on S

/W;

how

ever

, sh

e did

n't

show

evid

ence

of

read

ing c

om

pre

hen

sion

31948

Dim

ensi

on 1

:N

O-N

E

Dim

ensi

on 2

:O

K/N

O-N

E

Dis

k:

Judge

(D1&

2)

Work

shee

t: P

[?]

(D1

&2

)C

om

p.:

Auto

bio

gra

fía

(D2

)P

royec

to (

D2)

Pro

yec

to (

D1&

2)

Tap

e (l

is.)

(D

1&

2)

Tap

e (s

pea

k.)

(D

2)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: P

rom

isin

g

Note

: S

he

seem

s to

be

bet

ter

than

this

evid

ence

(bec

ause

of

the

wri

ting).

WorK

Shee

t (D

1&

2)

Ess

ay (

D1&

2)

Ess

ay d

ibujo

s (D

2)

Ess

ay (

D2)

Work

Shee

ts (

D1&

2)

Lis

t &

Par

agra

ph (

D2

)T

ape

(D2)

Tap

e (D

2)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

.

Dim

ensi

on 2

: D

eriv

ing m

eanin

g=

no e

vid

ence

/Expre

ss a

ccom

pli

shed

Note

: I

bel

ieve

she

may

be

bet

ter

than

pro

mis

ing b

ut

I don't

hav

e en

ough e

vid

ence

.

59

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Ag

reem

ent

Art

ifac

tsC

om

men

ts

30843

2 N

ewsp

aper

, A

rtic

les

(D1

&2

)

Com

p.:

C

háv

ez,

Mar

tí a

nd

Juár

ez (

D1

&2

)

A l

a D

eriv

a (D

2)

Auto

bio

gra

fía

(D2)

Vid

eo:

Ixtl

a/P

opo (

D2)

Vid

eo:

A l

a D

eriv

a (D

2)

Tap

e: T

est

(D2)

Fly

er:

Gra

cia[

?] (

D1

&2

)

Dim

ensi

on 1

: D

evel

opin

g

Dim

ensi

on 2

: D

evel

opin

g

Let

ter

(D2)

Ess

ay:

A l

a der

iva

(D2

)E

ssay

: M

artí

, J

uár

ez y

Cháv

ez(D

1&

2)

Exam

(D

2)

Note

s fr

om

pre

senta

tio

ns(

D1

&2

)L

as A

rtes

de

Mad

rid

(D1

&2

)A

rroz

con P

oll

o(D

1&

2)

[?]

Vid

eo (

D2)

Cas

sett

e (D

2)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: P

rom

isin

g

Note

: A

lthough s

tuden

ts s

eem

s to

be

bet

ter

wit

h r

eadin

g a

nd l

iste

nin

g, he

only

incl

uded

1 s

ample

of

each

to p

rove

his

pro

gre

ss w

hil

e he

incl

uded

man

y p

roduct

s to

eval

uat

e his

wri

ting a

nd s

pea

kin

g s

kil

ls

30342

Dim

ensi

on 1

: O

K

Dim

ensi

on 2

: O

K

Post

er:

Bla

són (

D1&

2)

Post

er:

[?]

(D2)

[?]

Bla

són

Was

h/B

olí

var

(D

1&

2)

Car

ta (

dim

ensi

on?)

Isab

el/M

alic

he

(D1

&2

)E

spañ

ol

(D1&

2)

Car

ta (

dim

ensi

on?)

Caj

as (

dim

ensi

on?)

Mar

tí/C

háv

ez (

D1

&2

)

Dim

ensi

on 1

: A

com

pli

shed

Dim

ensi

on 2

: A

ccom

pli

shed

(All

art

if.

use

d a

s m

easu

rin

g p

erfo

rman

ce i

n b

oth

Dim

ensi

on)

Wri

ting:

Mar

tíC

háv

ezT

ape:

Mal

inch

e/Is

abel

Wri

ting:

Esp

aña

Wri

ting:

Was

h./

Bolí

var

Wri

ting:

Caj

as d

e C

arto

nP

ost

er:

Bla

són

Post

er:

Quié

n s

oy?

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: P

rom

isin

g

Note

: S

tuden

t's p

ort

foli

o h

as e

vid

ence

for

all

aspec

ts o

f ea

ch d

imen

sion. V

ery s

trong

wri

ting a

nd s

pea

kin

g y

et o

ver

all

impre

ssio

n i

s of

a ver

y p

rom

isin

g p

ort

foli

o.

60

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Ag

reem

ent

Art

ifac

tsC

om

men

ts

30814

OK

-Perf

ect

Los

Paí

ses

(D1)

Fusi

on d

e C

ult

o (

D1--

NE

)T

ape:

Ju

árez

/Ch

ávez

(D

1&

D2

)L

anta

ro (

D1

&D

2)

Un

? p

oet

a (D

2)

Isab

el/L

a M

alic

he

(D2

)T

ape:

Mi

her

man

a (D

2)

Fel

iz c

um

ple

años

Mia

mi

(D2)

Culm

. A

ss.

(D2)

Dim

ensi

on 1

: P

rom

isin

g. S

tuden

t did

n't

incl

ude

enough p

roduct

s to

rea

lly s

ay t

hat

she

was

acc

om

pli

shed

on d

imen

sion 1

.

Dim

ensi

on 2

: A

ccom

pli

shed

.

Tap

e &

Vid

eo:

Sis

ter

(D2

)T

ape

& V

ideo

: P

op

o (

D2

)N

ote

s: L

os

paí

ses

(D1

)C

om

posi

tion:

J

uár

ez/C

háv

ez/M

artí

(D

1&

D2

)C

oll

age:

Lan

taro

Work

Shee

t: I

mm

igra

nts

(D

2)

Work

Shee

t: I

sabel

/ L

a M

alic

he

(

D1&

D2 -

did

n't

incl

ude

pag

e

w

hic

h r

equ

ires

in

fere

nce

s)C

om

posi

tion: F

eliz

Cum

ple

. N

EL

ette

r of

imm

igra

nt

(D2

)T

ape:

Med

io A

mb

ien

te (

D2

)T

ape:

Cum

bre

(D

2)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

Note

: put

a co

ver

on e

ver

yth

ing s

ubm

itte

d.

31296

Dim

ensi

on 1

:N

O/N

E

Dim

ensi

on 2

: O

K

Car

tel:

Logo d

e P

aca

(D2

)C

arta

(D

2)

Tap

e: E

spañ

a (D

1&

2)

Tap

e: M

adre

Ter

esa

(D1

&2

)L

ette

r: I

sla

(D1&

2--

1N

E)

Work

Shee

ts:

Coló

n, C

ort

és, Is

abel

, D

. M

arin

a (D

1)

Wri

ting:

Auto

bio

gra

fía

(D2)

Wri

ting:

Food (

D1&

2)

Sum

mar

y:

(D1&

2--

1N

E)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: A

ccom

pli

shed

Note

: C

hoic

e of

dim

ensi

on d

idn't

alw

ays

show

evid

ence

for

mak

ing a

ppro

pri

ate

judgem

ent.

S

tuden

t's w

riti

ng a

nd s

pea

kin

g d

emonst

rate

d w

ell

arti

cula

ted m

essa

ges

.

Post

er (

D2)

Car

ta (

D2)

Car

ta/l

ectu

ra (

D1&

2)

Work

shee

t (D

1&

2)

Tap

e (D

2)

Bio

gra

fía

(D2)

Receta

(D

2)

Res

earc

h (

D1

&2

)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

Dim

ensi

on 2

: P

rom

isin

g

61

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Ag

reem

ent

Art

ifac

tsC

om

men

ts

31948

Dim

ensi

on 1

:N

O-N

E

Dim

ensi

on 2

:O

K/N

O-N

E

Dis

k:

Judge

(D1&

2)

Work

shee

t: P

[?]

(D1

&2

)C

om

p.:

Auto

bio

gra

fía

(D2

)P

royec

to (

D2)

Pro

yec

to (

D1&

2)

Tap

e (l

is.)

(D

1&

2)

Tap

e (s

pea

k.)

(D

2)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: P

rom

isin

g

Note

: S

he

seem

s to

be

bet

ter

than

this

evid

ence

(bec

ause

of

the

wri

ting).

WorK

Shee

t (D

1&

2)

Ess

ay (

D1&

2)

Ess

ay d

ibujo

s (D

2)

Ess

ay (

D2)

Work

Shee

ts (

D1&

2)

Lis

t &

Par

agra

ph (

D2

)T

ape

(D2)

Tap

e (D

2)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

.

Dim

ensi

on 2

: D

eriv

ing m

eanin

g=

no e

vid

ence

/Expre

ss a

ccom

pli

shed

Note

: I

bel

ieve

she

may

be

bet

ter

than

pro

mis

ing b

ut

I don't

hav

e en

ough e

vid

ence

.

30457

Dim

ensi

on 1

:O

K/N

O-N

E

Dim

ensi

on 2

: N

O-

NE

Tap

e (D

2)

Post

er/E

ssay

(D

1&

2)

Car

ta (

D2)

[bla

nco

en d

oc.

] (D

1&

2)

Work

Shee

t (D

2)

Post

er (

D2)

Esc

ud

o (

D2

--N

E)

Ess

ay (

D1&

2)

Invit

acio

nes

(D

2)

Vid

eo (

D1)

Post

er (

D1&

2--

2N

E)

Sk

etch

(D

2--

NE

)

Dim

ensi

on 1

: D

iver

sity

=pro

mis

ing/N

ot

enough e

vid

ence

.

Dim

ensi

on 2

: P

rom

isin

g, but

my s

ense

is

that

wit

h m

ore

evid

ence

it

would

be

acco

mp

lish

ed.

Tap

e (C

.A.)

(D

2)

Pic

ture

seq

. (D

1&

2--

1N

E)

Car

ta (

D2)

I. P

ascu

a (D

2)

C.

de

hoy (

D2)

Coll

age

(D2--

NE

)E

scu

do

(D

2--

NE

)It

iner

ario

de

Via

je (

D2

--N

E)

Lis

ta c

om

idas

(D

2--

NE

)C

ena

de

honor

(D2

)L

eop

ard

o (

log

o)

(D2

--N

E)

Coll

age

(D2--

NE

)V

ideo

(D

1--

NE

)

Dim

ensi

on 1

: N

ot

enough e

vid

ence

Dim

ensi

on 2

: V

ery l

ittl

e ev

iden

ce, ca

n't

judge

over

all

dim

ensi

on.

62

Ap

pen

dix

5 -

Co

nti

nu

atio

nP

OR

TF

OL

IOS

RA

TE

D B

Y T

WO

RE

AD

ER

S W

OR

KIN

G T

OG

ET

HE

R

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30393

Pla

n d

e vuel

o:

May

as (

D1&

2)

Au

tob

iog

rafí

a (D

2--

NE

)

Foll

eto (

D1)

Bla

són:

Poem

a (D

1&

2)

Act

ivid

ad:

Pai

nti

ngs

(D1&

2)

Popo/I

xtl

a (D

1)

Sto

ry:

A l

a der

iva

(D2

)

Let

ter:

Isl

a de

Pas

cua

(D2

)

Poem

(D

1&

2)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: P

rom

isin

g

No

te:

Rea

ks

of

pro

mis

e m

ore

th

an [

?]

Note

: P

eaks

are

more

num

erous

in t

he

area

of

pro

mis

ing. O

ver

all

impre

ssio

n i

s of

apro

mis

ing p

ort

foli

o.

30357

Guid

e: E

spañ

a(D

1)

Wr.

Rep

ort

: M

éxic

o (

D1

&2

)

Cin

ta:

Tole

do (

D1&

2)

Sum

mar

y:I

xtl

a/P

opo (

D2)

Rep

ort

: M

artí

/Juár

ez (

D2

)

Inte

rvie

w:

Pro

feso

r L

. (D

2)

Rea

din

g:

Caj

as d

e C

arto

n (

D2)

Tap

e: L

a M

alin

che

(D2

)

Dim

ensi

on 1

: A

ccom

pli

shed

Dim

ensi

on 2

: A

ccopli

shed

Note

: O

ver

all

prd

uct

s dem

onst

rate

that

stu

den

t is

acco

mpli

shed

in b

oth

dim

ensi

ons.

No

te:

Ov

eral

l p

rdo

cut

dem

on

stra

tes

acco

mp

lish

ed p

rod

uct

ion

; h

ow

ever

, sp

eak

ing

[??

]

30131

Sum

mar

y:

Popo/I

xtl

a (D

1&

2--

1N

E)

Sum

mar

y:

Isla

de

Pas

cua

(D2

)

Car

ta (

D2)

Dia

rio (

D2)

Pro

yec

to:

Bio

gra

fía

(D2

)

Pro

yec

to:

Com

p. (D

1&

2)

Monum

ento

(D

2)

Tap

e: C

.A.

(D1&

2--

1N

E)

Deb

ate

(D1

&2

--1

NE

)

Pro

yec

to:

Anim

al (

D2

)

Deb

ate

(D

2)

Cum

bre

(D

2)

Dim

ensi

on 1

: B

egin

nin

g

Dim

ensi

on 2

: A

ccom

pli

shed

63

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30149

Los

May

as (

D1&

2)

Popo e

Ixtl

a (D

2)

Méx

ico y

Esp

aña

(D1

)

Anse

l A

dam

s (D

2)

Nues

tros

Hér

oes

(D

2)

C.A

. (D

2)

Auto

bio

gra

fía

(D2)

Cen

a de

Honor

(D2

)

Anim

ales

en p

elig

ro (

D2

)

La

mode

(D2)

Dim

ensi

on 1

: P

rom

isin

g, but

I’m

not

so s

ure

sin

ce t

her

e w

asn’t

enough e

vid

ence

to

jud

ge

two

asp

ects

.

Dim

ensi

on 2

: P

rom

isin

g, bas

ed t

wo s

ample

s.

Note

: In

gen

eral

, I

thin

k t

his

port

foli

o n

eeds

more

pro

duct

s to

hav

e en

ough e

vid

ence

to

judge

it.

31942

Tap

e (D

2)

Pár

rafo

: P

oet

a (D

2)

Ess

ay (

D2)

Spea

kin

g (

D2)

Rec

eta

(D1&

2)

Note

s fr

om

hea

din

g (

D1

&2

)

Ess

ay (

D1&

2)

Sp

eech

[?](

D2

)

Ess

ay (

D1&

2)

No

te

(D2

--N

E)

Dim

ensi

on 1

: P

rom

isin

g

Dim

ensi

on 2

: P

rom

isin

g e

xpre

sion o

nly

. N

ot

enough e

vid

ence

to j

udge

rece

pti

ve

skil

ls.

30819

Cla

ss N

ote

s (D

1&

2)

Art

ículo

(D

2)

Isab

el/L

a M

alin

che

(D1

&2

)

Work

Shee

t (D

1&

2)

Mad

rid y

las

art

es (

D2

)

Tap

e: A

la

der

iva

(D2

)

Com

p.:

Cháv

ez (

D2

)

Tap

e: P

opo (

D2)

Dim

ensi

on 1

: A

ccom

pli

shed

Dim

ensi

on 2

: A

ccom

pli

shed

No

te:

Po

orl

y o

rgan

ized

. V

ery

few

ev

iden

ce o

f in

fere

nci

ng

[?]

.

No

te:

So

me

con

nec

tio

ns

wer

e m

ade

bu

t w

ere

no

t ch

eck

ed a

s ev

iden

ce.

64

Ap

pen

dix

5 -

Co

nti

nu

atio

n

Stu

den

t #

Agre

emen

tA

rtif

acts

Com

men

ts

30470

Post

er (

D2)

Ess

ay (

D2)

Ess

ay (

D1&

2)

Ess

ay (

D1&

2--

2N

E)

Lis

ta C

om

ida

(D1

&2

--2

NE

)

Invit

acio

nes

(D

2)

Tap

e (D

2)

Tap

e: C

olu

mbia

(D

1&

2)

Outl

ine

(D1&

2--

2N

E)

Foto

(D

1&

2--

NE

)

Sket

ch (

D1&

2--

NE

)

Ess

ay (

D2)

Car

ta (

D2)

Wro

k S

hee

t (D

2)

[Rea

der

s d

idn’t

in

clu

de

eval

uat

ion

fo

r ea

ch d

imen

sio

n]

31274

Ess

ay (

D1&

2)

No

ne

(D2

--N

E)

Work

Shee

t (D

1&

2)

Cover

&E

ssay

(D

1&

2)

Work

Shee

t (D

1&

2)

Ess

ay (

D1&

2)

Ess

ay

(D2)

Ess

ay

(D2)

Work

Shee

t (D

1)

Ess

ay (

D1&

2)

Ess

ay (

D1&

2)

Work

Shee

t &

Cov

er S

hee

t (D

1&

2)

Work

Shee

t &

Cov

er S

hee

t (D

1&

2)

Dim

ensi

on

1:

Ad

van

ced

Dim

ensi

on 2

: A

ccom

pli

shed

Note

: co

nsi

der

edas

pec

ts n

ot

note

d b

y s

tuden

t

65

App

endi

x 5

- C

ontin

uatio

nPO

RT

FOL

IOS

RA

TE

D B

Y O

NE

RE

AD

ER

Stud

ent #

Agr

eem

ent

Art

ifac

tsC

omm

ents

3049

8T

ape

(D2)

Tap

e (D

2)

Tap

e (D

2)

Wor

k Sh

eet (

D1)

Ess

ay (

D2)

Dra

win

g (D

2)

Post

er (

D1&

2)

Ess

ay (

D2)

Lis

t (D

1&2)

Ess

ay (

D1&

2)

Soup

(D

2)

Dim

ensi

on 1

: Not

eno

ugh

evid

ence

Dim

ensi

on 2

: Pro

mis

ing

66

Appendix 5 - Continuation

Correspondence of Student # and Portfolio #

Student # Portfolio #

31938 1

30129 2

30831 3

31929 4

31296 5

30351 6

30358 7

30144 8

30460 9

30816 10

30435 11

30124 12

30476 13

30457 14

30394 15

30396 16

30390 17

31948 18

30843 19

30342 20

30814 21

30131 22

30149 23

30470 24

30498 25

31274 26

31942 27

30819 28

30393 29

30357 30

Phase II Appendix

The Participants: Roles and Responsibilities

69

Appendix A

Preparation for Portfolio Assessment Scoring SessionsBoth the Spanish and English Pacesetter courses use portfolios as a mea-sure of student achievement. In each subject, portfolio collections of regu-lar classroom work are developed by students throughout the school yearas a way to demonstrate their achievement in relation to the coursedimensions. In both Pacesetter English and Spanish, the scoring guidesused to evaluate the portfolios at the end of the semester or year are usedthroughout the course to clarify course expectations and serve as a guideto help students build strong collections of work.

Portfolios provide a way of looking at student achievement that is quitedifferent from the other types of Pacesetter assessments. Whereas com-mon tasks and even the culminating assessment look at student achieve-ment on a particular task at a given time, the portfolio provides a view of astudent’s achievement across a body of various kinds of class work.Teachers who have participated as Readers at portfolio scorings oftencomment that looking at work from other Pacesetter classes providesmodels of instructional activities and strategies they can adapt for theirown teaching.

Pacesetter English

As with the culminating assessment, portfolio assessment focuses on eval-uating student performance in the two course dimensions of MakingMeaning from Texts and Creating and Presenting Texts. Because of thedesign of the portfolio scoring guides, it is possible to gather performance-level information about four aspects of each of the two course dimensionsas well as the two dimensions themselves. Teachers - Readers indicate thatlooking at student performance on various aspects of the course dimen-sions is especially helpful in planning the course and reflecting on theirteaching.

Pacesetter Spanish

Like the English portfolio, the Spanish portfolio also focuses on evaluatingstudent performance in two course dimensions: DemonstratingKnowledge of Hispanic Cultures and Using Spanish to CommunicateEffectively. Portfolio scoring guides for Pacesetter Spanish also make itpossible to evaluate student performance on several aspects of thesedimensions through a single reading of a portfolio.

Scoring portfolios can provide information about student achievement andprovide ideas for staff development. Because of the complex nature ofportfolios and the time it takes to score them (about 25 minutes per read-ing), different purposes can make seemingly opposing demands on thetime and activities of a portfolio reading.

The Relationship BetweenScoring Purpose and

Scoring Design

Portfolio AssessmentOverview

70

Preparation for Portfolio Assessment Scoring Sessions

In order to monitor the technical quality of scores and to determine theirreliability, either a sample of or all of the portfolios need to be read twice.Having extended discussions about the scoring criteria and how they areevidenced in student work supports staff development purposes.

Decisions need to be made about the use of time and how that relates tothe purpose(s) of the reading. Should fewer portfolios be read so that timecan be allocated for extended discussions? Should a larger sample of port-folios be read and the discussions be limited to training issues? Shouldthere be a balance in the use of time to reflect both purposes? Are thereother local purposes that might have an impact on the portfolio scoringdesign?

The following are some examples of the connection between scoring pur-pose and design:

• If the purpose of scoring portfolios is to attain scores to monitor stu-dent achievement, then a representative sample of portfolios needs tobe read. Some or all of the sample needs to be read twice.

• In some cases the purpose of a portfolio reading might be to providefeedback to teachers about how they are applying the scoring criteriain their individual classrooms. In this design, teachers might be askedto prescore a representative sample of portfolios from theirPacesetter classes and submit those portfolios for rescoring at thedistrictwide reading. Scores from the centralized reading would bereturned to teachers.

• If the primary purpose of the reading is to provide staff development,maximizing the time for teacher discussion about student perfor-mances in relation to the scoring criteria is important. Scoring asmaller representative sample of portfolios would be appropriate.

Depending on the purpose for the portfolio scoring, Readers might beasked to score each portfolio for the course dimensions and the aspects ofthe dimensions through a single reading of a portfolio. Some believe thatthis design models how teachers evaluate student performances in theirclassrooms where teachers consider several aspects of performance at thesame time.

It is also reasonable to organize a portfolio scoring so that Readers scoreonly one dimension and its aspects at a time. If a district is especially con-cerned about a particular dimension of student achievement, this designallows Readers to focus more attention on one dimension of performance.If the intent is to gain scores for all dimensions of performance within asubject area, this design requires more time because each portfolio mustbe read separately for each dimension.

Your district may have other purposes for holding a portfolio reading thatmay necessitate other designs. Determining what information is needed bywhom and for what purposes should guide your district in designing aportfolio scoring to meet your needs. No matter the purpose of the read-ing, coming together to examine student work in portfolios is a powerfulway to help Pacesetter teachers clarify and internalize the Pacesetter con-tent and performance standards.

Preparation for Portfolio Assessment Scoring Sessions

71

For more information about building and assessing portfolios, please referto the introduction in the Teachers’ Guides for Pacesetter English andSpanish.

Scoring Coordinator

As with the culminating assessment, a portfolio scoring needs a coordina-tor within the district who can champion the decision making related tothe scoring design, take responsibility for planning the scoring, involveother personnel as needed, and make necessary arrangements. In somedistricts, one person may serve as both the Scoring Coordinator and ChiefReader.

Chief Reader(s)

Because portfolio assessment is a relatively new practice there are fewexperienced Readers. An experienced Pacesetter teacher or district admin-istrator who knows the Pacesetter course well, understands the coursecontent and performance standards fully, and who can talk about how tofind evidence of the standards in student performances, can make a fineChief Reader. Responsibilities of the Chief Reader include selecting bench-mark portfolios to use during the training, conducting the training, andfacilitating discussions about evidence of achievement in student work.

Because training for portfolio scoring requires Chief Readers to know thecourse dimensions, scoring criteria, and benchmark portfolios well, somedistricts, or Chief Readers, may feel most comfortable with more than oneChief Reader. Having multiple Chief Readers may be especially helpful in dis-tricts that are scoring portfolios for the first time, since it allows ChiefReaders to combine their expertise and knowledge. Chief Readers may bemost comfortable focusing on only one course dimension and its aspects. Insome cases, the design of a portfolio scoring may require more than oneChief Reader, especially if Readers will be scoring each dimension separately.

Table Leaders

Whether or not Table Leaders are needed at a portfolio reading is deter-mined by the size and purpose of the reading. Because of the time it takesto read portfolios, Table Leaders do not generally monitor the scoring ofReaders at their table in the same way they might at a culminating assess-ment reading. Instead, their primary responsibilities are to assist with thetraining and to answer questions that arise during the reading. Each 8 to10 readers will need a Table Leader (or Chief Reader) they can turn to withquestions.

The purpose of the reading will determine the number and role of TableLeaders.

• If gathering student achievement information is the primary purpose,Table Leaders might be expected to turn their attention first to read-ing portfolios and second to answering questions, because reading alarger sample of portfolios is necessary. They might be used as anadditional Reader for selected portfolios or as a discussion facilitatorto help Readers agree on a score.

• If staff development is the focus, Table Leaders might lead additionaltable -level discussions about particular issues, facilitate discussions

The Roles andResponsibilities of the

Participants

Preparation for Portfolio Assessment Scoring Sessions

72

about portfolios between scorers with differing scores, bring ques-tions and concerns to the attention of the Chief Reader, and onlyscore portfolios as time permits.

Readers

Readers are usually current or future Pacesetter teachers or administra-tors familiar with the Pacesetter course. In some cases, teachers who donot teach Pacesetter, but who are interested in portfolio assessment orstandards-based assessment, might be interested in the experience ofscoring Pacesetter portfolios as a professional development opportunity.Some districts plan to involve local teacher training institute staff or com-munity members in readings. As with the culminating assessment, com-pensation and other incentives or conditions are governed by local condi-tions.

Aides

Although Table Leaders and Chief Readers can, if necessary, distribute,collect, and organize portfolios and scoring guide forms, Aides can be atremendous help and allow more time for Table Leaders and Chief Readersto answer scorer questions and lead discussions. Generally, one Aide isneeded for each 12 Readers.

Number-Person

If scores from the portfolio reading are to be summarized at the districtlevel, it may be helpful to involve a staff member who can set up and main-tain a data base for portfolio scores and create score reports, includingscore distributions, as needed. There is no calculation needed to deter-mine a proficiency score for portfolios because the “raw” score is the pro-ficiency level as stated in the portfolio scoring guide for that subject.

73

Preparation for Portfolio Assessment Scoring Sessions

Sample Size

As with other decisions, the purpose of the reading is important in deter-mining the number of portfolios that can be read. If more time is used fortraining and discussion, a smaller sample of portfolios can be read. Thefollowing calculation can help determine the size of the sample that can beread by the number of available Readers in the allotted time.

Worksheet for Determining the Sample Size of Portfolios

Sample Selection

If your district determines that not all of the Pacesetter portfolios can bescored at a centralized reading, then a representative sample needs to beidentified for scoring. If a sample rather than all of the portfolios is beingread, the sample should be as random and representative as possible.Some possible scenarios for sampling are listed below.

• In many districts, selecting the sample may fall to the Pacesetterteachers themselves. If this is the case, teachers should be givenguidelines about how to select a sample that is representative andbalanced for gender or other demographic features.

• In some districts, a district-level staff member may identify a randomsample of students from whom portfolios would be gathered and read.

Narrative Explanation Algorithm Calculation

A. Total hours available Days x 5.0 =A= Number of days for scoring x 5.0 hours(total hours a day minus lunch and breaks)

B. Scoring hours available A - 6 = B= A minus 6 hours for training(training time may differ by district)

C. Number of Readers available Readers = C

D. Total possible readings B x C x 2.5=D= Multiply B by C by 2.5 (portfolios read per hour)

E. Number of portfolios Portfolios to to be scored twice be double-

scored = E

F. Number of portfolios that can be D - E = Fread including those double - scored= D minus E

G. Number of classes Classes = G

H. Number of portfolios that can be F ÷ G = Hread from each class= F divided by G

Identifying the PortfolioSample

74

Preparation for Portfolio Assessment Scoring Sessions

• In other districts, teachers might send all of their portfolios to a cen-tral location where district-level staff select a random sample, balanc-ing for gender or other demographic features.

If a large pool of scorers is available or if a decision has been made toscore all of the portfolios, then the calculation below can be used to deter-mine the number of Readers needed to score the portfolios.

Worksheet for Calculating the Number of Scorers Needed

Because portfolios are built throughout the school year, the timing of aportfolio reading is dependent upon the school calendar and teacher avail-ability. Holding a scoring a few weeks before the end of the school yearallows teachers to use the training they receive and any portfolio scoresthat are available to help them determine student grades. However, dis-tricts may choose to have teachers collect a representative sample of port-folios for use at a summer portfolio reading.

As with the culminating assessment, a comfortable, well-lit, quiet locationwith reading tables is needed for a portfolio scoring. Generally, two portfo-lio Readers can comfortably read at a cafeteria table. The Chief Reader(s)is likely to need flip charts, tape, an overhead projector, and marking pens.Depending on the scoring design, number of portfolios, and security of thescoring room, a preparation room for organizing and storing portfoliosmay also be necessary.

Determining theNumber of Scorers

Facilities: Planning aSpace for the Scoring

Session(s)

Scheduling ofPortfolio Scorings

Narrative Explanation Algorithm Calculation

A. Total number of portfolios Portfolios = Ato be scored

B. Number of portfolios that will Double be double -scored Readings = B(times each portfolio will be read)

C. Number of total readings A + B = C= Add A to B

D. Number of scorer hours needed C ÷ 2.5 = D= C divided by 2.5 (portfolios per hour)

E. Total hours available Days x 5.0 = E= Number of days for scoring x 5 hours(total hours a day minus lunch and breaks)

F. Scoring hours available E - 6 = F= E minus 6 hours for training(training time may differ by district)

G. Number of scorers needed D ÷ F = G= D divided by F

Preparation for Portfolio Assessment Scoring Sessions

75

Coding portfolios can ease the collection of scores as well as the return ofportfolios. Your district may already have a system of student numbers orsome other unique code that may prove useful in portfolio identification. Ifnot, you may want to create a simple coding system and enlist teachersand students to help with coding portfolios prior to the reading.

If a portfolio code is used, a unique number for each student is needed. Acode that includes a number for the teacher, class period, and studentmay be the easiest. Precoded labels that students affix to their portfoliosprior to the reading are helpful. With the portfolios submitted for the read-ing, teachers should include a list of students and portfolio code numbers.

In order to prepare portfolios for scoring, some districts may decide tohave portfolios sent to a central location prior to the reading. Organizingthe portfolios might include selecting a random sample to be read. It mayinclude checking or affixing portfolio codes or creating a data base for col-lecting scores. It might also include creating batches of portfolios fromvarious classes to be read at particular tables. In some cases, it may bepossible to have Aides organize portfolios on the morning of the reading, ifthey are not involved in the training.

The scoring guide forms (pages 53 - 60) are meant to be copied and used atthe reading. Readers mark their scores directly on these scoring guides. IfAides are available before the reading, they can precode the forms andplace them in the portfolios. If Aide time is more limited, on the day of thereading place copies of the scoring guides on the tables for Readers. Fororganizational purposes, it is easier to copy each dimension of a subjectarea on a different color paper.

Assigning each Reader a unique code that can be used as an identifier onthe scoring guide forms may make score and reliability analyses easier.Determining whether to give Reader codes depends on local circum-stances.

Each district holding a portfolio scoring will need to select benchmarkportfolios to use during the training. Because portfolios are complicateddocuments and require extended time to read and discuss, training is usu-ally conducted using a small number (three to four) of benchmark portfo-lios. As with the culminating assessment, the Chief Reader(s) should workwith two to three other Pacesetter teachers to select benchmark portfolios.

Benchmark portfolios should represent different classrooms and include avariety of types of performances. Suggested selection criteria for bench-marks include:

• one portfolio that demonstrates strong, but not necessarily exem-plary, performance;

• another that demonstrates uneven performance across dimensions oraspects; and

• one or two other portfolios that are likely to bring about discussion onimportant issues such as evaluating English Language Learner perfor-mances, group versus individual performances, or other local concerns.

In all cases, benchmark portfolios must include work that can be easilyread and copied.

Preparing AssessmentPortfolios for Scoring

Choosing BenchmarkPortfolios to Guide

the Training

Preparation for Portfolio Assessment Scoring Sessions

76

Making copies of benchmark portfolios can be a time-consuming processthat includes:

• making a “master copy” of each benchmark portfolio (may take up toan hour to make the first copy);

• blocking out student, teacher, and school names as well as other per-sonal information that would identify the student;

• tracing writing and other marks that are too light to read;

• numbering each page in the “master copy”;

• adding a title page (for example: Benchmark A) to the “master copy”; and

• making multiple copies for use at the training.

While some Readers find it difficult to read a portfolio with another per-son, it is not uncommon to have Readers share a copy of benchmark port-folios during training. Reading a portfolio in pairs tends to promote discus-sion by the pair about how specific evidence is viewed and the judgmentsthat are being made. These discussions are important in helping Readerscome to a common understanding of what constitutes evidence of thedimensions and how to apply the scoring criteria.

At portfolio readings where Readers are scoring all dimensions of perfor-mance at the same time, most of the training is conducted by the ChiefReader. Table Leaders work with Readers at their table to help them inter-nalize the scoring criteria and use the scoring guide forms correctly. TableLeaders answer questions and refer problematic portfolios to the ChiefReader, as appropriate.

If your design allows, Table Leaders can read and score portfolios as theother Readers do. Because it takes an average of 25 minutes to read andscore a portfolio, rereading scored portfolios is not practical unless it ispart of a double -scoring design or unless a Table Leader or Chief Readerhas concerns about a particular Reader.

Prior to the reading, Table Leaders should be given copies of the bench-mark portfolios so they can familiarize themselves with the evidence. Anynotes that might have been created by the Chief Reader(s) about theassigned score for each benchmark and the evidence that supports thescoring judgment will be helpful to Table Leaders as they answer Readerquestions. It is also helpful for Table Leaders to meet with the ChiefReader(s) prior to the reading to discuss the benchmarks and possiblequestions and issues that might need to be addressed at the reading.

Portfolio-Reading Assumptions

A centralized portfolio reading, as described in this Handbook, is based onseveral underlying assumptions, including:

• scorers need to come to a commonunderstanding of the scoring criteria;

• the criteria in the scoring guides need to be applied consistently;

• scorers need to focus on the evidence of achievement in the work;

Preparing TableLeaders

Training the Readers

Preparation for Portfolio Assessment Scoring Sessions

77

• holistic judgments are based on the quality not the quantity of evi-dence; and

• scores reflect holistic judgments across the body of work in a portfolio.

Suggested Strategies for Reading Portfolios

Because portfolios often contain a large and varied collection of studentwork, they can seem unwieldy to read at first. Most Readers will have littleor no experience looking across a body of work to make judgments. Someof the following suggestions might be useful for Readers.

• Skim the entire portfolio to see the types of performances that areincluded.

• If multiple drafts are included, read the “final” draft closely, then skimthe other drafts.

• If several performances reflect the same type of work, such as a readinglog or several poems, read some selections closely and skim the others.

• Keep notes about evidence as the portfolio is read (sample evidencecharts for English and Spanish are included at the end of this docu-ment). Some Readers prefer to use “yellow stickies” to mark particu-lar pages as they read.

• Ignore teacher grades and comments.

Assigning Scores

Often the quality of performances varies within a portfolio, making it diffi-cult for Readers to assign a score. Readers are generally asked to assign ascore that reflects the preponderance of the evidence.

Sample Training Design

This section describes training for a portfolio reading in which eachReader is scoring all dimensions and aspects of performance within a sub-ject area from a single reading of a portfolio. A training process reflectingthis design is summarized in the following chart. A narrative explanationof the activities follows the chart. Approximate times for each activityhave been included for planning purposes.

78

Sample Training Activity Approximate Time

Training on the first dimension with first benchmark portfolio

• Chief Reader introduces the scoring guide 25 min.for one dimension

• Readers work in pairs reading, discussing, 40 min.and scoring the first benchmark portfolio

• Scores and issues are discussed in the 25 min.whole group

Training for additional dimension with first benchmark portfolio

• Chief Reader introduces the scoring guide 25 min.for an additional dimension

• Readers work in the same pairs to review, 20 min.discuss, and score the first benchmark portfolio for the other dimension

• Scores and issues are discussed in the 25 min.whole group

Training on second benchmark portfolio

• Chief Reader reviews the scoring guide(s) 15 min.as needed

• Readers work either in pairs or individually 30 min.to read and score a second benchmark portfolio for all aspects of performance they will be scoring during the remainder of the reading

• Scores and issues are discussed at each table 15 min.

• Issues and questions are discussed as a 30 min.whole group

Additional Training

• Discussion about issues and questions related to the scoring guides takes place after each major break

• Training on additional benchmark portfolios is done as needed

• Revisiting previously read benchmark portfolios to discuss issues is done as needed

• Individual help is provided as appropriate

Preparation for Portfolio Assessment Scoring Sessions

79

Preparation for Portfolio Assessment Scoring Sessions

As with the culminating assessment reading, the Chief Reader will want toset a professional tone by welcoming Readers, outlining the scoring task,and introducing Table Leaders and any guests. It is important to acknowl-edge that Readers come to the reading with a sense of what strong studentperformance looks like. However, it is essential that Readers develop ashared understanding of the scoring criteria and learn to apply the criteriaconsistently.

Because portfolio scoring guides are complex, training is conducted onedimension at a time. The Chief Reader begins by reviewing the scoringguide, focusing on the definitions of the aspects and the criteria for the“accomplished” and “promising” levels of performance. Involving Readersin actively looking at and discussing the scoring criteria is important totheir understanding of the differences between various aspects and perfor-mance levels. Chief Readers should then discuss the “not enough evidenceto judge” category, and other issues that the Chief Reader and TableLeaders identified prior to the reading.

Readers then work in pairs, reading together and discussing the firstbenchmark portfolio. After a pair has reviewed the entire benchmark port-folio, they should discuss and come to agreement on scores. The ChiefReader then collects, usually by a show of hands, all of the scores for thatbenchmark portfolio. After the scores are tallied, the Chief Reader involvesReaders in a discussion of each aspect of performance. Part of helpingReaders reach a common understanding is sharing the consensus scorethat had been assigned before the reading and citing specific evidence thatwas used to arrive at that score. This process of reading the first bench-mark portfolio and training on one dimension will take about 90 minutes,depending on the number of Readers and issues that arise.

After discussion is completed for the first scoring dimension, repeat theprocess for each additional dimension. The Chief Reader should begin byreviewing the scoring guide and involving Readers in identifying importantfeatures for the “accomplished” and “promising” performance levels.Readers should continue to work in the same pairs, reviewing and scoringthe same benchmark portfolio, this time for the additional dimension.After Readers have rescored the portfolio for this dimension, the ChiefReader again collects the scores, and leads a discussion about the evi-dence and consensus scores for each aspect. Because Readers are assess-ing a portfolio they have read before, this process of scoring and dis-cussing the second dimension will likely take about 70 minutes.

Training on the second benchmark portfolio is slightly different from train-ing on the first. The scoring guides may need only a brief review beforeReaders start reading the portfolio. Depending on the purpose of the port-folio reading and the number of copies of benchmark portfolios that areavailable, Readers may read the second benchmark portfolio either in apair or individually. They should, however, be reading and scoring as manydimensions and aspects as they will be expected to score during the restof the reading. After a reasonable time for reading and scoring the bench-mark (40–45 minutes), Table Leaders should collect scores and lead a dis-cussion at their table, taking note of issues and questions. The ChiefReader can then conduct a room- wide discussion about the scoring of thebenchmark, addressing specific questions on the scoring guides and com-menting on issues and questions that were raised at the tables. The train-ing on the second benchmark is likely to take about 90 minutes.

80

At this point, if it seems that most Readers are scoring at or within onepoint of the consensus score, the Chief Reader and Table Leaders maydecide to move ahead with scoring rather than training on another bench-mark. It is also possible to train on the next benchmark at the table level,having Readers read, score, and discuss aspects on which they disagree. Ifmore than one reading room is available, it is possible to have someReaders who seem confident start scoring portfolios while others receiveadditional training.

After every significant break, such as for lunch or overnight, the ChiefReader or Table Leaders will need to answer questions that Readers haveabout the scoring guide. If necessary, the Chief Reader may decide to re-train Readers using a new benchmark portfolio on the morning of the sec-ond day. It is also possible to reuse one of the benchmark portfolios thathas already been discussed as a way to refocus Readers on particularaspects or issues.

Limiting the movement of people and portfolios will make a reading moreproductive. One design for distributing portfolios is to organize portfoliosfrom different classrooms in batches (10 –12) and distribute a batch toeach table at the beginning of the day, after lunch, and as needed. Readersselect and read a portfolio from the batch, avoiding any from their ownstudents. Portfolios that have been read are placed in a “read once” stack.If the reading design calls for portfolios to be read a second time, Readersshould select portfolios from the “read once” stack and when finished,place them in a “read twice/completed” stack. Completed portfoliosshould be removed from the table periodically. When the Readers havecompleted a batch, then a new batch should be brought to the table. Thisdesign both reduces the movement of portfolios in the room and easilyallows for double readings of portfolios, if the design calls for that.

The easiest system for collecting scores is to photocopy the completedscoring guide forms, making sure that scores and identifying informationfor the portfolio and Reader are clearly visible. Some districts may wish todevelop their own system for collecting portfolio and Reader informationand scores.

Portfolio scores are reported as performance levels (using level descrip-tors from the scoring guide such as “Promising,” “Accomplished,” etc.).However, it may be important to discuss which scores should be reportedto which audiences. While teachers may find the scores for aspects of thedimensions useful in planning instruction, school board members may notneed this level of detail. Deciding which scores to report is an importantdecision each district will need to make.

As with the culminating assessment, your district may want to produce anumber of different reports using the portfolio results, including individualscore reports. The easiest form of an individual report is a copy of theactual score form that includes performance - level descriptions. In addi-tion, many teachers and administrators may find distributions to be use-ful. Caution should be used in creating distributions at the classroom orschool level because small samples may not be representative of the stu-dent population.

Preparation for Portfolio Assessment Scoring Sessions

Collecting the Readers’Scores

Reporting the Results

Controlling the Flow ofPortfolios

81

APPENDIX B

COVER SHEET (MUST ACCOMPANY EACH ARTIFACT)

PACESETTER SPANISH PORTFOLIO ASSESSMENT

NAME:

Selection: Date work was done:

Why did you select this piece of work?

What does it show or tell about you?

What did you learn from doing this assignment?

This piece of work shows evidence of the following dimension(s) or aspect(s):

• Dimension 1: Demonstrating knowledge of Hispanic cultures

Showing awareness of the diversity of Hispanic cultures

Identifying contributions of Hispanic figures

Making connections with other disciplines and own culture(s)

• Dimension 2: Using Spanish to communicate effectively

Deriving meaning from printed materials and oral discourse

Expressing meaning in oral and written form

COVER SHEET (MUST ACCOMPANY EACH ARTIFACT)

PACESETTTER SPANISH PORTFOLIO ASSESSMENT

NOMBRE:

Título: Fecha de ejecución:

¿Por qué seleccionaste esta obra?

¿Qué muestra o dice esta obra sobre ti?

¿Qué aprendiste de esta tarea?

Esta obra indica evidencia en los siguientes aspectos y dimensiones:

• Dimensión 1: Demostrar conocimiento de las culturas hispanas

Demostrar conciencia de la diversidad de las culturas hispanas

Identificar contribuciones de figuras hispanas

Hacer conexiones con otras disciplinas y cultura(s)

• Dimensión 2: Usar el español para comunicaciones eficientes

Comprensión de textos y comprensión auditiva

Expresión oral y escrita

82

83

Please list all artifacts included in your portfolio. Indicate the aspect(s) anddimension(s) for which you are showing evidence with each artifact you selected.

DIMENSION 1: Demonstrating knowledge of Hispanic cultures

• Showing awareness of the diversity of Hispanic cultures

• Identifying contributions of Hispanic figures

• Making connections with other disciplines and own culture(s)

DIMENSION 2: Using Spanish to communicate effectively

• Deriving meaning from printed materials and oral discourse

• Expressing meaning in oral and written form

App

end

ix C

PA

CE

SE

TT

ER

SP

AN

ISH

¿N

OS

CO

NO

CE

MO

S?

DIM

EN

SIO

N 1

: D

EM

ON

ST

RA

TIN

G K

NO

WL

ED

GE

OF

HIS

PA

NIC

CU

LT

UR

ES

Po

rtfo

lio

Ass

essm

ent:

Ev

alu

ati

ng

asp

ects

of

the

dim

ensi

on

Sh

ow

ing

aw

are

nes

s o

f th

e d

iver

sity

of

His

pa

nic

cult

ure

s

Beg

inn

ing

�

Dev

elo

pin

g

�

Pro

mis

ing

�

Acc

om

pli

shed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

iden

ce t

o

jud

ge

�

• re

flec

tio

n o

n h

ow

div

erse

th

e H

isp

anic

wo

rld

is

• ex

amp

les

of

sev

eral

cu

ltu

res

of

the

His

pan

ic w

orl

d

(mo

re t

han

on

e)

• re

flec

tio

n o

n i

ssu

es l

ike

curr

ent

even

ts,

his

tori

cal

hig

hli

gh

ts,

geo

gra

ph

y,

lite

ratu

re,

etc.

Stu

den

t p

rese

nts

sup

erfi

cial,

va

gu

e a

nd

fra

gm

en

tary

ev

iden

ce

of

his

/her

aw

are

ness

of

the

div

ersi

ty o

f H

isp

anic

cult

ure

s.

Stu

den

t p

rese

nts

lim

ited

,

inco

mp

lete

an

d u

nev

en

evid

ence

of

his

/her

aw

are

ness

of

the

div

ersi

ty o

f H

isp

anic

cult

ure

s. S

tud

ent

pre

sen

ts

few

sp

ecif

ic e

xa

mp

les.

Stu

den

t p

rese

nts

ev

iden

ce o

f a

gen

era

l

aw

are

nes

s o

f th

e

div

ersi

ty o

f H

isp

anic

cult

ure

s. S

tud

ent

pre

sen

ts s

om

e g

en

era

l

insi

gh

ts a

nd

exam

ple

s.

Stu

den

t p

rese

nts

am

ple

evid

ence

of

his

/her

aw

are

ness

of

the

div

ersi

ty o

f H

isp

anic

cult

ure

s. S

tud

ent

pre

sen

ts

a w

ide

ran

ge

of

ex

am

ple

s, w

ith

so

me

clea

r in

sig

hts

.

Stu

den

t p

rese

nts

con

sist

entl

y i

nsi

gh

tfu

l

evid

ence

of

his

/her

aw

are

ness

of

the

div

ersi

ty o

f H

isp

anic

cult

ure

s. S

tud

ent

pre

sen

ts

va

ried

an

d e

xte

nd

ed

exa

mp

les.

Co

nta

ins

too

litt

le e

vid

en

ce

of

aw

are

ness

of

the

div

ersi

ty

of

His

pan

ic

cult

ure

s to

just

ify

a s

core

.

Iden

tify

ing c

on

trib

uti

on

s of

His

pan

ic f

igu

res

Beg

inn

ing

�

Dev

elo

pin

g

�

Pro

mis

ing

�

Acc

om

pli

shed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

iden

ce t

o

jud

ge

�

• ex

am

ple

s o

f se

vera

l H

isp

an

ic f

igu

res

• co

ntr

ibu

tio

ns

of

con

tem

po

rary

as

wel

l as

his

tori

cal

fig

ure

s o

f th

e H

isp

anic

wo

rld

in

dif

fere

nt

fiel

ds

• co

ntr

ibu

tio

ns

of

eth

nic

gro

up

s

Stu

den

t p

rese

nts

sup

erfi

cial,

va

gu

e a

nd

fra

gm

en

tary

ev

iden

ce

of

con

trib

uti

on

s o

f

His

pan

ic f

igu

res.

Stu

den

t p

rese

nts

lis

ts

wit

h a

few

na

mes

an

d

des

crip

tio

ns.

Stu

den

t p

rese

nts

lim

ited

an

d u

nev

en e

vid

ence

of

con

trib

uti

on

s o

f H

isp

anic

fig

ure

s. S

tud

ent

pre

sen

ts

few

sp

ecif

ic n

oti

on

s a

nd

des

crip

tio

ns

of

con

trib

uti

on

s.

Stu

den

t sh

ow

s ev

iden

ce

of

som

e g

en

era

l

con

trib

uti

on

s o

f H

isp

anic

fig

ure

s. S

tud

ent

pre

sen

ts

som

e i

nfe

ren

ces

ab

ou

t

pla

ce a

nd

im

pa

ct i

n

his

tory

.

Stu

den

t p

rese

nts

am

ple

evid

ence

of

con

trib

uti

on

s

of

His

pan

ic f

igu

res.

Stu

den

t p

rese

nts

am

ple

infe

ren

ces

ab

ou

t im

pa

ct

an

d r

ole

in

his

tory

.

Stu

den

t p

rese

nts

con

sist

entl

y t

ho

rou

gh

evid

ence

of

con

trib

uti

on

s

of

His

pan

ic f

igu

res.

Stu

den

t p

rese

nts

insi

gh

tfu

l in

fere

nce

s an

d

an

aly

sis

of

con

trib

uti

on

s,

as

wel

l a

s im

pa

ct a

nd

role

in

his

tory

.

Co

nta

ins

too

litt

le e

vid

en

ce

of

con

trib

uti

on

s

of

His

pan

ic

fig

ure

s to

just

ify

a s

core

.

Mak

ing c

on

nec

tion

s w

ith

oth

er d

isci

pli

nes

an

d o

wn

cu

ltu

re(s

)

Beg

inn

ing

�

Dev

elo

pin

g

�

Pro

mis

ing

�

Acc

om

pli

shed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

iden

ce t

o

jud

ge

�

• li

nk

s to

ow

n c

ult

ure

(s)

and

per

son

al e

xp

erie

nce

s

• co

mp

aris

on

an

d c

on

tras

t o

f H

isp

anic

cu

ltu

res

wit

h

stu

den

t's

ow

n c

ult

ure

(s)

• co

nn

ecti

on

of

his

tori

cal,

eco

no

mic

, li

tera

ry,

cult

ura

l,

po

liti

cal,

geo

gra

ph

ic,

and

en

vir

on

men

tal

info

rmat

ion

on

Sp

anis

h -

spea

kin

g c

ou

ntr

ies

to s

tud

ent's

wo

rk i

n s

oci

al

stu

die

s, E

ng

lish

, an

d o

ther

are

as o

f th

e cu

rric

ulu

m

Stu

den

t p

rese

nts

sup

erfi

cial

an

d i

sola

ted

evid

ence

of

con

nec

tio

ns

wit

h o

ther

dis

cip

lin

es

an

d o

wn

cu

ltu

re(s

).

Stu

den

t p

rese

nts

irrele

va

nt

ex

am

ple

s.

Stu

den

t p

rese

nts

lim

ited

,

part

ial

an

d u

nev

en

evid

ence

of

con

nec

tio

ns

wit

h o

ther

dis

cip

lin

es a

nd

ow

n c

ult

ure

(s).

Stu

den

t

pre

sen

ts e

xa

mp

les

tha

t

are

no

t a

lwa

ys

rele

va

nt.

Stu

den

t p

rese

nts

ev

iden

ce o

f so

me

gen

era

l co

nn

ecti

on

s

wit

h o

ther

dis

cip

lin

es

an

d o

wn

cu

ltu

re(s

).

Stu

den

t p

rese

nts

exa

mp

les

wit

ho

ut

mu

ch

dep

th o

r in

sigh

t.

Stu

den

t p

rese

nts

am

ple

evid

ence

of

con

nec

tio

ns

wit

h o

ther

dis

cip

lin

es a

nd

ow

n c

ult

ure

(s).

Stu

den

t

pre

sen

ts s

om

e

infe

ren

ces

an

d a

wid

e

ran

ge

of

insi

gh

tfu

l

exa

mp

les.

Stu

den

t p

rese

nts

con

sist

entl

y t

ho

rou

gh

evid

ence

of

con

nec

tio

ns

wit

h o

ther

dis

cip

lin

es a

nd

ow

n c

ult

ure

(s).

Stu

den

t

pre

sen

ts n

ew

, crea

tiv

e,

an

d o

rigin

al

exam

ple

s.

Co

nta

ins

too

litt

le e

vid

en

ce

of

con

nec

tio

ns

wit

h o

ther

dis

cip

lin

es a

nd

ow

n c

ult

ure

(s)

to j

ust

ify

a

sco

re.

84

85

PA

CE

SE

TT

ER

SP

AN

ISH

¿N

OS

CO

NO

CE

MO

S?

Po

rtfo

lio

Ass

essm

ent:

Ev

alu

ati

ng

Dim

ensi

on

1

DE

MO

NS

TR

AT

ING

KN

OW

LE

DG

E

OF

HIS

PA

NIC

CU

LT

UR

ES

Beg

inn

ing

�

Dev

elo

pin

g

�

Pro

mis

ing

�

Acc

om

pli

shed

�

Ad

va

nce

d

�

No

t en

ou

gh

ev

iden

ce

to j

ud

ge

�

• C

on

sid

er p

erfo

rman

ce o

n a

ll

thre

e as

pec

ts i

ncl

ud

ed i

n t

his

dim

ensi

on

.

Th

e co

llec

tio

n p

rese

nts

sup

erfi

cia

l, v

ag

ue,

an

d

fra

gm

en

tary

ev

iden

ce

of

kn

ow

led

ge o

f

His

pan

ic c

ult

ure

s.

Stu

den

t p

rese

nts

few

an

d i

rrele

van

t

exam

ple

s.

Th

e co

llec

tio

n p

rese

nts

lim

ited

an

d u

nev

en

evid

ence

of

kn

ow

led

ge

of

His

pan

ic c

ult

ure

s,

wit

h f

ew

sp

ecif

ic

exam

ple

s, w

hic

h m

ay

no

t al

way

s b

e re

lev

ant.

Th

e co

llec

tio

n p

rese

nts

ev

iden

ce o

f a

gen

era

l

kn

ow

led

ge

of

His

pan

ic

cult

ure

s, w

ith

so

me

gen

era

l ex

am

ple

s an

d

insi

gh

ts.

Th

e co

llec

tio

n p

rese

nts

am

ple

ev

iden

ce o

f

kn

ow

led

ge

of

His

pan

ic

cult

ure

s, w

ith

a w

ide

ran

ge

of

insi

gh

tfu

l

exam

ple

s.

Th

e co

llec

tio

n p

rese

nts

con

sist

entl

y i

nsi

gh

tfu

l

an

d t

ho

rou

gh

ev

iden

ce

of

kn

ow

led

ge o

f

His

pan

ic c

ult

ure

s, w

ith

var

ied

an

d o

rig

inal

anal

ysi

s an

d e

xam

ple

s.

Th

e co

llec

tio

n

con

tain

s to

o l

ittl

e

evid

ence

of

kn

ow

led

ge o

f

His

pan

ic c

ult

ure

s to

just

ify

a s

core

.

PA

CE

SE

TT

ER

SP

AN

ISH

¿N

OS

CO

NO

CE

MO

S?

DIM

EN

SIO

N 2

: U

SIN

G S

PA

NIS

H T

O C

OM

MU

NIC

AT

E E

FF

EC

TIV

EL

Y

Port

foli

o A

sses

smen

t: E

valu

ati

ng a

spec

ts o

f th

e d

imen

sion

Der

ivin

g m

ean

ing

fro

m p

rin

ted

ma

teri

als

an

d o

ral

dis

cou

rse

Beg

inn

ing

�

Dev

elop

ing

�

Pro

mis

ing

�

Acc

om

pli

shed

�

Ad

van

ced

�

Not

enou

gh

ev

iden

ce t

o

jud

ge

�

• sk

ills

in

clu

ded

: li

sten

ing

and

rea

din

g

• co

mp

reh

ensi

on

an

d i

nte

rpre

tati

on

of

pri

nte

d m

ater

ials

on

div

erse

to

pic

s o

f

var

yin

g l

evel

s o

f co

mp

lex

ity

an

d l

eng

th

• co

mp

reh

ensi

on

an

d i

nte

rpre

tati

on

of

ora

l d

isco

urs

e o

n d

iver

se t

op

ics

of

var

yin

g l

evel

s o

f co

mp

lex

ity

an

d l

eng

th

Stu

den

t p

rese

nts

lim

ited

evid

ence

of

der

ivin

g

mea

nin

g f

rom

pri

nte

d

mat

eria

ls a

nd

ora

l

dis

cou

rse.

Stu

den

t

un

der

sta

nd

s a

few

gen

era

l id

eas,

wit

h

con

tin

uou

s

mis

un

der

stan

din

gs.

Stu

den

t pre

sents

un

even

evid

ence

of

der

ivin

g

mea

nin

g f

rom

pri

nte

d

mat

eria

ls a

nd

ora

l d

isco

urs

e.

Stu

den

t u

nd

erst

an

ds

som

e

main

id

eas

on

fam

ilia

r

top

ics,

wit

h f

req

uen

t

mis

un

der

stan

din

gs.

Stu

den

t pre

sents

som

e

evid

ence

of

der

ivin

g

mea

nin

g f

rom

pri

nte

d

mat

eria

ls a

nd

ora

l d

isco

urs

e.

Stu

den

t u

nd

erst

an

ds

main

idea

s an

d s

om

e d

etail

s on

som

ew

ha

t fa

mil

iar t

op

ics

an

d m

ay m

ak

e so

me

infe

ren

ces

dep

end

ing o

n

co

mp

lex

ity

of

tex

t.

Stu

den

t pre

sents

ap

pro

pri

ate

an

d m

ean

ingfu

l ev

iden

ce o

f

der

ivin

g m

ean

ing

fro

m

pri

nte

d m

ater

ials

an

d o

ral

dis

cours

e. S

tuden

t

un

der

stan

ds

main

id

eas

an

d

most

det

ail

s ev

en o

n

un

fam

ilia

r to

pic

s, a

nd

ca

n m

ak

e i

nfe

ren

ces

ap

pro

pri

ate

to t

he

task

.

Stu

den

t pre

sents

th

oro

ugh

evid

ence

of

der

ivin

g

mea

nin

g f

rom

pri

nte

d

mat

eria

ls a

nd

ora

l

dis

cours

e. S

tuden

t

ex

tra

po

late

s in

form

ati

on

in n

ew w

ays,

in

clu

din

g

som

e so

cial

an

d c

ult

ura

l

refe

ren

ces

an

d a

ffecti

ve

ov

erto

nes.

Conta

ins

too

litt

le

evid

ence

of

der

ivin

g

mea

nin

g

from

pri

nte

d

mat

eria

ls a

nd

ora

l

dis

cours

e to

just

ify a

sco

re.

Exp

ress

ing m

ean

ing i

n o

ral

an

d w

ritt

en

form

Beg

inn

ing

�

Dev

elop

ing

�

Pro

mis

ing

�

Acc

om

pli

shed

�

Ad

van

ced

�

Not

enou

gh

ev

iden

ce t

o

jud

ge

�

• sk

ills

in

clu

ded

: sp

eak

ing a

nd

wri

tin

g •

dev

elo

p a

nd

pre

sen

t te

xts

ora

lly

• d

evel

op

an

d p

rese

nt

wri

tten

tex

ts i

n

fin

al d

raft

s

• d

emo

nst

rate

tec

hn

ical

co

mm

and

of

the

Sp

anis

h l

ang

uag

e (l

ang

uag

e u

sag

e,

stru

ctu

res

and

co

nv

enti

on

s)

Stu

den

t p

rese

nts

lim

ited

an

d f

ra

gm

en

tary

evid

ence

of

exp

ress

ing

mea

nin

g i

n o

ral

and

wri

tten

fo

rm. S

tud

ent

pre

sen

ts a

n i

na

deq

ua

te

tech

nic

al

com

ma

nd

of

the

Sp

an

ish

lan

gu

age

(e.g

., l

ack

of

flu

en

cy

,

iso

late

d s

ente

nce

s,

inco

mp

lete

utt

era

nces)

.

Co

mm

un

ica

tio

n i

mp

eded

by

in

terfe

ren

ce f

ro

m

an

oth

er l

an

gu

ag

e.

Stu

den

t pre

sents

un

even

evid

ence

of

exp

ress

ing

mea

nin

g i

n o

ral

and

wri

tten

form

. S

tuden

t pre

sents

lim

ited

tec

hn

ical

com

man

d

of

the

Sp

an

ish

lan

gu

age

(e.g

., l

imit

ed f

luen

cy,

un

co

nn

ecte

d d

isco

urs

e o

r

sen

ten

ces,

lim

ited

ran

ge

of

vo

cab

ula

ry).

Som

e

inte

rfe

ren

ce f

ro

m a

no

ther

lan

gu

age

may h

am

per

com

mu

nic

ati

on

.

Stu

den

t pre

sents

som

e

evid

ence

of

exp

ress

ing

mea

nin

g i

n o

ral

and

wri

tten

form

. S

tuden

t pre

sents

som

e

un

der

lyin

g

tech

nic

al

co

mm

an

d o

f th

e

Sp

an

ish

lan

gu

age

(e.g

.,

emer

gin

g n

oti

on

of

adeq

uat

e

reg

iste

r, a

deq

uate

ran

ge o

f

vo

cab

ula

ry,

self

-

co

rrecti

on

s).

Min

imal

inte

rfe

ren

ce f

ro

m a

no

ther

lan

gu

age

does

not

ham

per

com

mu

nic

ati

on

.

Stu

den

t pre

sents

ap

pro

pri

ate

an

d m

ean

ingfu

l ev

iden

ce o

f

exp

ress

ing

mea

nin

g i

n o

ral

and w

ritt

en f

orm

. S

tuden

t

pre

sents

pro

per

tec

hn

ical

com

man

d o

f th

e S

pan

ish

lan

gu

age

(e.g

., o

rgan

ized

and

art

icu

late

d m

essa

ges

,

pro

per

use

of

circ

um

locu

tio

n,

at t

imes

cre

ativ

e id

eas,

co

here

nt

use

of

vo

cab

ula

ry

and

mo

stly

ad

equ

ate

use

of

reg

iste

r).

Alm

ost

no

inte

rfe

ren

ce w

ith

com

mu

nic

ati

on

.

Stu

den

t pre

sents

hig

hly

crea

tive,

th

oro

ugh

, an

d

mea

nin

gfu

l ev

iden

ce o

f

exp

ress

ing

mea

nin

g i

n

ora

l an

d w

ritt

en f

orm

.

Stu

den

t pre

sents

cle

ar

tech

nic

al

com

man

d o

f

the

Sp

an

ish

lan

gu

age

(e.g

., w

ell

art

icu

late

d a

nd

org

an

ized

mess

ag

es,

ap

pro

pri

ate

use

of

reg

iste

r, w

ide r

an

ge o

f

vo

cab

ula

ry,

and

a h

igh

lev

el

of

flu

en

cy

).

Occ

asi

on

al

erro

rs d

o n

ot

inte

rfe

re w

ith

com

mu

nic

ati

on

.

Conta

ins

too

litt

le

evid

ence

of

expre

ssin

g

mea

nin

g i

n

ora

l an

d

wri

tten

form

to j

ust

ify a

sco

re.

86

87

PA

CE

SE

TT

ER

SP

AN

ISH

¿N

OS

CO

NO

CE

MO

S?

Po

rtfo

lio

Ass

essm

ent:

Ev

alu

ati

ng

Dim

ensi

on

2

US

ING

SP

AN

ISH

TO

CO

MM

UN

ICA

TE

EF

FE

CT

IVE

LY

Beg

inn

ing

�

Dev

elo

pin

g

�

Pro

mis

ing

�

Acc

om

pli

shed

�

Ad

van

ced

�

Not

enou

gh

evid

ence

to j

ud

ge

�

• C

on

sid

er p

erfo

rman

ce i

n t

he

two

aspec

ts i

ncl

uded

in t

his

dim

ensi

on

.

Th

e co

llec

tio

n p

rese

nts

lim

ited

ev

iden

ce o

f

usi

ng

Sp

anis

h t

o

com

mu

nic

ate

eff

ecti

vely

an

d a

n

inad

eq

uate

tech

nic

al

com

man

d o

f th

e

Sp

anis

h l

ang

uag

e.

Th

e co

llec

tio

n p

rese

nts

un

even

ev

iden

ce o

f

usi

ng

Sp

anis

h t

o

com

mu

nic

ate

eff

ecti

vely

an

d a

lim

ited

tec

hn

ical

com

man

d o

f th

e

Sp

anis

h l

ang

uag

e.

Th

e co

llec

tio

n p

rese

nts

som

e e

vid

en

ce o

f

usi

ng

Sp

anis

h t

o

com

mu

nic

ate

eff

ecti

vely

an

d s

om

e

un

der

lyin

g t

ech

nic

al

com

man

d o

f th

e

Sp

anis

h l

ang

uag

e.

Th

e co

llec

tio

n p

rese

nts

ap

pro

pri

ate

an

d

mea

nin

gfu

l ev

iden

ce

of

usi

ng

Sp

anis

h t

o

com

mu

nic

ate

eff

ecti

vely

an

d p

rop

er

tech

nic

al c

om

man

d o

f

the

Sp

anis

h l

ang

uag

e.

Th

e co

llec

tio

n p

rese

nts

crea

tiv

e,

tho

ro

ug

h

an

d m

ean

ingfu

l

evid

ence

of

usi

ng

Span

ish t

o

com

munic

ate

eff

ecti

vely

an

d c

lear

tech

nic

al c

om

man

d o

f

the

Sp

anis

h l

ang

uag

e.

Th

e co

llec

tio

n

conta

ins

too l

ittl

e

evid

ence

of

usi

ng

Span

ish t

o

com

munic

ate

effe

ctiv

ely

to

ju

stif

y a

sco

re.

APPENDIX D

Dimension 1 Dimension 2

Comments on Artifacts Comments on ArtifactsDiversity of Hispanic Cultures: 1 Deriving Meaning from Printed/Oral Materials: 1

-Would be useful evidence for aspect 3-Only underlined info.; needs to use cover sheet toexplain?-Only a list of foods-No ample inferences or impact-No insights included-Does not mention (Popo)-Was very well done but could have had moreexamples than just one.-Too much evidence; it was overwhelming

-Hard to judge; don't have a copy of the reading (H.Cortés)-Hard to tell what really knew; a lot was copied fromthe text-No evidence for reading-Cassette was marked (Los Hispanos en USA) forlistening but it was really for speaking.-This student must have done more projectsinvolving art/drawing that would have been betterevidence of his abilities/knowledge-Not the best example-Just by underlining hard to know if really reads well(noticias)-Hard to judge this artifact because not part of coursework (Leyenda)* Aspect I should refer to printed language instead ofprinted material

Contributions of Hispanic Figures: 2 Expressing Meaning in Oral & Written Form: 2

-No personal connection made (also applies toaspect 3)-Only mentioned one person-2 Non-Hispanics-Not in Pacesetter; outside source.-Seems to be a contradiction at the end-Same information in both artifacts-Does not show contributions-Not a good choice for aspect II (Popo e Ixtla)-Barely refers to the pictures-General awareness; El Greco-Several students conflate “Hispanic figures” andancient/pre-Colombian civilizations-Only mentioned one person-Piece is of only 1 Hispanic, not several (Blasón)

-Very bad quality of recording-It is not clear if the video included was supposed toaccompany this or not. This info would be helpful.-There wasn’t a prueba in the cassette-Could not find Martin Luther King-Not a reading activity and only had 2 shortquestions to listen to (Cinta-Ballena)-Video missing-Regarding aspect 2: difference between minimalinterference (accomplished) isn’t clear-No oral; good writing-Not enough evidence to judge speaking-Can’t hear tape-Student read his Maya essay and presented it asevidence of speaking; no evidence for listening

Connections W/Other Disciplines/Cultures: 3 -Not a good writing sample

-Fails to make connection-It is not clear what this is or how it is relevant-Linking to own culture lacking as well ascomparing and contrasting-No connection to own culture made-Does not demonstrate knowledge of Hispaniccultures; no personal connection (La Biografia)-Interview shows him interviewing applicant-No connection (Los Hispánicos)-Not a good example (Yo)-Only one sample, not enough evidence-Aspect III has at least two very distinctcomponents; should connection to self /own cultureand to other disciplines both be evident?-Unable to understand recording (Tape w/partner)-No mention of personal connection (Blasón)-No contribution to Hispanic figures (Dos leyendas)-Student is not clear on what connections w/otherdisciplines meanNo personal connections

-I’m finding hamper to be a key word in interpretingaspect 2; first language interference/influence rarelyimpedes communication, nor even hampercommunication, but rather only hamper the ease ofcommunication-An assignment on grammar is not good evidence ofwriting-Question of multiple evidence

88

89

Comments/Suggestions Gen.

-No cover sheet for more than half of the artifactspresented-A lot of the artifacts were irrelevant-Heroes essay is in English-No examples for Aspect-Well organized-In general, this student did not presentworksheets, but only cover sheets with reflections-Student should have chosen only the piecesstrongest in each aspect

-Shows a very positive attitude toward his learningand performance-No speaking samples-In many of the artifacts included as evidence ofreading comp, it was impossible to judge what theinformation included; the same happened with thehistory documents-Need more reading sample and also more listeningsamples-No speaking-Not clear what printed/oral materials meaning

-Cover sheet gave wrong dimension was supposedly derived from for 3 autobio--In many instances, the writing on the coversheet incopied from the writing in the artifact at length andthen is read on tape-This collection contains a lot. 13 artifacts for Dim1 to be evaluated in 27 aspects.-Portfolio is enormous; did not use but at most 8artifacts-Without clear info on coversheet, I would have notknown what the artifacts were proving (The coversheets are very important and should be part of theevaluation)-Student attached several artifacts to one coversheet* Well organized; We should use this sample tosuggest to other teachers and students how to

graphical pieces-No cover sheet-Very well rounded portfolio (02-B-13)-Students indicates 3 artifacts but only 2 found.-Not a listening or reading activity-In analyzing the time used it should be consideredthat this simultaneous processing of the activity,i.e., writing these notes, certainly takes some time.With a reasonable number of submissionscomprising a collection and disregarding technicaldifficulties, it doesn’t take that long to read/score-Do not know which artifact to use for speaking;side A had one speaking and side B had theassessment

send in their artifacts, cover sheets, etc.. (05-E- -No speaking or listening tasks were included07)-This portfolio is clearly organized- Some students have deliberately chosen workfrom throughout the year in order to demonstratetheir progress over time, which isn’t treated in therubrics. As it is weaker works submitted (maybefrom early in the year) for an aspect for whichstronger works also exist may lead to theimpression of limited or uneven overall-Many students either count Ixtla y Popo as“figures” or count the Nahna creators of the legendas “figures” for aspect 2

-Hard to tell if speaking is spoken or read-Maybe there should be a maximum # ofsubmissionsIt is very difficult to cue up tapes for eachrecording and identify which recordingcorresponds to which cover sheet.-Hard to know what to listen to without title orspeaking part.-No true writing sample-Needs to be more specific about what learned-Video with presentation was used to give score forspeaking-Too many artifacts were submitted; limit entries tofive.

200272