Writing2010AssessmentReport,2014-2017PreparedbyDr.JenniferAndrusDepartmentofWritingandRhetoricDirectorofFirst-YearWriting1.IntroductionTheDepartmentofWritingandRhetoriccompletesamajorassessmentofthefirst-yearwritingseries(WRTG1010and2010)everythree-years.Thefirst-yearwritingseriesculminatesinWriting2010(WRTG2010),makingittheidealsiteforassessmentoftheseriesoftwoclasses.Thepresentfirst-yearwritingassessmentreportsonacademicyears2014-2015,2015-2016,2016-2017.Thisassessmenthasfourgoals:

1. DevelopandrefineasetofThresholdCompetenciesfortheFirst-YearWritingseries.2. Assesshowwellstudents’writingmeetsthethresholdcompetenciesattheendofWRTG2010.3. Communicatestudentwritingcompetencytouniversitycommunity,includingfaculty,administrators,

parents,students,andotheruniversitiesintheUtahsystem.4. Revisecurriculumtoimproveandsupportlow-achievingthresholdcompetencies.

Weknowthatregular,programassessmentisnecessarytounderstandwhetherweareteachingstudentswhatwethinkweareteaching(Odell,1981).Programassessmentrequiresthatprogramsmakeexplicitexpectedlearningoutcomes.Orputdifferently,weneedtoassessfirst-yearwriting,inordertodeterminewhetherstudentsarelearningwhatwethinktheyarelearning(Huot,1996).Suchassessmentcreatesandmaintainsconsistencyacrosssections,whichisespeciallyimportantinalargeservicecourselikethisone.Regularassessmentalsomakesprogramgoalsandoutcomestransparentforstakeholdersinthelargeruniversitycommunity,namelystudents,administrators,facultyinotherdepartments,advisors,andthelike.TheassessmentreportedonhereincludestwowaysofmeasuringsuccessinWRTG2010.Thefirstwayofmeasuringsuccessinfirst-yearwritingisanoutcomes-basedassessment,developedtoassessthemultitudeofcompetenciesthatgointosuccessfulacademicwriting.Tothatend,wecompleteageneralreadingandratingofalargesampleofthefinalpaperssubmittedtoaWRTG2010course.Theratingiscompletedbytwoanonymousraters.Usingthresholdcompetencies,orameasurementoftheperformanceinasetofwritingcompetencies,weidentifyspecificstrengthsandweaknessesinstudentwriting,allowingustomakechangestothecurriculumthatwillbolsterstudentlearninginthoseparticularareas.“Thresholdconceptsareconceptscriticalforcontinuedlearningandparticipationinanareaorwithinacommunityofpractice”(Adler-Kassner&Wardle,2015).AccordingtoAdler-Kassner&Wardle(2015),thresholdconceptshavefourcharacteristics:

1. “Learningthemisgenerallytransformative2. “Onceunderstood,theyareoftenirreversibleandthelearnerisunlikelytoforgetthem.3. “Theyareintegrative,demonstratinghowphenomenaarerelated,andhelpinglearnersmake

connections4. “Theytendtoinvolveformsoftroublesomeknowledge[…]thatis‘alien’orcounterintuitive”(Adler-

Kassner&Wardle).Thresholdconcepts,thus,arethosethatarefundamentaltoafieldoflearning,thatchangestudentthinking,andthatwehopewillbecomesecondnature.Theyarebroadandwillhelpstudentsinavarietyofwritingsituationsastheyworkacrossthecurriculum.Thatis,thresholdconceptsextendbeyondasinglewritingorlearningsituation,helpingstudentsadapttothenewwritingandlearningsituationsthattheywillencounterintheUniversitysetting.Thethresholdconceptsanalyzedinthisreportare:sourceuse,synthesis,research,rhetoricalawareness,cohesion,andstyle&mechanics.Thesecondwayweassessstudentsuccessinfirst-yearwritingistakingacloselookatthegraderangeforallstudentswhocompleteWRTG2010.ThesefigurescomefromtheOBIAoffice.Thesenumbersofferapictureofgeneraltrendsinstudentsuccessoverafive-yearperiod.Thisinformationisusefultogetsenseoftheoverall

strengthofacurriculum.Forexample,ifthepercentageofstudentswhoneedtoretaketheclassbecauseoffailureandwithdrawalrateistoohigh,ashiftinthecurriculumiswarranted.2.AssessmentModel:ThresholdCompetenciesInmanycourses,agradeonafinalexamcanindicatehowwellstudentsareachievinginaparticularcourse.Writingrequiresanadditionalassessment,intheformofananonymousratingofthefinalpaperagainstprogramoutcomes,foranumberofreasons.Writinginstructionisuniqueinthatitinvolvesteachingaprocessofwritingandinventionandasetofbehaviors,allofwhicharedifficulttoassess(Haswell,2001;Yancey,1999;Yancey&Huot,1999).Wearelookingforaclusterofwritingcompetenciesandbehaviorsthatcanonlybeassessedbyabstractlyreadingthepapersagainstastandardthatparsesoutandseestherelationshipbetweenthecompetencies.Theintangiblesinwritingassessmentarefurthercompoundedbythefactthatwritingimprovementiscumulative—ithappenswithpracticeovertime.Itcantakeyearsfortheinstructioninafirst-yearwritingcoursetomature(Wardle,2007;Downs&Wardle,2007).AddedtothesecomplicatingfactorsisthefactthatWRTG2010andWRTG1010arerequiredcoursesthataretaughttomorethan1500students,yearly.ProgramlevelwritingassessmentthusrequiresanalyzingwritingfromacrossthesectionsofWRTG2010tosee,ingeneral,whatcompetencieswithwritingstudentshavewhentheycompleteWRTG2010.Thereareanumberofdifferentwaysofassessingwriting,developedinwritingstudies.Holistic,Analytical,andPrimaryTraits,allofwhichscoreonapointscale.Holisticscoringtakesanentirepieceofwriting,determiningasawhole,howwellthedocumentmeetstheoutcomescriteria;analyticscoringlistsoutindividualcriteria,assigningeachcriteriaascore;primarytraitssitsinthemiddle,assessinghowwelladocumentaccomplishesanumberofwritingtraits,whileconsideringtherelationshipbetweenthosetraits(Lloyd-Jones,1977).Whileholisticscoringaccountsforthewaysinwhichdifferentelementsinapieceofwritingworktogether,rhetorically,itdoesn’tprovideafineenoughaccountofwhatisworkingandwhatisnotworkinginapiece.Whereanalyticscoringgivesveryspecificfeedback,itdoesn’tprovideasenseofhowwellthecriteriaworktogether,rhetoricallyspeaking(Odell&Cooper,1980).Primarytraitsassessmenttakesaccountofboththebigpictureandthespecifics.Thefirst-yearwritingassessmentusesaprimarytraitsmodelcombinedwithathresholdcompetencymodel,inordertogetasenseofthebigpicturethatisdevelopedbylookingcloselyatspecificcompetencies.Primarytraitsallowustoidentifykeytraitsthatcombinecriteriasothateachtraitgetsascore,givingusinsightintothestudents’performanceonanumberofrhetoricallysalientelementsofacomposition.Forexample,studentsmaybestrongonsentencestructurebutlesssoonsynthesis.Primarytraitsassessmentallowsustoseethiswithoutlosingtheconnectionsbetweenelementsofatextthatleadtoitsultimatesuccess.Primarytraitsassessmentworkswellincombinationwithathresholdcompetencymodel,whichalsotakesanaccountingofspecificcompetenciesthataredesiredinapieceofwriting.Iborrowtheconceptofthresholdcompetenciesfrommanagementresearch,whereathresholdcompetencyisthatabilitythat“apersonrequiresinordertobeminimallycompetent”atataskorajob(Skulmoski&Hartman,2001,p.61).Irevisetheconceptheretoexplainanddescribestudentwritingandbehaviors.Forthepurposesofthisreport,acompetencyisarootabilitythatthosewhohavecompletedfirst-yearwritingshouldmeetataparticularthresholdinordertobesaidtobeproficientinthatcompetencyortrait.TheDept.ofWritingandRhetorichasdevelopedasetofsixthresholdcompetencies.TheThresholdCompetenciesforAcademicArgumentationthatweteachinWRTG2010are:1. SourceUse

o Sourcesusedasevidence,o Sourcesusedarevaried,o Sourcessupportclaimsandsub-claims,o Sourcesareappropriatelyattributed

2. Synthesiso Textcombinessources,

o Sourcesareintegratedinanumberofways,includingdirectquotationandparaphrase,o Connectionsaredrawnacrosssources,o Textcreatesanetworkofsourcesrelevanttopresentargument

3. RhetoricalAwareness

o Thesisindicatestherhetoricalpurposeofthetext,o Textshowsawarenessofandaccommodationtoaudience,o Textshowsawarenessofwritingcontext,o Wordchoiceandpaperfeaturesareappropriateforgenre,o Argumentissituatedinanacademicconversation,o Argumentcontributestoanacademicconversation

4. ResearchSkillso Sourcesarequalityo Researchfromavarietyofmediaandtypeofsources

5. TextualCohesiono Paperstructureisobvious,o Ideasaredevelopedcohesively,o Argumentsarewelldevelopedacrossthepaper,o Transitionalsentencesandphrasesareused,o Mainclaimandsupportingclaimsprogresslogically

6. Style&Mechanicso Academictone,o Conventionsofstandard,written,editedEnglish,o Spelling,o Punctuation,o Citationstyle

ThesethresholdcompetenciesarethecourseoutcomesthatarerequiredonthesyllabusforeverysectionofWRTG2010taughtattheUofU.WhiletherearemanywaysofteachingWRTG2010,thegeneralcourseoutcomesrequireallsectionstoteachtowardthesamegoals.Thesethresholdcompetenciesfunctionaslearningoutcomes—expectedcompetencies,abilities,skills,andbehaviorsthatthosewhocompleteWRTG2010shouldmeet,atleastataminimalthreshold(2onthe4pointscale).3.Method3.1AssessmentToolDevelopmentTheassessmenttoolsdescribedhereweredevelopedoverathree-yearperiod.(ThefullassessmentrubricsareinAppendix1.)Afterusinganinitialprimarytraitsrubricforthefirstratingsession,itwasrevisedtostreamlineitfortheratersandtogetamorenuancedaccountofwritingcompetenciesofstudentsattheendofthesemester.Bothrubricsusedthesame4-pointscale,describedinmoredetailbelow.ThefirstversiondevelopedandusedinFall2014issimilartotheformatofotherfirst-yearwritingrubrics.The4-pointscaleforthefirstrubricwas:4:Excellent,3:AboveAverage,2:Adequate,1:NeedsImprovement.Theprimarytraitsassessedinthe2014assessmentwere:

1. SourceUse:Keywords—Academicargument,thesis,synthesis,research,sourceuse,sourcequality,sourceattribution,evidencesupportsclaims,evidencedrawnfromsources,audienceawareness

2. Cohesion&Structure:Keywords—Paperstructure,ideacohesion,transitionalsentencesandphrases,logic&organization,claimstructure

3. Style&Mechanics:Keywords—Academictone,conventionsofstandard,written,editedEnglish,spelling,punctuation,citationstyle

Thisrubrictakesarhetoricalandacademicargumentationapproachtofirst-yearcomposition,focusingontheresearch,theuseofsources,evidencebasedreasoning,andformalfeaturesofacademicwritingconventionsandarguments.Whatwefoundinthefirstratingsessionisthatthescoresforthefirsttrait,sourceuse,werelowerthanexpected.Thisledmetowonderwhataboutsourceuseinparticularwerestudentsnotgetting.TherubricfortheinitialratingsessionwasrevisedinSpring2015.ThisrubricreconceivesoftheassessmentintermsofThresholdCompetencies,describeabove.Thatmeansthatwehavedeterminedthecompetenciesthatstudentsneedtomeetinordertoadequatelysucceedinacademicwritinginothercoursesintheuniversity.The4-pointscaleforthesecondratingsessionwasrevisedtothefollowing:4:Excellent,3:Adequate,2:Minimal,1:Infrequent.Ideally,thegroupofpapersratedshouldmeetthecompetencybetweenAdequate(3)andMinimal(2)foreachtraitinordertohitthesocalled,threshold,forthattraitforWRTG2010.Inordertobetterunderstandthewaysstudentswereusingsources,sourceusebeingfundamentalforacademicwriting,IrefinedtheSourceUsetraitfromtheinitialrubric,makingitintofourtraits:SourceUse,Synthesis,RhetoricalAwareness,andResearchSkills,withCohesionandStyleremainingstable.Thesecondrubric,then,measuresthefollowingsixcompetencies:1. SourceUse:Sourcesusedasevidence,Sourcesusedarevaried,Sourcessupportclaimsandsub-claims,

Sourcesareappropriatelyattributed2. Synthesis:Textcombinessources,Sourcesareintegratedinanumberofways,includingdirectquotationand

paraphrase,Connectionsaredrawnacrosssources,Textcreatesanetworkofsourcesrelevanttopresentargument

3. RhetoricalAwareness:Thesisindicatestherhetoricalpurposeofthetext,Textshowsawarenessofand

accommodationtoaudience,Textshowsawarenessofwritingcontext,Wordchoiceandpaperfeaturesareappropriateforgenre,Argumentissituatedinanacademicconversation,

4. ResearchSkills:Sourcesmeetacademicstandardsforquality,Researchfromavarietyofmediaandtypeof

sources5. TextualCohesion:Paperstructureisobvious,Ideasaredevelopedcohesively,Argumentsarewelldeveloped

acrossthepaper,Transitionalsentencesandphrasesareused,Mainclaimandsupportingclaimsprogresslogically

6. Style&Mechanics:Academictone,Conventionsofstandard,written,editedEnglish,Spelling,Punctuation,

CitationstyleThisrubricgivesusabetterpictureastowherethebreakdowninsourceuseishappening.Thiscloserlookallowsforarevisionofthecurriculum,traininstructors,andincreasesupportmeasuresthatbettermeetstheneedsofstudents.3.2AssessmentProcedureFollowingwritingassessmentbestpractices,weassessedacross-sectionofpapersfromacrosssectionsofWRTG2010attheendofFallandSpringsemesters.Instructors1wereaskedtosubmitarangeofsixpapers—twotop,twomiddle,twobottom—fromeverysectionofthecoursethattheytaught.Theinitialpaperselectionis1Instructorcompliancewithassessmentprotocolshasbeenanissue.

subjective,butitensuresthatwegetarangeofpapersfromtheperspectiveofthosewhoteachthecourse,leadingtoamorecompletepictureoftherangeofaccomplishmentwithwritingthanarandomsamplewould.ThepapersweresubmittedtotheDept.secretarywhonumberedthepapersandputthemintotwospreadsheets.Onehastheteachernameandthenumbersofthepapersfromthatteacher.Theotherjusthasthepapernumberandthescoresforeachtraitforeachrater.Thesecretaryprepareshardcopiesofthepapersforrating,includingaratingsheetattachedatthebackofeachpaper.Thepapersareratedinagroupratingsession.Beforetheratingsession,Ireadthroughthegroupofpapersandselectthenormingpapers.TheassistantWPAandIreadthroughthepapersandscorethemoneachtrait.Theratingsessionbeginswithanormingsession,inwhichalloftheratersreadandscoretheselectedpapers,andwehaveagroupdiscussiontopushtheratersintoasimilarwayofreadingandratingthepapers.Isolicitedratersfromthewritingprogramfacultyandinstructorsandpaidthemhourlyfortheirparticipationintheratingsession.Theratingsessionfollowedtheseprocedures.

1. CompetencyThresholdreview:IhandoutahardcopyofthePrimaryTraitsRubrictoeachrater,andwereviewwhattheymeananddiscussexamplesofthetrait.

2. Norming:Wereadanddiscusseachnormingpapersothatweallagreeontherequirementsforaparticularscoreoneachpaper.Thisensuresamoreconsistentandreliablescore.

3. Rating:Eachpaperisreadandratedforeachoftheprimarytraitsbytworeaders.Ifthereaders’scoresdifferbymorethanonepoint(a2anda4,forexample)onanytrait,thepaperisreadagainbyathirdrater.

4. Re-normingasneeded:Mid-rating,Ire-normbasedonquestionsaskedbyratersandmyspotchecksofscoring,inordertokeepratersscoringthesametraitsinthesameway.

Whentheratingsessionisover,thepapersaregivenbacktothesecretarywhoinputsthescores.Thescoringspreadsheetisthensenttomeforanalysis.4.FindingsandAnalysisIntheinitialrating(Fall2014),wefoundthatstudents’performanceisthelowestonfeaturesofacademicargumentation,especiallysynthesisandsourceuse.Inthesecondrating(Spring2015),wefoundthatsourceuseandsynthesishadthelowestnumbers,whilerhetoricalawarenessandresearchskillswererelativelyhigh.Idescribethefindingsindetailbelow.

4.1FALL2014Fall2014,werated147papers.TheOverallAveragewasdeterminedbyaveragingallofthescoresgivenacrossthetraitsbybothraters.Studentsperformquitedifferentlyinthedifferenttraits.Wheretheydowellwithstyle,asindicatedbythefairlyevenMinimal(1.5-2)andThreshold(2.5-3)figures,Cohesionhasfarpapersinthe1.5-2rangethanthe2.5-3range.Argumentalsohasmorepapersintheminimaltolowrange

1:10%1.5:24%2:31%2.5-3:24%3.5-4:1%

Figure1:OverallAverage,Fall2014Figure1showstheoverallaverageforpapersassessedinFall2014.Whatweseehereisthatthemoststudents,31%,aremeetingtheexpectationsofthecourseintheminimalrange.25%areatorabovethethresholdrangeand31%fallbelowtheminimalrange.Thequestion,then,iswhichfeaturesofwritingaredrivingdownthesescores.Thefollowingchartsandfiguresgiveafullerpictureofthatbreakdown.

1:16%1.5:14%2:29%2.5-3:30%3.5-4:11%

Figure2:ArgumentArgumentationinoneofthekeycompetenciestaughtinfirst-yearwriting.Itisalsoacompetencymadeupofanumberofsmallerskills,behaviors,andabilities.IntheFall2014assessment,thenumbersfortheArgument

0

5

10

15

20

25

30

35

1 1.5 2 2.5-3 3.5-4

Overall,Fall2014

05101520253035

ArgumentAssessment,2014

competencyaregood,with41%atorabovethecompetencythreshold,29%minimallymeetingexpectations,and30%fallingshortofthethresholdcompetency.Morethanhalfofthepapers,then,onlyminimallymetordidnotmeetthethresholdforthistrait.

1:9%1.5:21%2:29%2.5-3:30%3.5-4:11%

Figure3:CohesionThestoryincohesionlooksverysimilartotheoneinargument,with41%meetingorexceedingthethresholdcompetencyrange,29%minimallymeetingexpectations,and30%belowthethreshold.Inthiscase,morepapersareclosetotheminimalrange—21%--thanwithargument.Thesenumbersmirrorthoseofargument,indicatingthatpaperswithbetterargumentstructuresaremorecohesiveandviceversa.

1:6%1.5:12%2:29%2.5-3:39%3.5-4:14%

Figure4:Style&MechanicsStyle&Mechanicsiswherepapersdothebestinthisassessment.53%ofthepapersmetorexceededthecompetencythresholdlevel,with29%minimallymeetingthethreshold.Only18%fellbelowthethreshold.

0

10

20

30

40

RatingScore

1 1.5 2 2.5-3 3.5-4

CohesionAssessment,2014

051015202530354045

RatingScore

1 1.5 2 2.5-3 3.5-4

Style&Mechanics,2014

0

10

20

30

40

50

60

OverallAverage

1 1.5 2 2.5-3

0

10

20

30

40

50

60

OverallAverage

1 1.5 2 2.5-3 3.5-4.0

Figure5:Argument,Cohesion,Style&Mechanics,2014Takentogether,thisassessmentindicatesthatstudentsaremeetingexpectationsforlower-orderconcernsofstyleandmechanics,butstrugglingwiththemoredifficult,higherorderconcernslikeargumentandcohesion.Inordertobetterunderstandthistrend,theassessmenttoolwasrevisedtobettercapturewhatisgoingonwithissuesofargumentandcohesion,usingafiner-grainedassessment.4.2SPRING2015AssessmentRecallthatweuseda4-pointscoringscale,andthateachpaperwasscoredbytworaters,whoeachgaveindependentscoresforeachofthetraitsmeasuredinthatassessmenttool.Thethresholdforadequatecompetencyisa3andatwoindicatesminimalcompetency.Thus,anaverageof1showsthatthepaperdidnotmeetcompetencyexpectationsforthattrait,whilea4showsthatitexceededcompetencyexpectations.Ifapaperwithanaveragescoreof1.5receiveda1byonerateranda2byanotherforthattrait.Similarlya2.5receivedatleaston3anda3.5atleastonescoreof4.Inordertoseehowwellthepapersmeetcompetencythresholds,Ihavegroupedtheaveragedscoresthisway:

• Below:averagedscoreof1,thepaperbelowthethresholdforthatcompetency/trait• Minimal:averagedscorerangeof1.5-2,thepaperminimally/borderlinemeetsthresholdforthat

competency/trait• CompetencyThreshold:averagedscorerangeof2.5-3,thepapermeetsthresholdforthat

competency/trait• Exceeds:averagedscoreof3.5-4,thepaperexceedsthresholdforthatcompetency/trait

0

10

20

30

40

50

1 1.5 2 2.5-3 3.5-4

Overall,2014

Argument Cohesion Style

Overall1:7%1.5:18%2:32%2.5-3:32%3.5-4:11%MinimalCompetency:32%Competency+:43%

Figure6:OverallAverageScores,2015Asthegraphaboveshows,32%ofthepapersratedminimallymeetthecompetencythresholdsforWRTG2010,withanother18%borderlinecompetent.43%ofthepapersratedmeetorexceedthecompetencythreshold.(NOTE:Overallscoresareaveragesofalltraitsgivenbybothraters.)Wewouldlikemorestudentsinthe2.5-3range.

Thoughatleast37%ofstudentsareminimallymeetingthethresholdfortheSourceUsecompetency,manyofthoseareborderline(1.5)and22%arenotmeetingthethreshold.41%areatorabovethethresholdforthesourceusecompetency.TheproblemsmeetingthethresholdareevenmorepronouncedfortheSynthesiscompetency,with36%minimallymeetingthestandardand32%atorabovethethresholdstandard.Withthiscompetencywealsosee32%belowevenminimalcompetency,themostintheentireassessment.Synthesisisthusidentifiedasthecompetencythatneedstomostcurricularsupport.Thismeansthatjustlessthanhalfofstudentsarenotsufficientlyattributingsources,usingsourcestosupportclaims,noraretheyusingavarietyofsources,orcombiningandsynthesizingsourcestoshowthereaderwheretheargumentfitsinwithotherresearchonthesametopic.

0

10

20

30

40

50

60

70

SourceUse

1 1.5-2 2.5-3 3.5-4.0 0

10

20

30

40

50

60

70

Synthesis 1 1.5-2 2.5-3.0 3.5-4

Figure7:SourceUse1:22%1.5-2:37%(MinComp:37%)2.5-3:31%3.5-4:10%(Comp+:41%)

Figure8:Synthesis1:32%1.5-2:36%(MinComp:36%)2.5-3:24%3.5-4:8%(Comp+:32%)

StudentsarestrongerintheResearchSkillscompetency.44%arefindingvariedandhighqualityresources,andabout34%aremakingprogressindoingsecondaryresearch.Thisisinteresting,giventhemiddlingsourceusescores.Thismeansthatstudentsarefindingsourcesbutstrugglingtofigureouthowtoread,makesenseof,andmakeuseofthesources.Weknowthatsynthesiscomeswithcomprehensionandownershipoverthematerial.Studentscandotheresearch,buttheyrunintoproblemsusingtheinformationandsourcesthattheyfind.AnothercompetencyinwhichstudentsarebyandlargemeetingthecompetencythresholdisRhetoricalAwareness.45%areatorabovethe2.5competencythresholdandanother37%areminimallymeetingthestandard.Thismeansthatstudents’proseusesfeaturesofacademicargument,itorientedtoanaudience,usingappropriategenericfeaturesandtone.

Intermsofoverallorganization,textualcohesion,andmechanics,therearemixedresults.82%ofstudentsmeetthecompetencythreshold,buthalfofthat,41%,areintheminimalorborderlinerange.Thismeansthatstudentwritingisusingsometechniquesoftextualcohesionandstructure—transitionsentencesandphrases,overalllogicalstructures,connectionsbetweenclaimsandsubclaims—butmanystudentsaren’tusingthemconsistently.

0

10

20

30

40

50

60

70

ResearchSkills

1 1.5-2 2.5-3 3.5-4 0

10

20

30

40

50

60

70

RhetoricalAwareness

1 1.5-2 2.5-3 3.5-4

Figure9:ResearchSkills1:20%1.5-2:34%(MinComp:34%)2.5-3:34%3.5-4:12%(Comp+:44%)

Figure10:RhetoricalAwareness1:18%1.5-2:37%(MinComp:37%)2.5-3:37%3.5-4:8%(Comp+:45%)

0

10

20

30

40

50

60

70

80

Style&Mechanics 1 1.5-2 2.5-3 3.5-4

0

10

20

30

40

50

60

70

80

TextualCohesion

1 1.5-2 2.5-3 3.5-4

Figure11:TextualCohesion1:18%1.5-2:41%(MinComp:41%)2.5-3:31%3.5-4:10%(Comp+:41%)

Figure12:Style&Mechanics1:8%1.5-2:43%(MinComp:43%)2.5-3:43%3.5-4:6%(Comp+:49%)

TheStyleandMechanicscompetencyhasthehighestcompetencyscores,with49%meetingorexceedingthethresholdcompetencyandonly8%receivingascoreof1.Thisshowsusthatatthesentencelevel,papersarecomingtogether,withclausestructuresandpunctuationthataremostlycorrect,accordingtotheexpectationsofstandard,written,editedEnglish.Thisalsomeansthatstudentsarelearningtouseacademiccitationstyles.5.OBIAData:GradedistributionTheassessmentandrelatedcurriculumrevisionhavehadsomepositiveeffectsinWRTG2010,inparticularintheareaofoverallgradeimprovement.Asthefollowingchartshows,theDWEgradeshavecomedownsignificantly,withmorestudentspassingtheclass.Passingtheclassreducesthenumberofstudentswhohavetoretaketheclass,reducingtheso-calledbottleneckthatwewereseeinginWRTG2010andWRTG1010.

Figure13:Gradedistribution,Fall2011-Summer2016Wewillknowmoreaboutimprovementsinstudentabilitywhenwecompletetheassessmentof2017-2018attheendofSpring2018.

0

5

10

15

20

25

30

35

40

45

A A- B B- C C- D- D+ E EU I V W

GradeDistributioninWRTG2010

Fall2011 Spring2012 Summer2012 Fall2012 Spring2013

Summer2013 Fall2013 Spring2014 Summer2014 Fall2014

Spring2015 Summer2015 Fall2015 Spring2016 Summer2016

ErangepercentagesFall2011 6%Spring2012 10%Summer2012 11%Fall2012 11%Spring2013 10%Summer2013 12%Fall2013 6%Spring2014 8%Summer2013 11%Fall2014 7%Spring2015 9%Summer2015 8%Fall2015 6%Spring2016 7%Summer2016 6%

Figure14:D,E,Wgradedistribution,Fall2011-Summer2016NoticetheErangegrades.In2012and2013,theywerebetween13%and10%.Nowtheyaredownat6%,meaningthatthevastmajorityofstudentswhotakeWRTG2010passit.Similarly,theDrangegradeshavealsoimproved,movingfrom2%and3%downto1.5%.ForWRTG1010,thestoryissimilar.WeseeadropinEgradesoverthelast5years,withfairlyflatD-rangegrades.

Figure15:Gradedistribution,Fall2011-Spring2016

0

5

10

15

20

25

30

A A- B+ B B- C+ C C- D+ D D- E EU I V W

2011-2012

2012-2013

2013-2014

2014-2015

2015-2016

0

2

4

6

8

10

12

14

DWEGradesinWRTG2010

Drange Erange W

Erangepercentages2011-2012:13%2012-2013:10%2013-2014:10%2014-2015:9%2015-2016:9%

Figure16:D,E,Wgradedistribution,Fall2011-Spring2016Asyoucanseefromfigure16,thegradeshaveimproved,withtheEgradefalling4percentagepoints—from13%to9%.FewerstudentsaregettingcaughtrepeatingWRTG1010,whichisgoodforthestudentandthewritingprogramalike.6.CurriculumRevisionandTeacherTraining6.1CurriculumTheassessmentofWRTG2010showedthatstudentsareunderperformingintheareasthatweknowarefundamentalforacademicwriting—usingsourcesandsynthesizingthem.Thecurrentcurriculumhasbeendevelopedwiththosetopicsinmind.Writing2010isdesignedtoprovidestudentswiththefoundationalpracticesandcapabilitiesneededtoreadandwriteacademicargumentsproficiently.Academicwritingisn’tasoloendeavor,inwhichtheauthorsitsandcomesupwithideasbyhimorherself.Instead,WRTG2010isframedtothinkaboutacademicwritingasjoiningaconversation,embodiedinexistingresearch.Studentsaretaughtthroughaseriesofassignmentstoread,comprehend,andsynthesizeabodyofresearch.Studentsaretaughtthatstrongacademicwritingisfoundedonstrongresearchandreadingskills,whichallowstudentsto:• readtheargumentsandideasaboutatopicthathavealreadybeenwrit,• demonstrateknowledgeofthepriorresearchthroughaccurateandfairsummary,• synthesizepriorresearch,argumentsaboutandapproachestothetopic,• contributeyourownargumenttotheconversation,• situateyourargumentintheexistingbodyofresearchonthetopic,and• supportyourargumentusingevidence.Theassignmentsinthecoursewalkstudentsthroughthisprocesstwice,oncewithagroupofreadingsonatopicthattheteacherselects,andthenoncewithaddedresearchskillswithagroupofreadingsonatopicthatthestudentthemselvesselects.Thefollowingistheassignmentseriesforthecourse:

• Synthesis1:ConversationMap:Forsynthesis1,studentsreadanumberofarticlesaboutatopicasaclass.Studentsdiscusstheargumentativestrategiesusedandidentifythewaysthatthearticlesareinconversationwithoneanother,sometimesmoreexplicitlythanothers.Studentscreateasynthetic“map”ofthescholarlyconversation,bycreatingavisualrepresentationtheresearchconversation.Theyalsowriteathree-pagepaperinwhichtheysynthesizetheargumentsmadebythevariousarticlesinthetopical,scholarlyconversation.

• AnnotatedBibliography:Ingroups,studentsselectatopicandworktogethertoresearchit.Asagroup,theycollect21articles,books,visuals,etc.onthetopic(sevensourcespergroupmember)andmakea

0

2

4

6

8

10

12

14

D+ D D- E EU W

2011-2012

2012-2013

2013-2014

2014-2015

2015-2016

bibliographyusingAPAstyle.Finally,theyannotateeachentrywithathree+sentencesummaryoftheargumentthearticlemakes.

• Synthesis2:LiteratureReview:AftersubmittingtheAnnotatedBibliographyasagroup,studentsindividuallywritealiteraturereviewofthesourcescollectedasagroup.Aliterature(lit)reviewisacommonandimportantacademicgenre,inwhichstudentslearntoreviewandsynthesizeresearchonthetopic.Synthesis2willthenexplain,describe,anddefinethemultiplespositionstotakewithintheparticularscholarlyconversationaswellasthestakesassociatedwitheachposition.

• ArgumentCampaigno Academic,ResearchedArgument:Thecontributionpaperwillworkdirectlyoutofsynthesis2to

contributeanargumenttotheongoingconversationresearchedingroups.o Visual,Non-AcademicArgument:Inadditiontothecontributionpaper,studentscreateone

companionpiecetoaccompanyorotherwisepresenttheargumentmadeintheformalacademicpaper:adirectmail,postcard,pamphlet,postcard,whitepaper,etc.

• Presentation:Afterwritingandcontributingasingle-authoredacademicpaperandacompanioncomposition,studentsjoin-upwiththeirresearchgroupsagaintowriteandgivea10-minutepresentationonthetopic.

• Portfolio:Attheendofthesemester,studentscreateaPortfolioinwhichtheycollectandpresentalloftheworkcompletedinthesemester.Theyalsoreviseonepreviouslygradedessayandwriteaportfolioanalysisthatreflectsonandexplainswhattheylearnedoverthesemester.

Asyoucansee,synthesisandsourceuseareforegroundedandtaughtinavarietyofwaysacrossthesemesterInadditiontorevisingthecurriculum,Ihavedevelopedanacceleratedhalf-semesterversionofbothWRTG1010andWRTG2010.Thisversionofthecoursehasbeenverysuccessful,withahighpassrateandcorrelatively,verylowDWEgrades.Thisversionoftherequiredwritingcoursesallowwilystudentstocompletetheirwritingrequirementsinonesemesterratherthantwosemesters.6.2TeachertrainingThesefindingshaveledtoinnovationsinteachertraining.Firstofall,participatingintheassessment,asmanyoftheteachersdo,helpsthemunderstandnotonlythestakesofwritinginstruction,butalsothelargergoalsandstandardsfortheprogram.Havingreadandratedmorethan20papers,instructorsleavetheassessmentwithabettersenseofthefocusandrequirementsofWRTG2010.UsingtheassessmentrubricandworkingonanonymouspapersreallydriveshomethemajorconceptsandbehaviorsthatdriveWRTG2010,helpinginstructorsunderstandwhattheyshouldbefocusingonintheirclassrooms.Inadditiontothelessonslearneddirectlyfromtheassessment,thesefindingshaverequiredmetoadjustthecurriculumforthenewinstructortraining.Nowinadditiontothetypicaldiscussionsofpedagogy,wehavemoreandtargetedinstructioninteachingsourceuseandsynthesis,includingreadingarticlesonsourceuse,synthesis,bestpedagogicalpractices,andgradenorming.Newteachersdeveloparobusttoolkitforteachingandassessingsourceuseandsynthesis,followingalongwiththescaffoldingthatisbuiltintoWRTG2010itself.Instructorsarethemselvesaskedtoparticipateinwritingassignmentsthatrequiresynthesisandstrongsourceuse.Astheyexperiencetheprocessesthemselves,theyareaskedtoreflectontheprocess,thusinfluencinghowtheyteachsynthesisintheirclasses.7.ImplicationsandConclusionsThepresentreporthasshownthatWRTG2010issuccessfullyteachingstudentsattheUniversityofUtah.Thefailurerate(Egrade)inthecoursehasdroppedsignificantlyoverthelastfiveyears,from10%and11%in2012to6%in2016.In2016,76%ofthestudentsenrolledinWRTG2010completedthecoursewithagradesufficienttofulfilltheW2writingrequirement,upfrom63%in2012.

StudentswhosuccessfullycompleteWRTG2010arelearningdifficultandimportantacademicwritingcompetencies,namelySourceUse,Synthesis,RhetoricalAwareness,ResearchSkills,TextualCohesion,andStyle&Mechanics.Ofthose,theyexcelinstyleandmechanics(49%atorabovethreshold),researchskills(44%atorabovethreshold),andrhetoricalawareness(45%atorabovethreshold),whiletheystrugglemorewithsourceuse(41%atorabovethreshold),synthesis(32%atorabovethreshold),andtextualcohesion(41%atorabovethreshold).Thecurriculumandteachertraininghavebeenrevisedtotargetthecompetenciesofsynthesisandsourceuse.Thesecompetenciesarescaffoldedthroughoutthecourse,beginningwiththeveryfirstweek,andbuildingacrossthecourseofthesemester.Thisfinalsectionofthereportwilldiscusssomeoftheimplicationsforupper-divisionwritingcourses.Itwillalsolookforwardtoconsidertheworkofwritingassessmentinbroadertermsandlookforwardtofutureassessments.7.1UpperdivisioncoursesThebehaviorsandcompetenciestaughtinWRGT2010areintendedtopreparestudentsforupper-divisionwritingcourses.Tobetterunderstandwhatthatmeansinrealtermsonthiscampus,aneedsassessmentwascompleted.Iinterviewedthirteenfacultymembersfromacrosscampusandavarietyofdisciplineswhoregularlyteachclassesthatfulfilltheupper-divisionwritingrequirementattheUofU.Mystudyaimedatgettingasenseof1)whatfacultyvaluedaboutwriting,2)whattheywantedtoseeinwrittenassignments3)whatkindsofwritingtheyassigned,and4)wherestudentsranintotroublesuccessfullycompletingthoseassignments.Thefollowingisalistofkeywordsthatcameoutofdiscussionsinthoseinterviews.Thenumberindicateshowmanytimesthetopicorconceptcameupinthoseinterviews.7discussionsArgument

5discussionsEvidencedata Sourceusequotationcitation

4discussionsFlowconnection organization structure

3discussionspolished revised Logicalprogressionofideas

2discussionsanalysis mechanics Synthesis Reading Paragraphing clear/clarityresearch

Themostimportantvalueforfacultyteachingupper-divisionwriting,then,isargument,withevidence,data,sourceuse,quotation,andcitationfollowingclosely.TheseareallsourceuseskillsthataretaughtinWRTG2010.Thenextgroupofimportantconceptsareflowandorganization,orcohesionissues.RecallthatSourceUseandCohesionhavehistoricallybeentwoofthemostdifficultcompetenciesforstudentstomaster.Mentionedlessoftenareissueslikeclarityandmechanics,whichstudentsaremorelikelytogainmasteryover.Thefollowingisthesamesetofwritingconcepts,organizedaccordingtotheWRTG2010writingcompetencies.

SourceUseevidenceanddatasourceuse,quotation,citationanalysisreading

Synthesissynthesis

RhetoricalAwarenessargumentrevision

ResearchSkillsresearch

TextualCohesionflowandconnection organized structurelogicalprogressionofideasparagraphing

Style&Mechanicspolishedmechanicsclear/clarity

WhatthisshowsisthatWRTG2010isteachingtheconceptsthatarecoretomanyoftheupper-divisioncoursestaughtacrosscampus.Takenwiththeassessmentdatalaidoutabove,thismeansthatstudentsaregettingastartonmasteryoftheseconceptsinWRTG2010,butthattheworkofinstructioninthesecompetencieswillneedtobemaintainedinupper-divisionwritingcourses.Thatistosay,becausetheseconceptsaresodifficult,upper-divisionprofessorswillneedtotakesometimeinclassforexplicitandongoinginstructioninthesecompetenciesinorderforstudentstobesuccessful.Putanotherway,assigningwritingjustisnotenough.Therewillneedtobeovertinstructionintheseissuesinorderforstudentstoreachthethresholdlevelforthesecompetencies.Learningtowriteisamulti-semesterandmulti-courseendeavor.Learningtowritetakesplaceovertimeandoveranumberofdifferentwritingsituations(Haswell,2000;Wardle,2007).7.2Theworkoffirst-yearwriting:NewplacesforassessmentInfirst-yearwritingcourses,whatweteachstudentsishowtotakeabodyofknowledge,readit,comprehendit,synthesizeit,andfinallytocomposeit.Ourworkthen,asAdler-Kassner&Wardle(2015)putit,then,istoteachstudentstocomposeknowledgeinwaysthatcomportwiththebodyofknowledge,thereader,andthecontext,amongotherthings.Thisisatallorderandeasiersaidthandone.Thisreporthasexplainedanddescribedthewaysthattextsdemonstratefluencyinasetofsixthresholdcompetencies.Here,Iwillbrieflycommentontheless/in-tangiblebehaviors,skills,andconceptsthatareinmanywaystherealworkofWRTG2010,butthataredifficultifnotimpossibletorepresentwithasinglepieceofwriting.Thatis,thereisawholehostofrhetoricalknowledgethatstudentsgetinWRTG2010notrepresentedintheaboveassessment.Thesebehaviorsandknowledgesaredifficulttoassessbutareparticularlymeaningfulinthedevelopmentofawriter,particularlyimportantforacademicwriting.Thefollowinglistoflesstangibleconceptscapturesjustafewofthebehaviors,skills,andcompetenciesthatstudentsgaininWRTG2010:

• “Writingisasocio-rhetoricalactivity”(Roozen)Writingisanactivitythattakesplacebetweenpeople.Writersarealwaysattemptingtoaccomplishaparticulartask,inanoftenill-definedcontext,foraparticularaudience.Thuswritingisessentiallyrhetorical,copingwithchancedifficultiesandsocialrelationships.

• “Writingisaknowledge-makingactivity”(Estrem)Writingisundertakentodosomething;itisanactivity.Veryoftentheactivityistobringtogetherabodyofknowledgeandpresentitcohesivelyforareader.Studentsareaskedtomakesenseofnewknowledgeinandthroughwriting.

• “Writingaddresses,invokes,and/orcreatesaudiences”(Lunsford)Writingisforanaudience,buttheconceptofaudienceisunstable.Anytextwillbothaddressanaudience,butitwillalsoworktore/constructthataudience,basedontheinformationpresentedandthecontextsinwhichthewritingandreadingtakeplace.Everypieceofwritinginvokestheaudiencethattheauthorimagines,analyzes,andcopeswith.

• “Writingexpressesandsharesmeaningtobereconstructedbythereader”(Bazerman)Writingiswrittentoberead.Writingisforareader,whowilldothehardworkofrecreatingthemeaningthathasbeencarefullyconstructedbythewriter.Copingwiththereaderisafeatureofeverytext.Meaningisnegotiatedbetweenthewriter,thereader,andthetext.Itdoesn’tresideinanyindividuallocation.

• “WritingMediatesActivity”(Russell)Writingisatechnology—asetoftoolsthatallowustocommunicatewithothersacrosstimeandspace.Writingmediates,orcomesinbetweenandfacilitatestherelationshipbetweenpeople,usingtext.Inthisway,writingisactive;itisproductiveandrelational.Writinghelpstocreateandcommunicateinformationandmeaningbetweenpeople.

• “Writingenactsandcreatesidentities(Scott)Writingisawayofcreatingandcommunicatinganidentity.Itisavulnerableactthatrequiresanindividualtobringtheirownviews,ideologies,andbeliefsintocontactwithanotherperson.Becausewritingisvalue-laden,itisalwayshighlypersonal,makingthestakeshighforboththewriterandthereader.

• “Revisioniscentraltowriting”(Downs)Revisionisakeybehaviorforcreatingwell-formed,organizedtextthatanotherpersoncanpickupandread.Therevisionthatiskeyhastodowiththerhetoricalfoundationsofthetext—thecontext,content,audience,topic,largerconversation,developingknowledge.

Mypointinbringingupsuchbehaviors,skills,andcompetenciesistohighlightthefactthatweteachmanythingsinWRTG2010thatcan’tbeeasilyassessedinanassessmentsuchastheonedescribedhere(futureandidealassessmentsdescribedinthefollowingsection).Further,theseissuesandconcernsthatgounassessedarethosethatwillhelpstudentsassessandproperlyrespondtothewritingsituationsthattheywillencounterintheirtimeattheUofU.Forexample,understandingaudienceinanuancedandcompletewaywillallowstudentstounderstandprofessorsthattheywillwriteforinthefuture.Similarly,internalizingtheneedforandimportanceofrevisionwillhelpstudentsdobetteronpapersinfutureclasses,becausetheywillhavenaturalizedtheprocessofdraftingandrevisingintoamorecompleteandrobustdraft.Draftingandrevisioninparticularwouldbedifficulttoassessinastandardassessmentsuchastheonedescribedandappliedhere.7.3NextStepsInthissection,Iwilldescribeanidealassessmentandsomeoftheelementsofthisassessmentthatwillbeimplementedinthenextassessment.

PortfolioAssessment:Aportfoliomethodwouldbeideal.Benefits:Anidealportfoliowouldincludebothformalwriting(majorpapersanddrafts),informalwriting(dailywritingandprewriting),arevisedpaper,andareflectionpaper.Thiswouldallowviewintowhatstudentsdoacrossrevision,howtheyplanpapers,andhowtheyarethinkingthroughtheprocessesofwritingandcomposition.Costs:Aportfolioassessmentislaborintensive,costingasignificantamounttopaytheratersfortheirwork.Aportfolioassessmentalsotakeslonger,leadingtoraterfatigueandcreatingtheneedformorere-norming.ReflectionPaper:Akeypartofanyportfolioisareflectionpaper,orapaperinwhichthestudentreflectsonwhats/herlearnsinWRTG2010.Benefits:Thereflectionpapertypicallypunctuatestheportfolio,butitwouldalsobehelpfultoreadandassessthispaperapartfromtheportfolio.Thispapergetsatthemetaknowledgethatstudentslearn—howwelltheycanexplainwhattheyknowaboutandcandowithwriting.Thiswouldhelpusbetterunderstandwhatstudentsknowaboutthethresholdcompetenciesthattheyhavelearned.Costs:Reflectionpapersarenotoriouslyhardtoassess,becausetheyintroducesubjectivitythattroublestheassessmentrubric.Surveys:Asurveywouldallowustogetatattitudesandvaluesofstudents.Benefits:Betterunderstandingthevaluesandattitudesofstudentswouldallowusinsightintohowstudentsareuptaking,integrating,andinternalizingthelessonsaboutrhetoricalwriting,writingasasocialpractice,writingas

ideological,writingasidentityconstruction,andthelike.Addedtoastandardassessment,asurveywouldgivemuchneeded,deepcontextforthefinalnumbers,addingdimensiontothethresholdcompetenciesanalysisthatwealreadydo.Costs:Wewouldnotbeabletocorrelatethresholdcompetencyassessmentsdirectlywithsurveydata.WewouldattempttogetmostoftheWRTG2010studentstotakethesurvey,whileonlyasampleofstudentpapersareassessed.

Forthenextroundofassessment,wewilladdasurveythatwillbedistributedtoallWRTG2010students,allowingustogetatsomemoreofthemoreorlesstacitknowledgethatstudentsacquireinWRTG2010.Thissurveywillbecraftedtogetatthesocio-rhetoricalknowledgethatstudentsgaininawritingclass.7.4ConclusionsThisreporthasshownthatwearesuccessfulinthefirst-yearwritingprogramattheUofU.Moststudentsareatleastminimallymeetingthethresholdforthecompetenciestaughtinthefirst-yearwritingprogram(sourceuse,synthesis,rhetoricalawareness,cohesion,research,style&mechanics),withbetween35%and41%meetingorexceedingthethreshold,dependingonthecompetency.WehavesignificantlyreducedtheDWEfiguresandincreasedretentionandcompletionofbothWRTG1010andWRTG2010overthelast5years.Forfutureassessmentperiods,wesuggestanaugmentedassessment,whichwouldincludeareflectionpaperandasurveydistributedtoallstudentswhocompleteWRTG2010.Theseadditionswouldgiveusaccesstoabroaderspectrumofrhetoricalcompetencieslearnedinfirst-yearwritingattheUofU.PleasecontactJenniferAndrus(j.andrus@utah.edu)withanyquestionsorformoreinformation.WorkscitedAdler-Kassner,L.,&Wardle,E.(2015).Namingwhatweknow:Thresholdconceptsofwritingstudies.UniversityPressofColorado.Bazerman,C.(2015).Writingexpressesandsharesmeaningtobereconstructedbythereader.Downs,D.,&Wardle,E.(2007).Teachingaboutwriting,rightingmisconceptions:(Re)envisioning"first-yearcomposition"as"IntroductiontoWritingStudies".CollegeCompositionandCommunication,552-584.Estrem,H.(2015).Writingisaknowledge-makingactivity.InAdler-Kassner,L.,&Wardle,E.(eds),Namingwhatweknow:Thresholdconceptsofwritingstudies.UniversityPressofColorado.Haswell,R.H.(2000).Documentingimprovementincollegewriting:Alongitudinalapproach.Writtencommunication,17(3),307-352.--------- (Ed.).(2001).Beyondoutcomes:Assessmentandinstructionwithinauniversitywritingprogram(Vol.5).GreenwoodPublishingGroup.Huot,B.(1996).Towardanewtheoryofwritingassessment.Collegecompositionandcommunication,47(4),549-566.Lloyd-Jones,R.(1977).Primarytraitscoring.Evaluatingwriting:Describing,measuring,judging,33-66.Lundsford,A.(2015).Writingaddresses,invokes,and/orcreatesaudiences.InAdler-Kassner,L.,&Wardle,E.(eds),Namingwhatweknow:Thresholdconceptsofwritingstudies.UniversityPressofColorado.Odell,L.(1981).Definingandassessingcompetenceinwriting.ThenatureandmeasurementofcompetencyinEnglish,95-138.

Odell,L.,&Cooper,C.R.(1980).Proceduresforevaluatingwriting:Assumptionsandneededresearch.CollegeEnglish,42(1),35-43.Roozen,K.(2015).Writingisasocialandrhetoricalactivity.InAdler-Kassner,L.,&Wardle,E.(eds),Namingwhatweknow:Thresholdconceptsofwritingstudies.UniversityPressofColorado.Russel,D.R.(2015).Writingmediatesactivity.InAdler-Kassner,L.,&Wardle,E.(eds),Namingwhatweknow:Thresholdconceptsofwritingstudies.UniversityPressofColorado.Skulmoski,G.J.,&Hartman,F.T.(2009).Informationsystemsprojectmanagersoftcompetencies:Aproject-phaseinvestigation.ProjectManagementJournal41(1),61-80.Wardle,E.(2007).Understanding“transfer”fromFYC:Preliminaryresultsofalongitudinalstudy.WritingProgramAdministration31(2),65-85.Yancey,K.B.(1999).Lookingbackaswelookforward:Historicizingwritingassessment.Collegecompositionandcommunication,50(3),483-503.Yancey,K.B.,&Huot,B.(1999).AssessingWritingacrosstheCurriculum:DiverseApproachesandPractices.PerspectivesonWriting:Theory,Research,Practice.Volume1.AblexPublishing/JAIPressInc.

Appendix1:2014RatingForm

4: Excellent 3: Above Average

2: Adequate

1: Needs Improvement

Source Use Keywords: • Academic

argument • Thesis, • Synthesis, • Research • Source use, • Source

quality, • Source

attribution, • Evidence

supports claims,

• Evidence drawn from sources,

• Audience awareness

• The paper indicates strong use of research skills through reference to source material

• The paper thoroughly synthesizes relevant literature

• The paper contributes an argument to the literature on the topic

• The paper has an obvious and consistent thesis

• The thesis is well-situated in relevant literature

• The thesis is supported with verifiable evidence from reliable sources

• Evidence is drawn from a variety of quality sources

• Ideas, evidence, and words are attributed to the source from which they were drawn

• All information in the paper is relevant to the thesis

• The paper has strong indications that it was accommodated to an (academic) audience

• The paper indicates some use of research skills through reference to source material

• The paper mostly synthesizes relevant literature

• The paper contributes an argument to the literature on the topic

• The paper has an clear and consistent thesis

• The thesis is situated in relevant literature

• The thesis is supported with evidence that is mostly verifiable from sources that are typically considered reputable and reliable

• Evidence is drawn from a somewhat homogeneous variety of sources

• Sources used are mostly quality

• Ideas, evidence, and words are attributed to the source from which they were drawn

• All information in the paper is relevant to the thesis

• The paper has indications that it was accommodated to an (academic) audience

• The paper indicates use of limited research skills through reference to source material

• The paper synthesizes literature that is typically relevant

• On occasion, the paper lists and summarizes rather than synthesizing

• The paper focuses on the topic but with an unclear argumentative position

• The paper has a thesis that remains consistent for the majority of the paper with some wavering

• The thesis is mostly related to relevant literature but not explicitly situated

• The thesis is mostly supported with evidence from sources that are more or less reliable

• Evidence is drawn from a homogeneous sources with questionable quality

• Ideas, evidence, and words are mostly attributed to the source from which they were drawn

• Most information in the paper is relevant to the thesis

• The paper has some indication that it was

• The paper indicates use very limited research skills through very reference to limited and homogenous source material

• The paper lists or summarizes rather than synthesizing prior research

• The paper does not include a representation of prior research on the topic

• The paper thesis is weak or absent

• The thesis is opinion

• The thesis is supported by opinion

• Evidence is limited • Evidence is drawn

from unreliable or unverifiable sources

• Many ideas, evidence, and words are not attributed to the source from which they were drawn

• The paper has no indication that it was accommodated to an (academic) audience or the audience is not appropriate for the course

accommodated to an (academic) audience

Cohesion & Structure Keywords: • Paper

structure, • Idea cohesion, • Transitional

sentences and phrases,

• Logic & organization,

• Claim structure

Cohesion & Structure, cont’d.

• The introduction is substantive, providing thorough and relevant context and background information

• The introduction defines all of the key terms and concepts

• The introduction creates a clear and logical structure for the paper

• The structure established in the introduction is followed

• The structure is strong throughout the paper, maintained at the sentence and phrase level

• There are transitional sentences and phrases between paragraphs

• Ideas are organized logically at paragraph and whole paper levels

• The body of the paper includes sub-claims that provide support for the thesis and reinforce the argument established in the introduction

• The sub-claims are well-reasoned and organized

• Paper draws conclusions that follow from the claim structure of the argument

• The introduction provides strong context and background for the paper

• The introduction creates logical structure for the paper

• The introduction defines a significant portion of the key terms and concepts

• The structure established in the introduction is mostly followed in the body of the paper

• The structure is maintained with transition sentences and phrases

• Paragraphs are mostly connected with transitional sentences and phrases

• Ideas are organized logically at paragraph and whole paper levels

• The body of the paper includes sufficient sub-claims to provide support for the thesis and mostly reinforce the argument established in the introduction

• The sub-claims are mostly well-reasoned and organized

• Paper draws conclusions that follow from the claim structure of the argument

• The introduction provides minimal background and context for the paper

• The introduction minimally creates a structure for the paper

• The introduction defines most of the key terms and concepts

• The structure established in the introduction is somewhat followed

• The structure is somewhat maintained with transition sentences and phrases

• Paragraphs are somewhat connected with transitional sentences and phrases

• Most sub-claims reinforce the argument established in the introduction

• Ideas are mostly organized logically at paragraph and whole paper levels

• The body of the paper includes some sub-claims to provide support for the thesis, more or less reinforcing the argument established in the introduction

• The sub-claims are not well-reasoned

• The sub-claims are not organized

• The evidence is lacking in relevance and strength

• Paper makes an effort at drawing

• The introduction is brief and/or overly general and does not create a structure for the paper

• The introduction does not sufficiently define key terms and concepts

• The introduction does not establishes a structure for the rest of the paper

• The organization of the body is not structured in a way that logically supports the thesis

• The paper lacks strong transition sentences and phrases, especially between paragraphs

• There are not sufficient sub-claims to support the thesis

• Sub-claims are not connected to the argument logically

• Claims and sub-claims are not supported with evidence

• Claim structure lacks organization

• Ideas are not organized logically at paragraph and whole paper levels

• Paper lacks a conclusion or the conclusion is significantly off topic

conclusions from the presentation of claims and sub-claims, though not all conclusions may follow from the paper

Style & Mechanics Keywords: o Academic

tone, o Conventions

of standard, written, edited English,

o Spelling, o Punctuation, o Citation style

• The tone is academic

• The word choice and organization fully follow written academic conventions

• The paper has few if any spelling or punctuation errors

• The grammar of Standard Written English—verb agreement, etc.—is nearly perfect

• Source material is correctly referenced all of the time

• There is a works cited page

• The paper correctly uses in-text parenthetical citations

• An academic citation style has been applied consistently across the paper

• The tone is mostly academic

• The word choice and organization mostly follow written academic conventions

• The paper has some spelling or punctuation errors

• The grammar of Standard Written English—verb agreement, etc.—is very good

• Source material is correctly referenced most of the time

• There is a works cited page

• The paper uses in-text parenthetical citation

• An academic citation style has been applied consistently across the paper

• The tone is somewhat academic

• The word choice and organization follow written academic conventions on and off

• The paper has many spelling and/or punctuation errors

• The grammar of Standard Written English—verb agreement, etc.—could be improved

• Source material is referenced, with a few errors

• There is a mostly complete works cited page

• There are in-text parenthetical citation, but with some errors

• An academic citation style has been applied nearly consistently across the paper

• The tone is not academic

• The word choice and paper structure shows a lack of knowledge of written academic conventions

• The spelling and/or punctuation errors disrupt readability

• The grammar of Standard Written English—verb agreement, etc.—has many errors

• There is not a works cited page, or the works cited is very incomplete or incorrect

• Source material is often not referenced

• There is not sufficient use of in-text parenthetical citation

• There is not an obvious academic style applied in the paper

2015RatingFormCompetency 4: Excellent 3: Adequate 2:

Minimal 1: Scarce

7. Source Use • Sources used as evidence, • Sources used are varied, • Sources support claims and sub-claims, • Sources are appropriately attributed

8. Synthesis • Text combines sources, • Sources are integrated in a number of ways, including

direct quotation and paraphrase, • Connections are drawn across sources, • Text creates a network of sources relevant to present

argument

9. Rhetorical Awareness • Thesis indicates the rhetorical purpose of the text, • Text shows awareness of and accommodation to audience, • Text shows awareness of writing context, • Word choice and paper features are appropriate for genre, • Argument is situated in an academic conversation, • Argument contributes to an academic conversation

10. Research Skills • Sources are quality • Research from a variety of media and type of sources

11. Textual Cohesion • Paper structure is obvious, • Ideas are developed cohesively, • Arguments are well developed across the paper, • Transitional sentences and phrases are used, • Main claim and supporting claims progress logically

12. Style & Mechanics o Academic tone, o Conventions of standard, written, edited English, o Spelling, o Punctuation, o Citation style