pcapp negotiated project pcapp_clucas.docx 3 of 35 copyright © 2016 - by c. luacs

File:PCAPP_CLucas.docx 1of35 Copyright©2016-byC.Luacs

PCAPPNegotiatedProjectDrClaireLucas,SchoolofEngineering,UniversityofWarwick

ActionprioritisationandKANOanalysisofstudentsurveyresponses

1 Introduction

1.1 NSSandTEF

TheNationalStudentSurvey(NSS)isanannualsurveyoffinalyearundergraduatestudentsdesigned

toassessthequalityofundergraduatedegreeprogrammesbasesonresponsesto22itemsin6factors

(teaching, assessment and feedback, academic support, organisation and management, learning

resourcesandpersonaldevelopment).ThesurveyisconductedbyIpsos-MorionbehalfoftheHigher

EducationFundingCouncilforEngland.Itisproposedthatthescoresachievedwillbepartofaseries

ofmetricsusedtoformtheTeachingExcellenceFramework(TEF)from2017whichwillgiveascore

for university courses which is linked to fees in future. Succeeding in the NSS is therefore vitally

importantforthefutureincomeofthedepartment.TheSchoolofEngineeringinWarwickdoesnot

scorehighlyinNSScomparedtocompetitors.Therearemanypotentialreasonsforthisasdescribed

inthisreport.

1.2 ModuleSurveys

Theuseofmodulesurveyscanhelptoidentifygoodandbadindividualteachingwithinadepartment.

Theycanbeembeddedintoacontinuousimprovementprocesswithdataownedbythedepartment.

Modulesurveysareubiquitousamongstuniversitiesasamechanismforgathering feedbackabout

teachingfromstudents.

1.3 KanoModelling

Thevalidityofsatisfactionasameasureforgoodlearningwillnotbeaddressedinthisproject.Itis

evidentthatuniversitieswhichscorehighlyinNSSarenotthesameuniversitieswhichscorehighlyin

othermeasures.Whilstsomeconsiderthisevidencethatresearch-leduniversitiesdonotprioritise

goodteaching,otherssaythatthisisbecausetheexpectationsofstudents(anonstudents)differby

institution.ThelinkbetweenexpectationandsatisfactioninconsumerproductsisanalysedbyKano

modellingwhichcategorisesproductfeatureswithlabelssuchas‘essential’,‘nicetohave’,‘neutral’


and‘reverse’.KanoanalysiswouldsaythattheexpectationofRangeRovercustomersisdifferentto

thoseofFordFocusandsoRangeRovercustomersareconsequentlyhardertoplease.

1.4 MotivationandAims

WarwickSchoolofEngineeringNSSratingsarelowcomparedtocompetitorsandalsocomparedto

other departments within Warwick. However, our alumni, external examiners, professional

institutionsandindustrycontactspraisethecalibreoftheWarwickprogramme,evencomparedto

graduatesfromotheruniversitieswithhighersatisfaction.Mostlikely,thesurveymeasuresthelevel

ofdisconnectbetweenexpectationandexperience.Expectationisrelatedtotheperceivedqualityof

theproductandisthereforenon-uniformacrossthesector.Themotivationfortheresearchdescribed

hereistothereforeexploretheuseofKanoanalysisonmodulesurveystoidentifythecategorieswhich

influencestudentsatisfaction.

Warwick does not provide templates, guidance or an embedded process for module surveys.

Therefore, a furthermotivation of this project was to establish practicewhichmay benefit other

departments.

2 LiteratureReview

2.1 Satisfaction

Customersatisfactionistheconceptthatthewayinwhichproductsandservicesmiss,meetorsurpass

expectationsgivesrisetoalevelofsatisfactionexperiencedbythecustomer/user.

“Consumer satisfaction with a product refers to the favorableness of the individual’s subjective

evaluationofthevariousoutcomesandexperiencesassociatedwithbuyingitorusingit”.(Hunt1977).

Herzberg’s two-factor theoryrelates toemployeesatisfactionandsays that thereare independent

driversof satisfactionanddissatisfaction (Herzberg1959). In1984,NoriakiKanoextended this to

customer satisfaction determining that there are 6 categories of product attributes which affect

satisfactiondifferently.Thetheoryincludes‘thresholdattributes’whicharethebinaryfeatures(either

presentornot)whichmustbepresentinordertosatisfy(acarmusthaveasteeringwheel).There

maybevaryingthresholdexpectationsdependentonproductprice(amoreexpensivecarshouldhave

buttonsonthesteeringwheeltocontrolthemusic).Linearattributesarethoseforwhich ‘more is

better’and‘lessisworse’.Attractivequalities‘surpriseanddelight’,theyareunexpectedbuthavea

largeinfluenceonsatisfaction.Therearealsofactorstowhichcustomersareindifferent(removingthe


CDdriveonnewestlaptops).Further,thereareattributeswhichhaveonlyasmallinfluenceeitherway

andaretherefore‘notworththeeffort’.Finally,therearereversefeatureswhichactuallydecrease

satisfaction(thepesteringsalesman).

Table1:DescriptionofKanocategories

Figure1Witell,LarsandLofgren2007

In1991,theGapmodelSERVQUALemerged(Berry1991).Thisstatesthatsatisfactionoccurswhen

perceivedperformancemeetsorexceedsexpectationsandthatdissatisfactionoccurswhenthereisa


negativegap.Itfocusesonservicereceivedatvaryingstagesofproductdelivery.Thereareseveral

gapsdefined:

TheCustomerGapisthedifferencebetweencustomerexpectations(accordingtocostandinfluenced

bybackground,advertising,lifestyleandavailableproductinformation)andtheperceptionbasedon

interactionwiththeservice.

The Knowledge Gap is the gap between customer expectation and management perception. It

determinesthatmanagershavenotcorrectlyinterpretedwhatcustomersexpect.

ThePolicyGapreflectstheincorrecttranslationofrequirementsintoserviceguidelinesforemployees.

TheDeliveryGapisthedifferencebetweentheservicespecificationandemployeedelivery.

TheCommunicationGapisthedifferencebetweenpromisesmadeandtheservicedelivered.

Satisfactionisthereforeanemotionalresponsebasedoninternalcollationoffactorsincludingactual

product/processorservicequalitycomparedwithexpectationsandpreferences.

2.2 SatisfactionEvaluation

Mostcommonly,surveysaredesignedaround‘factors’whicharebelievedtobethoseimportantto

customers. By giving a set of statements the survey requests a response on the Likert-scale (a

symmetricagree/disagreescale,Likert1932).TheNSSemploysthismethod.Eventhoughthistypeof

questionnaireisstraightforwardtoanalyse,theresultscannotdeterminethelevelofimportanceof

eachfactor.Itislikelythattheoverallsatisfactionisrelatedtoonlyafew(especiallyrecent)attributes

(Elliot2002).Satisfactioncaneitherbedeterminedatanaggregate(overall)orattribute(multifactor)

level.Thelatterassessesfactorsatisfactionandthensumsintoanoverallscore(Elliot2002).Thetwo

approachesdonotnecessarilycorrelate(Mittal1998).Kanosurveysaretwo-dimensionalinorderto

establishtheeffectofthepresenceorlackofindividualattributes.

2.3 StudentSatisfaction

Thereisalargebodyofknowledgeandmethodsforestablishingandmeasuringproductperformance

in termsof satisfaction.Mostoften, theexpectationsof a customerare linked to thepriceof the

productor service theyareusing. InUKHigherEducation, theprice factor isnotpresent sinceall

universitieschargethesamefees(thoughtheyarefreetochargeless). It isthereforeimportantto

contextualisetheroleofstudentsinrelationtothepreviouslydiscussedterminology.


Guolla(1999)describeshowstudentsareoftenconsideredascustomerswhoconsume/experience

andconsiderthemselvesexpertconsumers.Thisisincontrasttostudentsasclientswhounderstand

theirneedsbutarereliantonexpertprofessionaladvice(suchasvisitingthedoctor).Studentscanalso

be the producer (Armstrong 1995) where the instructor is the resource. Finally, students can be

considered products which are marketed to potential employers. In the latter, the student is

concernedabouttheprojected‘brand’oftheUniversity.

Theexpectationofstudentsissetbyanumberofinputs:theexperienceofschoolinformingstudent

perceptionsoflearning,opendayvisitsandprospectusesdescribingthecoursetostudentsandword

ofmouthdescribing the reputationof theuniversity.Therearealsoanumberof compiled league

tableswhichtakeintoaccountemploymentratesaftergraduation,researchsuccessoftheuniversity,

contacttimeandretentionratesofstudentswhichimpactonstudentperceptions.Appleton-Knapp

(2006)presented thedifficultyofusingperception topredict satisfaction comparing the resultsof

expectationspriortostartinguniversityandattheendofthefirstsemester.

Rapert (2004) describes two quality dimensions in higher education: process and outcome. The

provision (teaching delivery and operation of course) is related to students as customers and is

different to the value of education (usefulness of knowledge gained) where students are clients.

Douglas,Douglas andBarnes (2006) separate the service-product into threeelements: facilitation,

explicit service and implicit service. The facilitation relates to lectures, tutorials, facilities and

resources,theexplicitserviceistheknowledgeofstaff,teachingabilityetc.andtheimplicitserviceis

thekindnessoflecturers,treatmentofstudentsandthefeelingthateffortisrewarded.

There is also therefore amotivational aspect to student satisfaction: can students see a clear link

between working harder, askingmore questions, going to the library and achievingmore (either

intrinsically in satisfactionof knowledge gainedor extrinsically by grade). Thismotivational aspect

relatestostudentsasproducersandisexploredintheUSviathestudentengagementsurveyinwhich

studentsdescribetheopportunitiestheyreceivetoengageinlearning.

The self-confidence and actual knowledge gained by students relates to them as products, the

expectationonstudentsfromRussellGroupuniversitiesisdifferenttothatofotherUniversitiesand

studentsmustfeelthattheycanliveuptotheseexpectations.

NSS works with 6 domains (teaching, assessment & feedback, academic support, organization &

management,learningresourcesandpersonaldevelopment)withanoverallsatisfactionratingrelated

toquality.Theapproachof some league tables (tocalculateanarithmeticmeanresponse toeach


questionratherthantaketheoverallsatisfactionvalue)assumesequalweightingtoallfactorsandno

additionalaspectofsatisfaction.

2.4 Previousstudies

Alves (2007) used structural equation modelling finding relationships between satisfaction and

studentsperceptionofvaluerelatedtootherfactorsfindingthatinstitutionalimagewasthebiggest

influencerofvariationinsatisfaction.

1. InstitutionalImage

2. StudentExpectations

3. PerceivedValue

4. PerceivedQuality

ElliotandShin(2002)highlighteddifferencesbetweenwhatstudentsclaimedtobeimportantandthe

drivers of overall satisfaction (calculated using a multiple-item weighted gap score analysis). The

calculatedsignificantfactorsweredifferenttothoserankedas‘ideal’bythestudents.

Table2:ElliotandShinAnalysisvsStudentIdeal

TheuseofHerzberg’stwo-factortheoryinmodellingsatisfactionanddissatisfactionseparatelywas

demonstratedbyDeShields(2005).Thegroupweresplitinto‘highlysatisfied’and‘highlydissatisfied’

in order to test the theory. They found that the classroom experiencewas a dissatisfier (i.e. in a

positiontodissatisfy)andthefacultywasasatisfier(i.e.theopportunitytosatisfy).

TheuseofKanotheorywasappliedbyPetruzzellisetal.(2006)andAerfi(2012).

Table3KanosatisfactionHigherEducation,comparisonofstudies

Petruzzellisetal.(2006) Aerfi(2012


Must-

have

Tutoring

Administrativeservices

Contactwithstaff

Library

Teachingequipment

Lecturehalls

Laboratories

Enablingstudentstocarryoutresearch

Abilityoflecturertoconveymaterial

Ability of professor inmotivating students about subjectmatter

Transparentregulations

Coordination of course material (vertical/horizontalintegration)

Appropriateclassroom

Transparentassessmentstandards

Assessmentduringterm

More is

better

Scholarships

Counselling

Internships

Internetaccess

Refectories

Enablingstudentstounderstandwhatnottolearn

Uptodatematerial

Appropriatematerialtopreparestudentsforfuture

Collaborationwithstudentsinprocess

Relationshipbetweenexamcontentandemphasisoftaughtmaterial

Delighters Careerplacement

Leisuretime

Accommodation

Internationalrelations

Languagecourses

Onlineregistration

Acquisitionsoftransferableskills(e.g.IT)

Friendlinessofprofessorcommunication

Appropriatelabfacilities

Incentivestolearnmore(e.g.freebookprovided)

Indifferent Appropriatelevelofdifficulty

Appropriatelevelofself-study

Allocationofpartialcredittopracticalactivities

Douglas,DouglasandBarnes(2006)catagorisedfactorsbyimportancevssatisfaction

Table4CategorisationofimportancevssatisfactionaccordingtoDouglas,DouglasandBarnes


2.5 Summary

FactoranalysisofNSSfindssatisfactionwithteachingtobethebestindicatorofoverallsatisfaction

(ChengandMarsh2010)whichalsoalignswithotherstudiesdescribedhere(Douglas,Douglasand

Barnes2006,ElliotandShin2002).Itassumesstudentsarereceivingaservicefromexpertswhodefine


whatitistheyneedtoknowandisseparatefromthedeliveryofthatservice.Therefore,thescopeof

thisstudyistodesignasurveyandanalysistoolwhichcanbeimplementedinawaywhichimproves

teachingandthereforeresultsinhigheroverallsatisfaction.

3 Methodology

3.1 Previousstateofstudentfeedbackinthedepartment

Beforecommencingthisproject,theSchoolofEngineeringdidnothaveamodulesurvey.Previously,

therehadbeenanonlinesurveywithverypoorresponserates(<1%inmanycases).TheSchoolalso

hasaStudentStaffLiaisonCommitteewhichisunsuccessfulinfeedingbacktostudents.Eachmodule

has an online forum with poor utilisation and the peer review observation of lectures by other

academicswasnotinaction.Therewerethereforeverylimitedestablishedmechanismsforstudent

feedbacktomodules.

3.2 Useofpapersurveys

Thereare considerablebarriers to theuseofpaper in-module surveys; the time takenaway from

lectures, the logistics of distribution within a full-cohort (360 student) lecture setting and the

transcribingofresultsintoadigitalformat.Lecturerenthusiasmisalsoapotentialbarrier.Finally,only

studentsattendingthelecturewillanswerthesurveysoitdoesnotaddressstudentswhochoosenot

tocome(perhapsduetodissatisfaction).

3.3 Questiondesign

ApilotpapersurveywasintroducedinTerm12015-2016.Theformsweredistributedandcollected

bybursarystudentsandsenttoITservicesforscanning.Lecturerswereaskedtoprovidearesponse

tostudentswithin2weeksofthesurvey(orbytheendofterm)indicatingtheirthanksandproposed

actions.Surveyresponserateswerealmostallstudentswhoattendedlecturesandstudentshave

expressedthanksfortheopportunityandespeciallyfortheresponsesthey’vereceived.Noguidance

isofferedbytheuniversityonstorageanddataprotectionissues.

ThesurveywasdesignedtomimictheformatoftheNSSwithastatementandasetofpossibleanswers

ontheLikertscale.Questionsweredividedinto3categories:Content,ManagementandDelivery.The

aimofthethreesectionswastodivideresponsibilitybetweenthecurriculum(oftenownedoutsideof

the lecturers by the Discipline Degree Leader, or devolved toModule Leader), the administration


(timetabling,roometc.)andthelecturerthemselves(lecturingstyle).Anoverallsatisfactionquestion

wouldallowforcalibrationbetweenmodules.

Therewasadeliberatedecisionnottoincludeareflective/engagementquestioninthesurvey.This

wasbecausestudentswithintheschoolarenotwell-practicedatreflection.Twoquestionsabout‘the

amountofself-studyrequired’and‘theavailabilityofresourceslistedinthereadinglist’wereincluded

togaugestudentsviewontheirownroleinlearning.AstheNSSmovestopotentiallyincludequestions

aboutengagement,thesurveycanbealteredtoreflect.

Thereweretwophasesofquestionsusedwithaninternalreviewafterthefirstterm.Academicswere

invitedtoproposefactorstheybelievedinfluencedsatisfactionwhichcouldbeturnedintoquestions.

Engagementwasverylowwithonly4attendingtheopenmeeting(includingtheHeadofSchool).The

SSLCfeedbackwasalsousedtoinfluencequestionchoiceandquestionswereintendedtoreflect‘basic

expectations’ofstudents.Question1bwaschangedfrom‘Themoduleisatasuitableacademiclevel’

to‘Theself-studyrequiredwasaboutright’.


Figure2:SchoolofEngineeringSurveyTerm2

3.4 ImprovementStrategy

.Threeimprovementstrategieswereconsidered:

1 Movefrom‘neitheragreenordisagree’to‘stronglyagree’

2 Movefrom‘stronglyagree’to‘definitelyagree’

3 Movefrom‘disagree’to‘agree’

Thereare twoaspects to strategy1: reduce thenumberof studentsanswering ‘neitheragreenor

disagree’ who chose that incorrectly (either because they assumed it was a neutral rather than

negative response,orbecause theyconflated itwithnotapplicable)and topersuade intentionally

neutralstudentstobemorepositive.Strategy1isimportanttoinvestigatesincestudentscannotbe

coachedduringtheNSSonhowthesurveyisanalysed.Theycan,however,takesimilarsurveyswhere

it ismadeclearthattheseanswersarenegativeandthat‘notapplicable’ is, insomeinstances,the


correctanswer.Althoughanswering‘notapplicable’doesnotaffectthe%satisfied,itdoescontribute

tothesecondNSSmetricwhichisthearithmeticscoreforsatisfaction.

StrategywasdiscountedbecauseitdoesnotaffectthemainNSSmetricandreliesonindividualstudent

culture.

Strategy3shouldbesubdividedasA:‘picktheworsttoimprove’orB:‘picktheeasiesttoimprove’or

C:‘pickthemeasurestoimprovewhichwillhavethegreatestimpactonsatisfaction’.Simpleanalysis

willresultinAwhichisthelevelthat,sofar,thedepartmentNSSactionplanhasusedalongwithB.

TheproposalofthisinvestigationistoworktowardsC,judgetheimpactofeachmeasureandprioritise

byeffectonoverallsatisfaction.

Theprioritymustthereforebeanystatementswhichfallintothe‘must-have’withhighdissatisfaction.

Theseareexpectationsnotbeingmetby thedepartment. Following this, questions in the ‘Linear’

classificationcanbeusedtopushstudentsatisfactionupthescale.

3.5 Surveyanalysismethod

Theanalysisisdesignedtoreducetheeffectofindividualstudentscalelookingatcontributingfactors

totheoverallsatisfactionscore.Theresearchquestionistherefore“whatinparticularcausedyouto

bedissatisfiedwiththismodule?”.TheShapelyvaluesaysthatwithagroupofplayers(here,aspects

ofacourse),somewillcontributemoretotheoverallcoalitionandsomewillbeaparticularthreat.

Further,somesmallimprovementsbyplayerscanhaveabiggereffectthaninothers.Particularstrong

improvers or threats are here called ‘Drivers’ (Satisfaction Drivers or Dissatisfaction Drivers).

Satisfactionwas judged to be a positive response (Strongly orDefinitelyAgree) and (aswithNSS)

neitheragreenordisagreewasjudgedtobedissatisfactionratherthanneutral.

Theaimoftheanalysisisthereforetocalculateametricforsatisfaction/dissatisfactiondriverandto

presentinanintuitivewayasareport.Thesimplestmetricisthepercentageofstudentswhowere

satisfiedwithaparticularquestionandoverall.


Table5:CalculationofDrivers

Theactual%SatisfactionandDissatisfactionresponsetoeachquestionwascombinedwiththemetric

above to form the ‘risk’ levelofeach factor. Forexample,ahighdissatisfactiondriverwithahigh

dissatisfactionscoreisabiggerriskthanonethatisalowdriverwithequallyhighdissatisfaction.

Withmultipledimensionsofdata(Questionnumber,satisfactiondrivermetric,satisfaction)abubble

chartwasusedtopresentresults.

Figure3:Examplebubblechartofsatisfaction


Figure4:Examplebubblechartofdissatisfaction

Thesizeofthecircleindicatesthedrivermetric(withredbeingparticularlystrongandgreenbeing

weak)andthey-axisgivesthe%satisfied/dissatisfiedforthatquestion.Thegraphshouldbeusedto

identifylargeredcircleswithhighdissatisfaction,orlargeredcircleswithlowsatisfaction.

Inordertomakebestuseofthisanalysis,studentexpectationsmustfirstbeestablished.Therefore,

allresultshavetobecollatedcentrallysothat individualmodulescanbejudgedagainsttheglobal

drivermetric,ratherthanthatfortheindividualmodule.Thedifferencebetweendriversofindividual

modulescanbeanalysedfurther.

The‘raw’answerstoindividualquestionsarealsoincludedinthereportforlecturers


Figure5:Exampleofbarchartdisplay

Finally,theanswersaregroupedintosatisfactionscoresforeachquestionandcategoryalongwiththe

meanarithmeticscore.Thisiscalculatedbygivingascoreof5to‘Stronglyagree’andmultiplyingby

thenumberofstudentswhoanswer‘Stronglyagree’(andsoon).


Figure6:Factoranalysisofanswers

4 SurveyResults

Overthecourseoftheyear,3260individualsurveyresponseswerecollectedwith71%ofmodules

conductingasurvey.Thisrepresentsamuchhighersetofdatathanwaspreviouslyavailable.

Theoverallsatisfactionwithmoduleswas82%,thisislowerthanthetargetof90%formodules(based

on the Russell Group NSS score). The category related to module content received the highest

satisfactionwiththedeliveryreceivingthelowestscore.

Looking at the raw results, the weakest areas were laboratories (though for many this was not

applicable) and lectures being ‘engaging’. To address the question of ‘what makes an engaging

lecture?’focusgroupswereconductedinthedepartmentasthefocusofanotherresearchproject.


Figure7:Resultsforacademicyear2015-2016

Thebiggestdriversofdissatisfactionwere‘Content’,‘Management(inparticularpreparationofthe

lecturer)andDelivery(inparticulartheabilityofthelecturertoanswerquestions).Theseweresome

ofthelowestscoresfordissatisfactionthough,onlyDeliveryisconsideredathreat.

Thebiggestdriversofsatisfactionwere‘interestingcontent’,‘management’(inparticularpreparation)

and delivery (in particular engaging lectures and provided resources). Of these, those related to

deliveryscoredthelowestsatisfaction.

4.1 KANOdescriptors

Byplottingdriversofsatisfactionagainstdriverofdissatisfaction,thequestionscanbecategorised

byKanodescriptors.


Figure8:Kanoanalysisofsurveyresults

Table6:Kanoclassificationofsatisfactionfactors

ComparingtheresultsshowninTable6withthefindingsofAerfi(2012);thereissimilarityinthatthe

contentisa‘must-have’itemaswellasrelevance(motivation)andapplications(enablingof

students).The‘moreisbetter’qualityofcollaborationispartiallyindicatedbylecturerresponseto

questions.Laboratorieswerea‘delight’factorinbothsurveysandstudentswereindifferenttoself


studyinboth.Themaindifferencewasthatdeliverywasa‘musthave’inAerfibut‘Attractive’in

Lucas.ThismaybebecausethedeliverywasvariableintheSchoolofEngineeringgivingmore

granularitytotheresults.

Table7:ComparingAerfiandLucasKanoAnalysis

Aerfi(2012 Lucas(2016)

Must-have Enablingstudentstocarryoutresearch

Abilityoflecturertoconveymaterial(like3)

Abilityofprofessorinmotivatingstudentsaboutsubjectmatter(like1d)

Transparentregulations

Coordinationofcoursematerial(vertical/horizontalintegration)(like1)

Appropriateclassroom

Transparentassessmentstandards

Assessmentduringterm

1Content

1aRelevance

1dClearapplications

2aPreparation

2cWell-Run

Moreis

better

Enablingstudentstounderstandwhatnottolearn

Uptodatematerial(like1a)

Appropriatematerialtopreparestudentsforfuture(like1d)

Collaborationwithstudentsinprocess(like3b)

Relationshipbetweenexamcontentand

emphasisoftaughtmaterial

2Management

3Delivery

3bRespondtoquestions

Delighters Acquisitionsoftransferableskills(e.g.IT)

Friendlinessofprofessorcommunication

1cInterestingcontent

3aEngaginglectures


Appropriatelabfacilities(like3c)

Incentivestolearnmore(e.g.freebookprovided)

3cGoodlaboratoriesandseminars

Indifferent Appropriatelevelofdifficulty

Appropriatelevelofself-study(like2b)

Allocationofpartialcredittopracticalactivities

2bReadinglist/self-study

2dSupportingmaterials

4.2 SWOTanalysis

Thestrategicoutcomeofthesurveyisinunderstandingwheretoprioritiseaction(ascarriedoutby

Douglas,DouglasandBarnes).Thoughtheirsatisfactiondiffered,therelationshipwithimportancewas

thesame.

Threatswherewemustreducedissatisfaction:

Thequalityofdeliveryisbothalineardriverofdissatisfactionandalsoacurrentweakness.

Opportunitieswherewemustconcentrateeffortstoincreasesatisfaction:

Laboratoryactivitieswhichenhancethetopic,engaginglecturesandinterestingtopicsarefoundtobe

bothdriversofsatisfactionandalsocurrentweaknesses.

4.3 Neitheragreenordisagree

A further analysis considers whether students conflate ‘Neither agree nor disagree’ with ‘not

applicable’.WearenotallowedtoremindstudentsofthisforNSSbutcanusethesesurveyresultsto

discussthedifference.ItisanidentifiedweaknessoftheNSSandisalsoseenhere.

Questionswithalarge%responseforneitheragreenordisagreeare:

1c:Themodulecontentwasinterestingtome(17%)

1d:Theapplicationofthetopicsarecleartome(11%)

2b:Thereadinglistmaterialwasreadilyavailabletome(14%)

2d:Supportingmaterialsweremadeavailabletome(13%)

3a:Thelectureswereengaging(19%)


3b:Thelecturerrespondedwelltoquestions(13%)

3c:Thelaboratoryactivitieshelpedmetounderstandthetopicbetter(16%)

3d:Theresources(e.g.handouts)aregood(15%)

3band3careespeciallygoodexamplesoftimeswhenstudentshaven’tactuallyhadalaboraskedany

questions.Question3aispossiblyacaseofanambiguousquestion.

Moreconcerningaretheanswerstoquestions2band2dwhichimplythatstudentsaren’taccessing

extramaterialforprivatestudy.

4.4 Variationbyyearofstudy

Animportantconsiderationisthevariationofsatisfactionoverthecourse.Thereweremanymore

studentssurveyedinYear1thanotheryearswithmanysharedmodulesrequiringmultiplesurveys

tocovereachlecturer.

3rd and 4th year students were the most satisfied. This may be because their expectations have

changed, because they aremore challenged or because they are studying in smaller classes. The

satisfaction with content is fairly consistent excluding a dip in Year 2 where there were two

programmingmodules(onecore,oneoptional)whichachievedsatisfactionscoresof59%and58%.

Modulemanagementwasalso lower inyear2andmoduledeliverywassignificantly lowerthan in

otheryears.

Thereissomeuseful informationfoundbycomparingKanodescriptorsforeachyear.Forallyears,

moduledeliveryislinearlyrelatedtosatisfaction:improvementsorweaknessinthisareacanmove

overall satisfactionupanddown.Therefore, the face-to-face contact time ismore important than

contenttosatisfactiondifferences.

Inmostyears, thereading listandsupportingmaterialswere judgedtobe ‘indifferent’, thishasa

seriousimplicationthatstudentsarenotcommittingtofurtherself-studyaspartofmodules.Anotable

exception is Year 3 where supporting material is a ‘must have’ for satisfaction, the year with a

noticeablestepupindifficultyandabstraction.

Goodcontentisabasicexpectationforalmostallyears(nudgingintolinearforYear3).Inparticular,

relevance is a must-have in Year 1 (who are perhaps not best placed to judge relevance). Clear

applicationsofthetopicarekey inyear1butotheryears judgethistobemore linearlyrelatedto

satisfaction.Adequatepreparationbythelectureraremusthavesinthefirsttwoyearsbutlesssoin


Years3and4whentheyarelinearfactors.Themodulerunningwellisalsoofhighimportancetofirst

years.

Engaging lectures are a key opportunity for all years improving satisfaction. However, poor

engagementdoesnotleadtodissatisfactiondirectly.InYears2and3,goodlaboratoriesandseminars

arealsoattractivebut4thyearsareindifferent.Interestingcontentisalsoa‘nicetohave’forallyears

whichboostssatisfactionbutisnotacauseofdissatisfaction.

ThistypeofKanoanalysisisnormallydoneinadvanceofproductdesign,askingcustomersabouttheir

expectationsratherthantryingtogleaninformationfromsurveyresults. It isvitally importantthat

theseexpectationsareexplored further.Goodcontent is aparticular strength inall yearsand the

department should therefore have confidence in the curriculum.Module delivery, however, is an

opportunity for improvementespeciallywhensupplementedbygood labsandextraseminars.The

departmentshouldconcentrateeffortinthisarea.


Table8:Variationinsatisfactionbyyearofstudy

4.5 ResultscomparedwithNSS

The2016NSSresultswereworsethanpreviousyearswith83%overall (-1%)butvariedbycourse.

Withinthesamedepartment,onewouldexpectthatonlytheteachingwouldvarybetweencourses

whilst the policies related to management of the course, the learning resources and personal

developmentwouldbethesame.Further,thecoursesshareacommonfirst2years.


Table9:NSSresults2016bycoursecomparedwithmodulesurvey

Comparingwithyear4surveyresultsshowslowersatisfactionthanthatmeasuredonamodulebasis

butwithsomecorrelationbetweenhighermodulesurveyresultsandhigherNSSresults.NSSis

dominatedbythelargenumbersofMechanicalstudents(60comparedto14electronicand41Civil).


Table10:ModuleSurveyresults2016bymodule

ComparingthemodulesurveyresultsshowsgoodconsistencyintheCivilofferingcomparedwiththe

Mechanicalmoduleswhereonemoduleachievedonly26%satisfactionbasedondeliveryofthe

course.


5 ConclusionsandRecommendations

TheaimofthisstudywastoestablishbestpracticeforsurveyanalysiswithintheSchoolofEngineering

andestablish theusefulnessasapredictorofNSSscoresaswellasofferingananalysisof student

satisfactionbasedonthegapbetweentheirexpectationandtheservicereceived.

The study was able to gather large amounts of data by using a paper-based survey analysed by

universityITservices.Ananalysistoolwasintroducedwhichgavelecturersinformationabouttheir

module performance and an overall analysis of importance was used to create a strategy for

concentrationofeffortonthreatsto,andopportunitiesfor,increasedstudentsatisfaction.

Thedepartmentstrengthisitsreputation,backedupbythecontentofthedegreewhichreceived90%

satisfaction.However,thevariationinteachingqualityisathreatandthoughtherewasevidenceof

outstandingpractice(14/73modulesreceived100%satisfactionfordelivery)therewerealsosome

poorresults(19modulesscoredlessthan60%satisfaction).

Laband seminarprovisionwouldenhanceoverall satisfaction (according to theanalysis) but their

absence does not correlate with dissatisfaction. Similarly, concentrating on making the content

interestingandengagingshouldnotbedoneattheexpenseofclearapplications(viaexamples)and

preparationofthelecturer.

Module satisfaction summed intooverall satisfactionby course is aweakpredictorofNSS scores.

Furtherworkinthisareawouldbebothinterestingandbeneficial.Theresultsofsubsequentsurveys

byyearwouldshowimprovementandverifythestrategyhere,toconcentrateonspecificthreatsand

opportunities.

Thisstudybringstogetherworkdoneatauniversitylevelandappliesittomoduledeliveryshowinga

potentialmethodtoberolledouttootherdepartmentswithintheUniversity.


6 References1. ALVES, HELENA, ANDMÁRIO RAPOSO. "CONCEPTUAL MODEL OF STUDENT SATISFACTION IN HIGHER EDUCATION."

TOTALQUALITYMANAGEMENT18.5(2007):571-588.

2. APPLETON-KNAPP,SARAL.,ANDKATHLEENA.KRENTLER."MEASURINGSTUDENTEXPECTATIONSANDTHEIREFFECTSONSATISFACTION:THE IMPORTANCEOFMANAGINGSTUDENTEXPECTATIONS." JOURNALOFMARKETINGEDUCATION28.3(2006):254-264.

3. AREFI,MAHBOUBE, ETAL. "APPLICATIONOFKANOMODEL INHIGHEREDUCATIONQUALITY IMPROVEMENT:STUDYMASTER’SDEGREE PROGRAMOF EDUCATIONAL PSYCHOLOGY INSTATEUNIVERSITIESOFTEHRAN."WORLDAPPLIEDSCIENCESJOURNAL17.3(2012):347-353.

4. ARMSTRONG, J.SCOTT."THEDEVIL'SADVOCATERESPONDSTOANMBASTUDENT'SCLAIMTHATRESEARCHHARMSLEARNING."JOURNALOFMARKETING59(1995):101-106.

5. BERRY,LEONARDL.;A.PARASURAMAN(1991).MARKETINGSERVICES:COMPETINGTHROUGHQUALITY.NEWYORK:FREEPRESS.ISBN978-0-02-903079-0.

6. BROWNE, BEVERLY A., ET AL. "STUDENT AS CUSTOMER: FACTORS AFFECTING SATISFACTION AND ASSESSMENTS OFINSTITUTIONALQUALITY."JOURNALOFMARKETINGFORHIGHEREDUCATION8.3(1998):1-14.

7. MARSH,HERBERTW., AND JACQUELINE CHENG. "NATIONAL STUDENT SURVEY OF TEACHING INUK UNIVERSITIES:DIMENSIONALITY, MULTILEVEL STRUCTURE, AND DIFFERENTIATION AT THE LEVEL OF UNIVERSITY AND DISCIPLINE:PRELIMINARYRESULTS."RETRIEVEDJUNE23.2010(2008):603-641.

8. CHENG, JACQUELINE HS, AND HERBERT W. MARSH. "NATIONAL STUDENT SURVEY: ARE DIFFERENCES BETWEENUNIVERSITIESANDCOURSESRELIABLEANDMEANINGFUL?."OXFORDREVIEWOFEDUCATION36.6(2010):693-712.

9. DESHIELDSJR,OSCARW.,ALIKARA,ANDERDENERKAYNAK."DETERMINANTSOFBUSINESSSTUDENTSATISFACTIONANDRETENTIONINHIGHEREDUCATION:APPLYING

10. DOUGLAS, JACQUELINE, ALEX DOUGLAS, AND BARRY BARNES. "MEASURING STUDENT SATISFACTION AT A UKUNIVERSITY." QUALITY ASSURANCE IN EDUCATION 14.3 (2006): 251-267.HERZBERG'S TWO-FACTOR THEORY."INTERNATIONALJOURNALOFEDUCATIONALMANAGEMENT19.2(2005):128-139.

11. ELLIOTT,KEVINM.,ANDDOOYOUNGSHIN."STUDENTSATISFACTION:ANALTERNATIVEAPPROACHTOASSESSINGTHISIMPORTANTCONCEPT."JOURNALOFHIGHEREDUCATIONPOLICYANDMANAGEMENT24.2(2002):197-209.

12. EOM,SEANB.,H. JOSEPHWEN,ANDNICHOLASASHILL."THEDETERMINANTSOFSTUDENTS'PERCEIVEDLEARNINGOUTCOMESANDSATISFACTIONINUNIVERSITYONLINEEDUCATION:ANEMPIRICALINVESTIGATION."DECISIONSCIENCESJOURNALOFINNOVATIVEEDUCATION4.2(2006):215-235.

13. EMERY, CHARLESR. "AN EXAMINATIONOF PROFESSOR EXPECTATIONS BASEDON THEKANOMODELOF CUSTOMERSATISFACTION."ACADEMYOFEDUCATIONALLEADERSHIPJOURNAL10.1(2006):11.

14. GUOLLA,MICHAEL."ASSESSINGTHETEACHINGQUALITYTOSTUDENTSATISFACTIONRELATIONSHIP:APPLIEDCUSTOMERSATISFACTIONRESEARCHINTHECLASSROOM."JOURNALOFMARKETINGTHEORYANDPRACTICE7.3(1999):87-97.

15. HERZBERG,FREDERICK;MAUSNER,BERNARD;SNYDERMAN,BARBARAB.(1959).THEMOTIVATIONTOWORK(2NDED.).NEWYORK:JOHNWILEY.ISBN0471373893.

16. HUNT,H.KEITH,ED.CONCEPTUALIZATIONANDMEASUREMENTOFCONSUMERSATISFACTIONANDDISSATISFACTION.NO.77-103.MARKETINGSCIENCEINSTITUTE,1977.


17. KANO,NORIAKI;NOBUHIKUSERAKU;FUMIOTAKAHASHI;SHINICHITSUJI(APRIL1984)."ATTRACTIVEQUALITYAND

MUST-BE QUALITY". JOURNAL OF THE JAPANESE SOCIETY FORQUALITY CONTROL (IN JAPANESE). 14 (2): 39–48.ISSN0386-8230.

18. LIKERT,RENSIS(1932)."ATECHNIQUEFORTHEMEASUREMENTOFATTITUDES".ARCHIVESOFPSYCHOLOGY.140:1–55.

19. MARSH,HERBERTW.,ANDLAWRENCEA.ROCHE."MAKINGSTUDENTS'EVALUATIONSOFTEACHINGEFFECTIVENESSEFFECTIVE:THECRITICALISSUESOFVALIDITY,BIAS,ANDUTILITY."AMERICANPSYCHOLOGIST52.11(1997):1187.

20. MITTAL,BANWARI, ANDWALFRIEDM. LASSAR. "WHYDO CUSTOMERS SWITCH?THEDYNAMICSOF SATISFACTIONVERSUSLOYALTY."JOURNALOFSERVICESMARKETING12.3(1998):177-194.

21. PETRUZZELLIS, LUCA, ANGELA MARIA D'UGGENTO, AND SALVATORE ROMANAZZI. "STUDENT SATISFACTION ANDQUALITY OF SERVICE IN ITALIAN UNIVERSITIES."MANAGING SERVICEQUALITY: AN INTERNATIONAL JOURNAL 16.4(2006):349-364.

22. RAPERT,MOLLYINHOFE,ETAL."THEMEANINGOFQUALITY:EXPECTATIONSOFSTUDENTSINPURSUITOFANMBA."JOURNALOFEDUCATIONFORBUSINESS80.1(2004):17-24.

23. WITELL,LARS,ANDMARTINLÖFGREN."CLASSIFICATIONOFQUALITYATTRIBUTES."MANAGINGSERVICEQUALITY:ANINTERNATIONALJOURNAL17.1(2007):54-73.


7 Appendix

7.1 ModuleSurvey2016

NotApplicable

Module Survey Form

Date of Survey (dd/mm/yyyy) / /Code: E S

Title:

Subsection if applicable:

Your feedback helps us to improve the courses offered by the School. We review your comments andnormally provide summary conclusions within 2 weeks - outlining planned actions when appropriate.

Please show your opinion by making the appropriate box with a cross(if you change your mind, fill in the box and put a cross in the correct box)

5DefinitelyAgree

4MostlyAgree

3Neither Agreenor Disagree

2MostlyDisagree

1DefinitelyDisagree

1. Overall, the module content is appropriate

1a. The module is relevant to my degree

1b. The self-study required was about right

1c. The module content was interesting to me

1d. The applications of the topic are clear to me

2. Overall, the module is well managed

2a. The lecturer was adequately prepared

2b. The reading list material was readilyavailable2c. The module ran well (e.g. changescommunicated)2d. Supporting materials were made availableto me3. Overall, the module was well delivered

3d. The resources (e.g. handouts, slides) aregood

3c. The laboratory/seminar/class activitieshelped me to understand topics better

3b. The lecturer responded well to questions

3a. The lectures were engaging

4. Considering everything (content,management, and delivery), I am satisfied withthis {part} module.

Page 1 of 2

7981189171


7.2 SurveyReportallYears2016

ModuleSurveyResults

Collated_All_years.xlsx Page1of4 Report

Code: ES#### DateofSurvey: DD/MM/YYYY

TitleSubsection

NumberofRespondents 3260%ofcohort unknown

OverallSatisfactionwithModule[part]* 82% %agree

OverallSatisfactionwithModule[part]** 82% arithmeticmeanThechartbelowshowsthedistributionofanswerstoallquestions:

15_16_Term_2ModulePart

Question4givesanindicationoftheoverallsatisfactionwiththemodule.*Thepercentageofrespondentsscoring'DefinitelyAgree'or'MostlyAgree'**Thearithmeticmean

47%

52%

34%

30%

41%

44%

56%

44%

48%

43%

41%

33%

44%

20%

33%

36%

43%

38%

45%

43%

42%

40%

35%

37%

41%

39%

39%

36%

38%

23%

43%

46%

6%

7%

15%

17%

11%

10%

6%

14%

8%

13%

10%

19%

13%

16%

15%

11%

1%

2%

4%

7%

4%

2%

2%

3%

2%

3%

4%

8%

2%

5%

6%

4%

0%

0%

2%

3%

1%

1%

0%

1%

0%

1%

1%

4%

1%

2%

2%

1%

3%

0%

1%

0%

1%

3%

1%

2%

1%

1%

5%

1%

2%

33%

2%

1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

1:content

1a:relevance

1b:self-study

1c:interesting

1d:application

2:management

2a:preparation

2b:readinglist

2c:administration

2d:material

3:delivery

3a:engaging

3b:questions

3c:laboratoriesetc

3d:resources

4:Overall Satisfaction

QuestionAnswers

DefinitelyAgree MostlyAgree NeitherAgreeNorDisagree

MostlyDisagree DefinitelyDisagree NotApplicable/Blank


7.3 ResultsbyModule2016

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Code

Overall,the

module

contentis

appropraite

Themodule

isrelevantto

mydegree

Themodule

contentisat

asuitable

academic

level

Themodule

contentwas

interesting

tome

The

applications

ofthetopic

areclearto

me

Overall,the

moduleis

well

managed

Thelecturer

was

adequately

prepared

Theroom(s)

was

appropriate

forthe

module

Themodule

ranwell(e.g.

changes

communicat

ed)

Supporting

materials

weremade

availableto

me

Overall,the

modulewas

well

delivered

Thelectures

were

engaging

Thelecturer

responded

wellto

questions

The

laboratory

activities

helpedmeto

understand

topicsbetter

The

resources

(e.g.

handouts,

slides)are

good

Considering

everything

(content,

managementand

delivery)Iam

satisfiedwiththis

{part}module

1 1a 1b 1c 1d 2 2a 2b 2c 2d 3 3a 3b 3c 3d 4

Respondents %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied %Satisfied

1 27 27 ES3D1_2 96% 100% 93% 93% 100% 100% 100% 100% 96% 100% 100% 100% 100% 100% 100% 96%

28 53 26 ES4E2 96% 100% 96% 96% 96% 96% 100% 81% 96% 92% 92% 100% 100% 100% 100% 100%

54 69 16 ES4D8 94% 100% 75% 88% 81% 81% 94% 69% 88% 81% 81% 94% 100% 81% 69% 94%

70 116 47 ES386_1 87% 96% 91% 74% 85% 94% 100% 94% 94% 94% 91% 91% 89% 13% 83% 91%

117 289 173 ES183_MJC 95% 97% 83% 77% 83% 91% 94% 79% 89% 85% 88% 62% 81% 27% 81% 93%

290 301 12 ES335 83% 92% 92% 67% 67% 75% 83% 67% 92% 83% 67% 42% 75% 67% 58% 67%

302 340 39 ES386_JRK 95% 92% 67% 59% 87% 87% 90% 74% 97% 87% 82% 79% 92% 26% 87% 85%

341 378 38 ES438 95% 89% 79% 89% 97% 95% 97% 66% 92% 82% 95% 92% 97% 74% 87% 97%

379 405 27 ES3D7 89% 96% 74% 85% 93% 93% 96% 70% 96% 85% 89% 93% 93% 85% 93% 96%

406 560 155 ES174_2 90% 92% 46% 57% 85% 86% 87% 66% 90% 69% 77% 50% 63% 3% 63% 81%

561 581 21 ES3B2 100% 100% 95% 100% 100% 100% 100% 95% 100% 95% 100% 95% 95% 33% 95% 100%

582 589 8 ES97E 88% 88% 88% 88% 100% 100% 100% 100% 88% 88% 100% 88% 100% 88% 100% 88%

590 603 14 ES3A7 100% 100% 86% 93% 100% 100% 100% 93% 100% 93% 100% 100% 100% 64% 100% 100%

604 607 4 ES97G 100% 100% 75% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 100% 100%

608 620 13 ES4A4 100% 85% 92% 92% 85% 100% 100% 69% 100% 85% 85% 92% 92% 38% 100% 92%

621 647 27 ES2A5 100% 93% 81% 93% 89% 96% 100% 70% 93% 85% 89% 74% 89% 30% 78% 81%

648 694 47 ES386 87% 96% 91% 74% 85% 94% 100% 94% 94% 94% 91% 91% 89% 13% 83% 91%

695 732 38 ES4D4 97% 97% 87% 84% 92% 92% 100% 79% 97% 95% 95% 92% 97% 79% 92% 97%

733 739 7 ES97Eb 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0% 100% 100%

740 774 35 ES3D9 91% 91% 80% 83% 100% 100% 100% 91% 97% 91% 100% 97% 100% 83% 94% 100%

775 785 11 ES4C3 91% 100% 100% 82% 91% 100% 100% 91% 91% 91% 91% 82% 82% 18% 91% 91%

786 799 14 ES94Y_1 100% 100% 100% 93% 100% 79% 100% 100% 100% 86% 86% 93% 100% 0% 86% 71%

800 814 15 ES94Y_2 73% 93% 73% 100% 93% 73% 93% 100% 93% 87% 73% 93% 93% 0% 87% 80%

815 916 102 ES185_2 94% 85% 75% 62% 79% 94% 96% 77% 87% 83% 90% 76% 94% 63% 62% 84%

917 929 13 ES3C8_2 85% 77% 69% 62% 69% 77% 46% 77% 100% 85% 46% 54% 85% 23% 46% 54%

930 957 28 ES2B0_2 89% 93% 79% 82% 93% 93% 100% 79% 93% 86% 79% 68% 79% 61% 86% 79%

958 998 41 ES2B3_2 88% 80% 66% 83% 90% 78% 88% 73% 88% 63% 85% 83% 73% 15% 80% 88%

999 1000 2 ES176 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 50% 100% 100%

1001 1034 34 ES3D9_2 97% 88% 79% 76% 100% 88% 97% 100% 97% 97% 91% 76% 94% 97% 85% 91%

1035 1049 15 ES4A4_2 100% 93% 100% 100% 100% 100% 100% 80% 93% 87% 100% 100% 100% 80% 93% 93%

1050 1055 6 ES97E_3 100% 100% 83% 100% 100% 100% 100% 83% 67% 100% 100% 100% 83% 17% 100% 100%

1056 1066 11 ES4E6 82% 91% 82% 100% 91% 73% 100% 100% 100% 55% 73% 73% 100% 36% 73% 82%

1067 1111 45 ES177 87% 91% 91% 84% 89% 93% 100% 51% 91% 64% 87% 53% 80% 93% 82% 96%

1112 1186 75 ES2B5 93% 96% 75% 87% 89% 87% 93% 89% 95% 83% 87% 72% 87% 52% 73% 96%

1187 1201 15 ES96V 80% 93% 47% 87% 87% 53% 87% 33% 73% 27% 80% 73% 93% 87% 27% 73%

1202 1217 16 ES96Q 94% 100% 94% 88% 94% 94% 100% 94% 81% 81% 88% 94% 100% 94% 75% 88%

1218 1240 23 ES3D5 91% 91% 83% 74% 91% 70% 83% 70% 83% 78% 91% 52% 65% 74% 83% 87%

1241 1278 38 ES4D5_1 92% 89% 79% 82% 92% 87% 97% 68% 89% 84% 82% 71% 87% 50% 92% 89%

1279 1298 20 ES4D5_2 90% 90% 80% 60% 85% 60% 80% 50% 60% 65% 65% 60% 85% 75% 70% 70%

1299 1321 23 ES441 57% 96% 30% 61% 61% 35% 43% 70% 65% 61% 22% 30% 43% 61% 30% 26%

1322 1590 269 ES187_2 93% 89% 81% 73% 86% 97% 99% 83% 95% 90% 95% 89% 94% 72% 93% 93%

1591 1636 46 ES4D4_B 93% 98% 80% 83% 93% 87% 100% 85% 93% 98% 78% 61% 80% 33% 91% 91%

1637 1663 27 ES4D9 89% 85% 56% 78% 89% 74% 100% 85% 93% 89% 74% 85% 85% 56% 74% 81%

1664 1778 115 ES185_b 79% 79% 52% 41% 63% 41% 45% 64% 39% 57% 38% 23% 51% 50% 30% 41%

1779 1924 146 ES183_b 81% 82% 55% 49% 67% 60% 81% 55% 71% 68% 50% 34% 59% 24% 60% 52%

1925 1945 21 ES3D2 100% 100% 95% 81% 90% 86% 95% 86% 90% 95% 95% 86% 90% 90% 81% 90%

1946 1972 27 ES2A9 93% 89% 89% 59% 70% 74% 93% 96% 96% 89% 70% 41% 85% 26% 74% 74%

1973 2020 48 ES2B4_2 75% 88% 79% 81% 67% 73% 92% 94% 88% 71% 60% 60% 63% 60% 58% 67%

2021 2080 60 ES3C3 85% 92% 80% 43% 73% 73% 88% 90% 77% 82% 50% 30% 63% 5% 57% 62%

2081 2121 41 ES4C9 100% 100% 95% 93% 98% 90% 98% 95% 98% 93% 90% 90% 100% 88% 83% 98%

2122 2128 7 ES4D2 100% 86% 86% 86% 86% 100% 100% 100% 100% 100% 100% 100% 86% 14% 86% 100%

2129 2153 25 ES4D6 92% 80% 92% 68% 88% 88% 100% 36% 84% 80% 92% 80% 100% 92% 72% 84%

2154 2167 14 ES442 93% 93% 86% 86% 79% 93% 93% 79% 100% 86% 100% 64% 100% 7% 79% 86%

2168 2195 28 ES3820 82% 82% 86% 39% 71% 79% 96% 79% 75% 82% 68% 64% 86% 11% 82% 86%

2196 2448 253 ES184 97% 99% 92% 82% 89% 89% 100% 88% 94% 92% 87% 75% 85% 42% 83% 91%

2449 2463 15 ES3C5 87% 93% 87% 87% 93% 93% 87% 93% 100% 100% 80% 73% 87% 60% 93% 87%

2464 2540 77 ES2B1 81% 73% 78% 44% 56% 65% 74% 74% 75% 70% 56% 35% 69% 35% 53% 56%

2541 2556 16 ES3D1 88% 100% 75% 44% 63% 75% 81% 94% 88% 88% 56% 25% 69% 69% 81% 63%

2557 2589 33 ES4A8 94% 97% 97% 94% 97% 94% 100% 94% 97% 97% 94% 94% 100% 33% 91% 97%

2590 2736 147 ES183 99% 99% 91% 86% 93% 97% 99% 90% 95% 97% 95% 92% 89% 18% 86% 97%

2737 2905 169 ES174 84% 85% 82% 58% 68% 85% 88% 84% 92% 75% 80% 53% 61% 21% 72% 74%

2906 2931 26 ES3D6 96% 100% 85% 88% 88% 96% 100% 100% 96% 96% 96% 100% 100% 8% 96% 100%

2932 2936 5 ES97b 80% 80% 80% 100% 80% 100% 100% 80% 100% 100% 100% 100% 100% 80% 100% 80%

2937 2973 37 ES3D2_1 100% 100% 97% 84% 89% 95% 100% 92% 97% 95% 97% 97% 100% 0% 89% 100%

2974 2990 17 ES4B6 100% 82% 100% 94% 100% 82% 94% 100% 94% 76% 100% 100% 88% 29% 82% 100%

2991 3054 64 ES2B2 81% 73% 86% 63% 88% 91% 94% 95% 92% 77% 80% 77% 86% 11% 80% 88%

3055 3078 24 ES3B60_3 100% 100% 100% 79% 88% 79% 92% 92% 88% 71% 79% 67% 92% 29% 79% 88%

3079 3115 37 ES1810 41% 62% 41% 57% 51% 32% 54% 57% 51% 35% 32% 30% 54% 8% 27% 24%

3116 3146 31 ES440 97% 90% 90% 81% 84% 87% 97% 87% 90% 81% 90% 87% 87% 90% 81% 84%

3147 3162 16 ES4C8 56% 75% 63% 56% 81% 63% 63% 94% 100% 75% 63% 56% 75% 88% 75% 63%

3163 3203 41 ES2B4 90% 83% 85% 88% 73% 76% 98% 98% 93% 76% 68% 85% 78% 66% 68% 73%

3204 3220 17 ES3A3 71% 88% 53% 59% 41% 47% 41% 82% 65% 65% 41% 35% 47% 47% 24% 35%

3221 3260 40 ES4B5 95% 93% 93% 88% 95% 95% 93% 98% 95% 98% 78% 50% 95% 95% 85% 98%

3261 3261 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3262 3262 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3263 3263 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3264 3264 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3265 3265 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3266 3266 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3267 3267 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3268 3268 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3269 3269 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

3270 3270 1 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

0%

20%

40%

60%

80%

100%

120%

%Satisfied

pcapp negotiated project pcapp_clucas.docx 3 of 35 copyright © 2016 - by c. luacs

Documents