a developmental evaluation framework for the city of vancouver solutions lab · 2020-04-07 · a...
TRANSCRIPT
A Developmental Evaluation Framework for the City of Vancouver Solutions Lab
Maggie Low
Greenest City Scholar Final Report
September 15th, 2017
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab
GreenestCityScholarFinalReport,August2017
1
What’stheCityofVancouverSolutionsLab?
TheSolutionsLabisanewapproachtoseekingbreakthrough,transformativesolutionstosomeofthecity’smostcomplex
problems.It’saplacewhereCitystaff,communitymembersandstakeholderscollaboratetodeeplyunderstandcomplexchallenges
fromthepointsofviewofthepeoplemostaffectedbythem,andwherewerapidlyprototypeandtestinnovativeresponsestosee
whatwecanlearnthroughco-creationandsomerisktaking.It’sanexcitingplacewherewedialogueandlistendeeply,trynew
processesandcollaborationtools,andlearnandhavefuntogether.TheCityofVancouverSolutionsLabisusingacombinationof
TheoryUandhuman-centreddesigntogetatbothpersonal/culturalandsystemstransformationsandtoreachbreakthrough
solutions(pleaseseeFigure1.below).
TheCityofVancouverSolutionsLabmodelispartofabroaderpictureof“publicsectorinnovationlabs”(PSIL’s)beingdevelopedin
otherplacesaroundtheworld,especiallyinEurope.PSIL’stakeavarietyofformsandfocusareas;theywillvaryintheirpurpose,
theoryofchange,operatingmodel,methodsandimpact.WhatPSIL’sor“Labs”dohaveincommonistheirstrongaction-reflection
processesandtheyregularlyreviewtheirimpactsandevolvetheirstructure,governanceandfundingmodels,leadership,methods
andsharetheirlearning(Westley,2016).
TheSolutionsLabiscurrentlyinasoftlaunchstage(phase2),withcoremethods,operatingmodel,andappropriatetypesofLab
challengesandteamsallbeingdevelopedbyworkingonfourcomplexchallengesintheLabcontext.FromMaythroughmidAugust
2017,IheldtheGreenestCityScholarpositionandwastaskedwithsupportingtheLabManagerwiththerolloutofLabprocesses
anddevelopinganinitialdevelopmentalevaluationframeworktoevaluatetheimpactoftheLab(moredetailsaboutdevelopmental
evaluationonpg.5).TheSolutionsLabwasinitiatedbyanactiononCollaborativeLeadershipintheHealthyCityStrategy,andhas
onestafflead,aSteeringCommitteeandseniorCitystaffleadership.Todate,communitymembersandstakeholdersonspecific
LabteamshaveconsistedofstaffandvolunteersfromNeighbourhoodHouses,localNGOsandnon-profits,artsorganisations,
TourismVancouverandBusinessImprovementAssociations(BIAs).ThepurposeofthisreportistosharewhattheCityof
VancouverSolutionsLabhasbeendoingandwhy,howwe’vebeenevaluatingouractivitieswithDevelopmentalEvaluationand
basedonourevaluations,whatkindsofquestionscouldwebeaskingforthenextiterationoftheLab.Thehopeisthatthe
SolutionsLabwillofferCItystaffandcommunitypartnersviablewaystoreachdifferentkindsofsolutionstothesustainability
challengesweallface.Morebroadly,solutionsfoundinVancouvercanpotentiallybescaledtocitiesgloballyduetoVancouver’s
leadershippositionandgrowingreachandimpact.
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
2
SummaryofActivitiesTimelineofSoftLaunch(Phase2)ThesoftlaunchphaseoftheSolutionsLabhasbeenunderwaysinceJanuary2017.ThisreportconsistsofactivitiesandfindingsfromMaytoAugust2017whentheLabManagerandIputtheLabintopracticebyconvening4differentCityproblemsintheSolutionsLabprocessesthatareunderdevelopment.PleasenoteagreatdealofworkwasdonebytheLabManagerandSteeringCommitteebeforemypositionbeganinMay(pleaseseeFigure2)andasummaryofeachLabproblemisinTable1below.
Figure1.(left)AdaptedfromTheoryU(Scharmer,2016)TheoryUisaprocessthatallowsustolearnbynotonlyreflectingontheexperiencesofthepastbutalsolearningfromthefutureasitemerges.Figure1.depictsTheoryUasaprocessthattakesadeepdiveintoaproblemby“lettinggo”ofwhatwethinkweknowandre-emergingwithdifferentsolutionsthanwouldhaveotherwisebeenreached.Human-centreddesignisaprocessusedbydesignersthatinvolvesbuildingempathywiththepeopleyouaredesigningfor.Itstartswiththeusersandendswithnewsolutionsthatarecustommadetosuittheirneeds.
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
3
Figure2.TimeandKeyActivitiesofSolutionsLabRecentActivities(January2017-August2017)ForeachindividualLabproblem,aLabteamconsistingofCitystaffandcommunitypartners-thoughtfullycuratedbytheLabManagerandLableaders-areledthroughtheSolutionsLabprocessesbasedonTheoryUandhuman-centreddesignprinciples.Theindividualsessionsare1-2daylongworkshopsandhavebeenoccurringover4-6weekperiodsdependingonCitystaffschedules.InthiscurrentiterationoftheLab,eachstageofTheoryU(i.e.Co-initiating,Co-sensing,Presencing,etc.)hashadit’sownsessionwiththesameteammembersinvitedtoparticipateineverystage.Thesessionsarerunbyexperiencedfacilitatorsandheldinlocationsoutside“City”meetingroomssuchasCityStudio,StanleyParkandtheQueenElizabethTheatre.BelowisadescriptionoftheLabsthatarecurrentlyunderwayaswellasotherproblemsthatareintheinitialdevelopmentstages(i.e.-beforeaLabteamiscuratedandbegintheLabprocess).Table1.
Project ConveningQuestion/Purpose Status
EngineeringCustomerServiceLab
HowmightweimprovecustomerexperiencesinEngineeringServices?Wewillusethefrontdeskasafocalpoint,andusewater+sewerpermitsandmajorprojectinterruptionstostreetsasexamples.
Earlyco-creating/prototyping
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
4
PublicEngagementLab Howmightwebemoreconsistent,collaborative,andvalues-alignedinourpublicengagementworkattheCity?
Movingintopresencing+co-creating
PublicSpace+PublicArtLab
HowmightwebuildacultureofcollaborationbetweentheCityandcommunitypartnerstounlockfurtherpotentialforcreativeandengagedcity-buildingthroughourpublicspaces.
Movingintopresencing+co-creating
CommunityConnections+ResilienceLab
Howmightweincreaseresilience,connectedness,andbelonginginaneighbourhoodwithgrowingdiversityandincreasingdensification?
Co-initiating
WhatisDevelopmentalEvaluation?Developmentalevaluation(DE)isafieldofevaluationmeanttoassistsocialinnovatorscreate-orradicallyadapt-aprogrambasedontrialanderror.DEisdifferentfromothertypesofevaluations;whereassummativeevaluationshelpjudgethevalueofaprogramandformativeevaluationsassistintheimprovementofaprogram,DEevaluateswhatthenextstepsareinthecreationofanemergingprogram.DEasksquestionssuchas:Whatisdeveloping?Whataretheoptionsforthenextiterationoftheprogram?(Cabaj,2014).ThedifferenceisthatinDEtheturnaroundbetweenquestionasking,questionanswering,questioninterpretationanduseforactionoftenhappensinshort,iterativeandongoingcycleswithfocusofdevelopment(Patton,2011,p.232).AnotherwaytothinkaboutDEisas“realtimestrategiclearning”wheretheevaluatorgathersrealtime-datatoinformongoingdecision-makingandadaptationstotheinnovationsbeingimplemented.Patton(2011,p.2)explains:“Theevaluatorisoftenpartofadevelopmentalteamwhosememberscollaboratetoconceptualize,design,andtestnewapproachesinalong-term,ongoingprocessofcontinuousdevelopment,adaptation,andexperimentation,keenlysensitivetounintendedresultsandsideeffects.Theevaluator’sprimaryfunctionintheteamistoinfuseteamdiscussionswithevaluativequestions,thinking,anddataandtofacilitatesystematicdata-basedreflectionanddecisionmakinginthedevelopmentalprocess.”Simplyput,thepersondoingtheevaluatingisoftencloselyinvolvedinthedevelopmentprocessesoftheinnovativeprogram.TokeepdevelopmentalevaluationoftheSolutionsLabsimple,effectiveandtimelythe“What,SoWhat,NowWhat?”inquiryframeworkhasbeenusedbytheSolutionsLabteamtodate(pleaseseeFigure2).Thismodelsimplyappliesbasicevaluative
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
5
thinkingtoinquireaboutsocialinnovationsanditisanapproachusefultothosenewtoevaluation.Patton(2011,p.232)startswiththefollowingquestionswhenusingDEinanewenvironment:Whatistheinnovationnow?(What?)Whatdotheresultsofinnovationeffortsmeannow?(Sowhat?)Whatdothefindingsrevealaboutnextstepsnow?(Nowhat?)
Figure2.(left)AdaptedfromDevelopmentalEvaluation(Patton,2011)TheSolutionsLabisusingdevelopmentalevaluation(DE)becausetheLabisinadiscoveryphaseanditisconstantlyinmotion.TheSolutionsLabisbeingbuiltaswegoandthereisalotoflearningtodoastheprocessesunfoldinrealtime.Inotherwords,theteamhasbeenintegratingreflectionandadaptationintolabpracticesateverystep.NowweturntothemethodsusedandkeylearningfromPhase2.
Whathavewebeendoingandwhy?AstheSolutionsLabunfolded,DEhelpedtoevaluatethecorepurpose,methods,operatingmodel,andimpactoftheLabfortheCityofVancouveraswellasforitscommunitypartners.OurDEinvolved“trackingemergentandchangingrealities,illuminatingperspectivesandrealities,andfeedbackintomeaningfulfindingsinrealtime”(Patton,2011,p.3).
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
6
Wehavebeenperformingactionresearchtodothisrealtimelearningandusingthe“What,SoWhat,NowWhat”inquiryframeworktoinformPhase2betweenJuneandAugust2017.Specifically,Iwasintenselyinvolvedinthedeliveryof3outofthe4Labproblems,andbydoingso,wasabletocapturerealtimedatatohelpinformeachphaseoftheLabanddocumentimportantlearningrelevanttootherLabs(pleaseseeFigure3.foranexplanationofhowDEisandwasusedattwodifferentscales).AsperCabaj(2014,p.3),theevaluationdesignandmethodshavebeen“diverse,highlyadaptive,evolvingandlightweightdesignsandemphasisonseekingpatternsacrossexperimentstoinformstrategy.”Specifically,theDEinquiryframeworkof“What?”“SoWhat?”“NowWhat?”hasbeenusedtotestthecoredesignandmethodsusedintheLab(i.e.TheoryU),conveningquestions,Labteamcomposition,facilitatorcombinations,spacerequirements,theoveralloperatingmodelandtheroleoftheLabManager.Wehavebeenpayingcloseattentiontowhatishappeningbefore,duringandaftertheLabsessions(pleaseseeFigure1forarefresherofLabprocesses)tomakesenseofwhatishappening,whyit’simportant(ornot)andifandhowwewanttodoitnexttime.Thistypeofevaluationhasbeenbasedonthefollowingrealities:● Conductedpre-interviewswiththeCommunityConnections+ResilienceLabteamleadsaskingthemtoreflectontheir
expectationsofthelab● Participatedinthedesign,implementation,facilitationandwriteupof3outofthe4labquestions● ParticipatedinthepreparationanddebriefwiththefacilitatorsandLabManagerfor3outofthe4labs● Preparedandcollectedfeedbackfromlabparticipantsattheendofeachsessionandincorporatedthisfeedbackintothe
planningofnextlabs● Assistedwiththewriteupof3labreports● Tookdetailednotesandpicturesduringthelabsessions● ConsultedwithDEexpert,MarkCabaj,andreadrelevantDEliterature
Muchhasbeenlearnedontheoperationalscale(bothwithineachLabandacrossLabs)andwearestartingtoprobeandgleaninsightsintowhatthelargerimpacttheLabcouldhavefortheCityasawhole.BelowisasummaryofkeylearningfromtheLabactivitiessofar.PleaseseeAppendixAforasamplefeedbackformusedattheenddayforeachLabsession.
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
7
Scale1:WithinandBetweeneachofthe4LabsDEwasusedtoevaluateeachphaseofindividualLabs(i.e.EngineeringCustomerServiceLabORPublicEngagementLab)bycollectingrealtimedataandeitheradaptingtochallengesandquestionsastheyaroseorbytakingnoteofoutcomesofourdecisionsinoneLabandusingwhatwelearnedinthedesignofotherLabs.ThegraphictotheleftdepictsanexampleofonephaseofaLab(Co-sensing)thatcanbeevaluated.AllotherphasesoftheLabwerealsoevaluatedusingDE.Example#1:TrackingwhoandwhenLabteammembersattendtheLabsessionhasbeenaninterestingchallenge,forexamplewhennewpeoplewereinvitedtoparticipateintheco-sensingpartoftheLabwithoutbeingpartoftheco-initiatingprocess.Initiallyitwasthoughtthenewparticipantswouldbeanunhelpfuldistraction.However,withminortweakstothe“re-cap”sessiononthefirstdayoftheco-sensingsession,newlabparticipantswerebroughtuptospeedandaddednewperspectivesandenergytotheroom.
Scale2:TheSolutionsLabprocessesasawholePreliminaryideasabouthowtomeasuretheimpactoftheLabhavebeguntosurfacefromseveralplacesincludinglessonslearnedfromDEineachLab,debriefmeetingswithfacilitators,reflectionsfromSteeringCommitteemembersandmeetingswithseniorCitystaff.AtthisphaseitwouldbeusefultotakestockofwhatwehavelearnedanddevelopaconcreteevaluativeframeworktomeasuretheimpactoftheLabintermsofbreakthroughs,transformationsandrelationships.Pleaseseebelowfordetaileddefinitions.Example#2:ThroughouttheLabprocess,thetimecommitmentrequiredbyCitystaffandcommunitypartnershasbeensubstantial.WeshouldaskandtestwhetherpartsoftheLabcanbeusedfordifferentkindsofproblems.DoeseachproblemhavetogothroughtheentireUtoreachdifferentsolutions?
Figure3.DevelopmentalEvaluationusedatdifferentscales
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
8
KeyLearningfromPhase2oftheSolutionsLab:
● Whoisontheteamreallymatters.○ Theteamleadsneedtobe“pioneers”.Thesearepeoplewhoonlyneedtoknowthewhy,andnotnecessarilythe
“what”orthe“how”.Theteamleadswillparticipateintheprocessinvariousways,frombeingastrongervoiceinthe
roomtomoreofaquietyetencouragingvoice.Differentvariationsofthisspectrumwillwork,whatisimportantis
thattheteamleadsareadvocatesoftheprocessandarewillingtoparticipatewithopenmindsandhearts.○ Diversityreallymatters!Participantsmadeveryclearthatdiverseperspectives,educationalbackgrounds,worldviews,
andethnicitiesareextremelyimportanttoanauthenticLabexperience.TheLabManager,supportandteamsleads
needtotakeaccountforthistothebestoftheirabilityfromtheverybeginningoftheprocess.○ Findingthe“pioneer”staffleads,andtheproblemswiththerightamountofurgencyandcomplexitytothem,takes
timeandintentiontogetitright.
○ Teammembersdon’tnecessarilyneedtocommittothewholeprocessrightaway;theycansayyesatthebeginning
butthingschange.Itwasusefultoaskforeachperson’scommitmentattheendofthe“Co-initiating”workshopasall
participantsexpressedinterestandcommitmentingoingfurtherintheprocesstogether.● Timeisprecious.
○ WearelearningtodesigntheLabprocesswithpeople’stimeandschedule’sinmind,whilestillfacilitatinga“deep
dive”intotheproblem(i.e.designsprintand/orshortersessionoverafewweeks,allowingforacertainamountof
coming/goingoflabteammembersduringsessions).ThemoreintensedesignsprintformatusedinoneoftheLab’s
allowedustogothrough(almost)thewholeprocessandcometoconceptsforprototypes.Whilestressful,itwas
importantforustodoitthiswaybecausewelearnedusefulinformationaboutwhenandhowtonarrowthescopeof
theproblem,andhowchallengingitcan(andwillbe)togettothe“Co-embodying”phase.○ Planningaroundpeople’stimeandschedulesisalwayschallenging;duringthesummermonthswe’vebeentryingto
workaroundvacationtimeandstilltryingtohitthesweetspotoftiming(i.e.-1,2or3weeks)betweeneachphase
intheLabprocess.Iftoomuchtimelapsesbetweeneachsession,doestheteamlosesteamandinterest?Sofarthe
answerisno,theLabteamsseemgenuinelycommittedtotheentireLabprocess.
● Productivetension?(PressurecookermetaphorfromMarkCabaj)○ Thecook(beingthefacilitationteam)hascontroloftheturningtheheatupordown.Ifthepressuregoesbeyondthe
carryingcapacityofthevessel,thepressurecookercanblowup.Ontheotherhand,withnoheatnothingcooks.We
don’twanttoeliminatestress/tensioncompletely.Wehavebeenlearningtomanagethesetensionsastheycomeup
intheLab:
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
9
■ Outcomesversusprocess:Thereisastrongtensionbetweenprocessandoutcomes;weallknowachieving
tangibleoutcomesisimportantforamunicipalgovernment,sohowmuchtimecantheLabfocusonthe
“deepdive”processwithoutlosingthegoalofcomingtotestableprototypes?■ Consensusversusdebate:howtobalancecollegialandopenspacewithdiscomfortandconflictingviews■ Supplyversusdemandoftime:howmuchtimeistoomuch,ortoolittle?Shouldwebespendingmoretime
unpackingthelearningjourneys?● Everyonelovesalearningjourney.
○ Maybenoteveryonelovesthispartofco-sensing;however,it’sbeenaninvaluablepartoftheLabprocessintermsof
pushingCityStafftoseetheirworkthroughthelensofthosemostimpactedbyit.It’smorethan“consultationwith
stakeholders”;it’saboutlistening,observingandaskingquestionstogainanewperspectiveontheproblem,and
thentakingthoseinsightsbacktotheideationstage.● Facilitatorsarekey,soweneedmore.
○ DifferentconsultantshavedifferentabilitiestofacilitatethecurrentSolutionsLabframeworkandbringtheirown
specifictool-kitstoit.Therecurrentlyseemstobeenoughcapacityintheconsultantecosystemtocreativelyfacilitate
theUprocess;howevertheremaybefewerconsultantsinVancouverthatcanactually“build”avarietyofdifferent
typesofprototypes.● LabManagerasaconvener,withalittlebitofhelp.
○ TheroleofLabManagerhasbeentoconvene,organize,coordinate,andhelpfacilitatetheLabinitsentirety.Thisrole
hasbeenaboutmaintainingrelationships,bridgingtoimplementation,andensuringasafespaceinwhichLabprocess
canoccur.Facilitationconsultantscanthenprovidemoreuncomfortablepushesifneeded,withoutriskingthese
relationships.TheLabManagerholdsacertainsenseofaccountabilityto/fromteamparticipants.Thismodelhas
workedwellsofar.
○ Having4problemsintheLabatonetimehasbeenmanageableandcancertainlycontinueaslongasthereis
supportstaff(i.e.-eachlabtakesquiteabitoftimetoplan,unfoldandwrapup).
● It’sallaboutthefood...andspace.○ It’sbeencrucialforLabstobeoutofregularCoVmeetingspacesandcontextsandthereforewe’vebeenmoving
aroundcityintodifferentneighbourhoods,accessingoutsidespace(i.e.THNKHome,ParkBoard/StanleyPark,Art
Starts,CityStudio,SlocanParkFieldHouse).Weprovidenicefood,warminvitations,andpersonaltouchesasmuchas
possible.Peopleappreciatethisasthisdemonstrationofcare,andhonouringpeople’sparticipationhasbeen
importanttocreatingagraciousspace.○ Thislevelofdetailhasalsobeenlogisticallychallenging(i.e.carryingboxesofsuppliesonourbikes).Itisworthwhile
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
10
havingapermanentLabspace(thatisbrightandwithakitchen)ifitcanprovidean“outofCoV”experience.● WhatmakesaLabaLab?
○ WeneedmoretimeintheLabtofigureout“whatisaLabandwhatisnotaLab”?It’sanimportantquestionthat
keepscomingforwardfromtheLabSteeringCommitteeandseniorCitystaff.Wemayneedtothinkabouthowwe
canapplywhatwehavelearnedsofarelsewhereineasy/simpleways.Canalabbeone½daysessionthatoffersa
creativepush?Doesitneedtobeaseriesofin-depthworkshopstogetatbreakthroughsolutions?
○ DevelopmentalEvaluationhasbeenhelpfultogetustoshapeandrunwhatisseeminglikeawellroundedLab
process,thoughastheLabmovesintoitsnextiterationotherDEinquiryframeworksmaybenecessarytotrackthe
fullimpactoftheLabintermsofbreakthroughs,transformationandrelationships.Howdowemeasuretheoverall
impactoftheSolutionsLabfortheCityofVancouver?Thiscouldincludefollowing“keyinformants”throughtheLab
processfrombeginningtoend.PleaseseeTable2.below.
What’snextfortheSolutionsLab?
AstheLabcontinuesintothefall2017,3outofthe4LabswillmoveintotheCo-creationandCo-embodyingsessionsandpotentially
2newproblemswillbetakenonbytheLab.Astheproblemsbecomemorecomplex,itwillbeimportanttocontinuetoadaptthe
overallpurposeoftheLabwiththeLabManager,SteeringCommittee,andDeputyCityManager.AstheLabunfoldedinpractice,
DEhashelpedmakethecasethatmeasuringtheLabintermsofbreakthroughs,transformationsandrelationshipsisclearand
simple,andalsoconveysambitionandastrongcalltoaction.Howmightweunderstandandmeasuretheseimpacts?
MuchoftheinitialthinkingabouthowtomeasuretheoverallimpactoftheLab,oralsoknownastheLab’s“theoryofchange”took
placebeforemyGreenestCityScholarpositionbeganinMay2017.PleaseseeAppendixAforaninitiallistof“UnitsofAnalysis”to
useaspotentialindicators.Todate,thefollowingdescribespossiblewaysofmeasuringtheoverallimpactoftheLabintermsof
breakthroughsolutions,transformation,andrelationships:
Breakthroughs(akagettingtodifferentkindsofsolutions):Innovation,risk-taking,experimentation,action,complexproblems,prototyping,failure-as-learning,creativedisruptionTransformations(akapeopleshowingupdifferentlyandseeingthemselvesasagentsofchange;transformingthecultureandsystemsthatwe’reworkingwithin):Systematic,personal,leadership,action/reflection,developmental,intrapreneurship,newmetaphors+stories
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
11
Relationships(akabreakingdownsilos):Diverseperspectives,collaborative,empathy,newinsights,wholesystemintheroom,deeplisteningPotentialmethodstomeasurethethreeareasofimpactinclude:
Table2.
Method PotentialQuestions
UserProfileWhatwillyouwanttoknowabouttheoverallimpactoftheLab?
Identifyprimaryusersoftheevaluation
andcompleteuserprofileasperCabaj
(2014).
Organizeandprioritiseusercasesto
ensuretheevaluationispurposefuland
relevant.Usetheuserprofiletoorient
DEtotracklargerimpactof
breakthroughs,transformationsand
relationships.
1)Howwouldlifebebetterifthisprojectworkedwell?
2)Whatisitaboutthelabthatyouthinkwillmakeadifference?
3)Howwillthelabovercomelikelychallengesandbarriers?
PleaseseeAppendixCforotherpotentialinterviewquestionsto
completetheuserprofile.
Pre+postinpersoninterviewsWhatarecoreteammemberslearningbygoingthroughtheLab?Arestaffseeingthemselvesin
Conductpreandpostinterviewswith
coreteammembersforSolutionsLab
problems.Preinterviewswillaskeach
intervieweetoreflectontheirmindset
goingintothelab.Postinterviewswill
askeachintervieweetoreflectonwhat
theyfeelwouldnothavehappened
withoutthelab.Thepreandpost
interviewsareintendedtogleana
deeperunderstandingofhowthecore
methodsusedwithinineachlabare
PotentialPreInterviewQuestions
Howdoyoucurrentlyperceive/understandtheproblem?
WhatareyourhopesandfearsfortheSolutionsLab?
WhataresomeofthebarrierstoinnovationfortheCity?(Tryto
getpeopletoseetheyhavemoreagencyinthesystemor
somethingthatgetsthemtothinkaboutroleinbiggersystem.
Provideanexample…)
HowdoyouthinktheSLabwillhelpwiththisproblem?
PotentialPostInterviewQuestions
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
12
moreexpansiveways?
working,whyorwhynot.WriteupallreflectionsandpreandpostinterviewdatainexcelsheettotrackandcodethepotentialchangestakingplacethroughouttheSLabprocesses.
Howdidthelabhelpchangethisperception?Inyouropinion,whatwasthebiggestchallengeyouovercameinthisprocess?Whatwouldyouhavedonedifferently?Whatwouldhavenothappenedwithoutthelab?
PreandpostLabonlinesurveysPromptstogetatmindsetsbeforecomingintothelabaswellasmindsetsaftertheLabexperience.
PrepareashortonlinesurveytosendtoallLabteammembers1-2daysbeforetheLabbegins.ThiswillgiveyouasenseofwhattheteammembersknowandfeelabouttheLab,andhowtheyperceivetheconveningquestion.Sendthesame,orsimilaronlinesurveytoeachLabteammemberafterthefullLabhasbeencompletedtogleanifhowtheirunderstandingoftheconveningproblemchangedthroughouttheLabprocessandwhy.
ThefollowingpreLabonlinesurveyquestionswereusedfortheCustomerServiceinEngineering:Inyouropinion,howimportantisittoimprovecustomerserviceinEngineering?Inyourexperience,whatarethetopthreeissuesor‘areasoffocus’forimprovingcustomerserviceinEngineering?Whatarethemostusefulandeffectivetools,equipment,processes,etc.thatareavailabletoyou,andenableyoutoprovidegreatcustomerservice?Whatarethemostfrustratingtools,equipment,processes,etc.thatgetinthewayofprovidinggreatcustomerservice?WhenEngineeringhasimprovedourcustomerservice,whatdoesthatLOOKandFEELliketoyou?WhataresomeofthebarrierstoinnovationinEngineering?WhataresomeofthebarrierstoinnovationattheCityingeneral?
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
13
BiggerQuestionsfortheSolutionsLab
BelowarethemainquestionsIwouldrecommendtheLabfocusoninitsDEworkmovingforwardintothePhase3andbeyond:
Labpurpose● DoestheLabneedanidentity?● Whatarethebasicrequirementsforworktobedonein“theLab”?(i.e.itmustbeacomplexproblem,itmusthavea
committedstaffleadwhohastheproblemontheirworkplan.)● HowcantheSolutionsLabbesthelpachievecitypriorities?● What,ifany,partsofspecificLabprocessescanbeleveragedindividuallytobenefitotherprojectsandprograms?● Howcanweiterateourtheoryofchange?
Impact● Tomeasureimpactshouldweincreasethesamplesize?(i.e.-usedatafromthenextiterationofLabs)● Howcanwebestmeasurebreakthroughs,transformations,andrelationships?
Methodsandprocess● WhataretheconditionsthatmakeTheoryUappropriateornottotakeadeepdiveintotheproblem?● WhatpartsofTheoryU,human-centereddesign,andotherpotentiallabmethodscanbeusedindependentlybydifferent
departmentsandforwhichproblems?
● WhatLabmethodsaremostappropriate,highimpactandengagingindifferentsituationsandfordifferentproblems?● Re-evaluatetheintentionaldifferencesbetweenLabmodels-designsprintversusoveralongerperiodoftime-what
workedwellandwhatdidn’t?● Howdoweensureprototypesdon’tdieoff?● Howcanwesupportrisk-takinginuncertainproblemspaces?
● HowdoweensurethatsuccessfulprototypesareintegratedintoCityactivitiesandoperationsineffectiveways?
Typesofconveningquestionsandteams● Whattypesofproblemsarecomplexenough?● WhatarethequalitiesofaLabteamthatneedtobeinplace?
Space+aestheticexperience● WouldapermanentLabspacecreatethesameatmosphere(i.e.-helpLabteam“showupdifferently”)
OperatingModel● WhatarethebenefitsandlimitstothemodelofaLabmanagerandanecosystemofconsultants?
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
14
● Whatdoessuccessionplanningneedtolooklike?● Canwecommunicate“successstories”oftheLabinternallyandexternallytogeneratebuy-inandunderstandingaboutthe
approach?
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
15
References
Cabaj,M.(2014).WhatWeKnowSofarAbouttheStrategyContinuum.HeretoThereConsulting.
Hwalek,M.,&Williams,M.G.(2011).Thereal-timeevaluationmemo:Atoolforenablingevaluativethinkingandlearningin
foundationsandnonprofits.TheFoundationReview,2(3),25-35.
Patton,M.(2011).DevelopmentalEvaluation:ApplyingComplexityConceptstoEnhanceInnovationandUse.London:TheGuilfordPress
Scharmer,O.(2009).LeadingfromtheFutureAsItEmerges:TheSocialTechnologyofPresencing.Berrett-KoehlerPublishers
Westley,F.,Laban,S.,Rose,C.,McGowan,K.,Robinson,K.,Tjornbo,O.,Tovey,M.(2016).SocialInnovationLabGuide.Waterloo
InstituteforSocialInnovationandResilience.
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
16
APPENDIXA-ExampleFeedbackFormfromDay5ofEngineeringLab
EngineeringCustomerServiceLabDateandLocation
Thankyouforyourblood,sweatandtearsoverthelasttwoweeks.Beforeyougo,weneedyourfeedbacktohelpusdeterminetheimpactoftheSolutionsLabs,andplansuccessfulLabsinthefuture.1.I’velearnednewmethodsand/orIhavenewtoolstobringbacktomywork/team?
StronglyDisagree
Disagree SomewhatDisagree
Neutral SomewhatAgree
Agree StronglyAgree
2.Ihavedevelopednewrelationshipsand/ordeepenedexistingrelationshipsbecauseoftheSolutionsLab.
StronglyDisagree
Disagree SomewhatDisagree
Neutral SomewhatAgree
Agree StronglyAgree
3.Asalab,wecametosolutions/prototypesthatwouldnothaveotherwisebeencreated?
StronglyDisagree
Disagree SomewhatDisagree
Neutral SomewhatAgree
Agree StronglyAgree
4.WhatwasthebiggestpersonalchallengeforyoutoovercomethroughouttheSolutionsLabprocess?5.DoyouunderstandtheproblemofcustomerserviceinEngineeringdifferentlythanwhenyoustarted?Ifso,how?6.WhatadvicedoyouhaveforusforfutureLabs?
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
17
APPENDIXB-Evaluation“UnitsofAnalysis”● Fundingleveraged● Prototypestested● Professionalandpersonaldevelopmentopportunities–teammemberslearnednewskill,teamworkimproved,verticaland
horizontaldevelopment,moreengagedincity,broughtwholeself● Clarified/honedroleforlabwithinbroadercitycontext● Successfullyfailed● Generatedexcitementinternally● Didn’tgetboggeddowninhierarchy● Newmethodstried● Communityrelationshipsstronger.Builttrust.Sharedpower.● Collaboration● Breakthroughs● Internalcoverage/storytelling● Personaltransformation;signsofsystemstransformation● Builtandsupportednetworkofprocessexperts● Eachprojectteamcansay:wegotabituncomfortableandtriedsomenewthings.Welearnedsomenewmethodsthatwecan
bringbacktofutureprojects.We’reastrongerteam.We’vebuild/strengthenedcommunityrelationships.● Whatisonethingtheyareworkingonthatwillrequiretheygrowtoaccomplishit● Howaretheyworkingonit● Whoelseknowsandcaresaboutit● Whydoesthismattertothem
ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017
18
APPENDIXC-InterviewQuestionstoCompletetheUserProfile
1. Whatbenefitsmightyouexpecttosee?2. Whatarethecurrentbarrierstoachievingthis?Howwillthelabovercomethese?3. Howwouldlifebebetterforstaff,stakeholders,thecityifthisprojectworkedwell?4. Whatisitaboutthelabthatyouthinkwillmakethedifferenceforthem?5. Whatelsewillbeneededtoachievetheseresults?6. Whatmightbesomenegativeimpacts?Howmighttheycomeabout?Howmightwereducetherisksofthoseimpacts?