June2017

EVALUATIONREPORT

FormativeEvaluationoftheUNFPAInnovationInitiative

VOLUME2Reconstructionofthetheoryofchange

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

1

Tableofcontents2.1Reconstructionoftheimplicittheoryofchange......................................................................2

2.2Theex-posttheoryofchange.........................................................................................................6

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

2

2.1Reconstructionoftheimplicittheoryofchange

A theory of change is a comprehensive description and illustration of how and why adesiredchangeisexpectedtohappeninaparticularcontext.1Itisatooltohelpidentifyingthelinksbetweenadesiredchangeandthesequencethatwillmakeithappen.Theoriesofchange are used in planning and inevaluation. A widely accepted definition isshowcasedinBox4.2

The InnovationFund and the Initiativedidnothaveanyexplicit,documentedtheoryofchange. However, the basic elements for atheory of change (outcomes, assumptions,rationales and interventions) weredescribed in a number of documents. Theevaluationteamreconstructedthetheoryofchange of the Innovation Initiative,including the Innovation Fund, using fourdocuments. The first two were seminaldocuments reflecting the initial thinking inUNFPAonhowtoapproachinnovation:theUNFPAInnovationConceptPaper(September2014), and the Updated Vision of Innovation at UNFPA 2015-2017 of April 2015.3 Theother two documents are the UNFPA Strategic Plan 2014-2107 and the CorporatePriorities and Projects 2014.4 These two strategic documents include direct or indirectremarks on the expected role of innovation and the institutional approach to it. Theevaluation team examined the documents, identified the elements (outcomes,assumptions, rationales and interventions) and reconstructed the logic in a visual form(Figure8).5ThetheoryofchangeoftheInnovationInitiativeinanutshell

As shown in the snapshot figure, the thinking behind the Innovation Initiative is that aproper enabling environment that generates a culture that nurtures innovation willeventuallyresultinenhancedorganizationalperformance.Organizationalperformanceisunderstood from two perspectives: Performance in terms of the ability to respond toemerging challenges with flexible (innovative) solutions, and performance in terms ofimprovements in business processes leading to increased organizational efficiency. Asshowninthefigure,thesequence‘enablingenvironmentgeneratesacultureofinnovation’issupportedbyameasure,communicateandlearnstrategythat,inturn,alsocontributestothedevelopmentofaculturethatnurturesinnovation.

It is worth noting that the theory of change of the innovation initiative has a verydistinctive feature when compared to approaches in other UN sister agencies: It

1http://www.theoryofchange.org2D.Taplin,H.Clark,E.CollinsandD.Colby(2013).TechnicalPapers:ASeriesofPaperstosupportDevelopmentofTheoriesofChangeBasedonPracticeintheField(PDF).NewYork:ActknowledgeandTheRockefellerFoundation.3”UNFPAInnovationConceptPaperofSeptember2014:Innovationandcreativitycorporateprojectproposal:NurturingInnovationatUNFPA”andthe“UNFPAInnovationIDWGRetreat,MeetingReportandRecommendationsforanUpdatedVisionofInnovationatUNFPA,2015-2017”ofApril2015.4”CorporatePrioritiesandProjects2014,UNFPAExecutiveCommitteePaper,17February2014”5Thisisoneofseveralpossibleinterpretations.Totheevaluationteam,thisisthemoreplausibleversion.ItwaspresentedandvalidatedattheUNFPAInnovationPlanningRetreatinFebruary2017.

Box4.Whatisatheoryofchange?

“Broadly, a set of beliefs and assumptions aboutwhat changes need to happen and how to bringthem about, to reach a stated goal. As amethodological practice, Theory of Change is aprocess through which participants construct adescriptivemodelof (bothgraphicandnarrative)that explains the outcomes sought, why they areneeded, how they will be achieved, and howprogressonthemcanbemonitored”http://www.actknowledge.org/resources/documents/ToC-Tech-Papers.pdfSource:http://www.theoryofchange.org

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

3

putsculturerightatthecentre.ThisisauniquefeatureoftheUNFPAapproach.OtherUNagenciesalsocareaboutculture,butwhentheybeganwithinnovationtheytendedtoput the “solutions” (the innovations) at the centre. Thatwas the case inUNICEF. Afterseveral yearsof focusingon tech solutions though, someofwhich renownedworldwide(EduTrac, U-Report), UNICEF has started amove towardsmainstreaming innovation intheorganization(forallstaff),puttingcultureinthecentre.

Thebubblesinthesnapshotdesignatecriticalassumptions(greenboxesinthecompletetheoryofchangeinFigure8.6Assumptionsarebeliefsaboutconditionsthatmustbemetforchangestobegeneratedandoutcomestobeachieved.Atheoryofchangeworkswellas long as its assumptions are valid, as they explainwhy the chain of changes can andshouldwork.Assumptionsinthetwofiguresarecriticalassumptions,thatis,thecoreandessentialhypothesisbehindthelogicoftheInitiative.Figure1.SnapshottheoryofchangeoftheInnovationInitiative

Critical assumptions for the Innovation Initiative were, in summary: that complexdevelopmentissuescannotbeaddressedthroughusualbusiness–theyrequireinnovativesolutions;thatanenablingenvironmentthatpromotesinnovation(withfeaturingphysicalandorganizationalstructures)mustbeinplace;thattheInnovationFundisamechanismable to generate a cultural shift in the organization; that growingmore innovative as aorganization depends on the creative capacities of the staff (culture of innovation isequatedtocreativity);thatfailingfastandcheapisthekeytoenhancedperformance;thatthebestwaytoincreaseorganizationalefficiencyisinnovatinginbusinessprocesses;and

6Acoreaddedvalueoftheoriesofchangeisthatmakeassumptionsexplicitinacontextwhereassumptionsareusuallypresumedandtacit-whichmakesitverydifficulttomonitorthem.

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

4

that an evidence-based “public face” is attractive to donors and will lead tomobilizingfurtherresourcesforinnovation,allowinginnovationtothrive.TheroleoftheInnovationFund

Blue boxes in Figure 8 designate interventions, that is, activities that the Initiative,including the Innovation Fund, were set to carry out in order to generate changes inculture and subsequent changes in performance. The 8-prongs in the Update Vision ofInnovationareallreflectedintheblueboxes.

The theory of change presumes that an enabling environment to nurture a culture ofinnovationshouldincludeinnovation-friendlypolicies,incentivesforstafftobecreative,asafespacetotesthigh-riskideas,unusualpartnershipsandamechanismtogenerateandfund innovative ideas (the Innovation Fund). This enabling environment will empowerstafftoinnovate. TheInnovationFundisacentralelementinthistheoryofchange:it isthemechanismtomakeitwork;atoolthatcombinedwithotherelementsoftheenablingenvironmentwillempowerstafftoinnovateandinstilaculturethatnurturesinnovation.Theoriginalidea,inshort,wasthatprojectssupportedbytheInnovationFundwouldhelpcreating a risk-conscious, learning-based, failure assimilating culture that would thengenerateinnovativesolutionsthatcouldmakeadifference.Measure,communicateandlearn

The measure, communicate and learn pathway of the theory of change builds on theprecondition that the InnovationFundwillhavea functioningM&Emechanism inplace.As shown in the right-to-left pathway in Figure 8, thisM&E system had to be one thatallows demonstrating results and prompting staff to learn from success and failure.Simultaneously,itwasexpectedthatcommunicatingthesedemonstrableresultsprovidedby the M&E system, would generate en evidence-based public face that would givevisibilityandpositiontheUNFPAbrandforinnovation.Thisinturn,wouldbringinfurtherresourcemobilizationforinnovation.

Section3.4of the reportpresents the ex-post theoryof change i.e. the samediagram inFigure 8 but incorporating findings of the evaluation. Several findings in sections 3.1through3.3shedlightonwhethercriticalassumptionsheldtrueaswellasonwhethertheexpected cause-effect relationships in the theory of change are actually occurring, how,andthereasonswhytheydidordidnot.

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

5

Figure2TheoryofchangeoftheInnovationInitiative

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

6

2.2Theex-posttheoryofchange

Section2.4presentedareconstructionofthetheoryofchangeoftheInnovationInitiative,includingtheInnovationFund.Thissection3.4isarevisionofthatdiagraminlightofthefindingsoftheevaluationpresentedintheprecedingsections.

ThediagraminFigure31illustratestherevisedversionofthetheoryofchangepresentedinchapter2.4.Inlinewiththeformativenatureoftheevaluationthefocusoftherevisionisoninsightsthatmayleadtoimprovements.Therearethreecategoriesofinsights:

Explicit (critical) assumptions that have not hold true either partially or

entirelyThesewereassumptionsexplicitlymentioned in thenarrativeof thedocumentsused toreconstructthetheoryofchange.Theseassumptionsaredepictedingreenboxes.Theredcircleindicatesthatthereisevidencethattheassumptiondidnotunfoldasexpected. Tacit(critical)assumptionsthathavenotholdtrue

Thesewereassumptionsnotexplicitlymentionedinthenarrativeofthedocuments.Theywereidentifiedasaresultoftheevaluation’sworkandaredepictedinwhiteboxes.Theyellowcircleindicatesthereisevidencetheassumptiondidnotholdtrueeitherpartiallyorentirely. Outcomepathwaysthathavenotunfoldedasexpected

This are reflected in the revised theoryof change inbubblediagramswithblack circlesattached.Theyindicatethatthereisevidencethatthecause-effectrelationshipsbetweenactivities(interventions)andoutcomesand/orbetweenoutcomesatdifferentlevelshavenotoccurredasexpected.The revision puts the attention on three aspects. The critical assumptions, the mostsignificant bottlenecks affecting planned sequences of outcomes and the role of theInnovation Fund, given that the Innovation Fund is themain focus of the scope of thisevaluation.Therevisionhasbeenconductedwithaviewtogenerateuseful insightsthatinformthenextphaseoftheInitiativeandtheInnovationFund.

Explicit(critical)assumptionsthathavenotholdtrueeitherpartiallyorentirely

1 Creativityisimportantbutthereismoretoinnovationthancreativity

F Forfurtherdetailsseethesub-sectionaboutinsightsonthelimitationsoftheInnovationFundtogenerateimpactsolutions,undersection3.1.4.

Thisassumptionpresupposed thatonlyby investing in thecreativecapacitiesofUNFPAstafftheorganizationcouldgrowmoreinnovativeandthatacultureofinnovationmeanscreativityatalllevels.7AmainimplicationofthiscriticalassumptionisthattheInnovationFundwas designed around it, operating less as amechanismproviding seed funding toturngoodideasintosolutionswithimpactandmoreasasourceoffundstostimulategoodideas (creativity). Critical to this critical assumption was that creativity is key to bringabout the culture that will generate impact solutions (flexible solutions to solvedevelopmentchallenges).

7Seepage6oftheUpdateVisionofInnovationVisionandpage2oftheConceptPaperininnovationrespectively.

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

7

Asshownbythefindingsoftheevaluationcreativityisimportantbutthereismuchmoretoinnovationthanbeingcreativewithintheorganization.Linkswiththeecosystemareasmuch important as internal creativity. As revealed also by the comparative analysis,creativitydoesnotnecessaryhavetobe insidetheorganization–otherUNagencies tapon creativity from the outside and convene, facilitate and advocate to make impactsolutionshappen.2 TheInnovationFundhadlimitationsasatooltogenerateaculturalshift

F ForfurtherdetailsseesubsectionaboutinsightsonthelimitationoftheInnovationFundtonurtureacultureofinnovation,undersection3.1.3ThiscriticalassumptionheldthattheInnovationFundisatoolthatcangenerateaculturalshift.8The findingsof the evaluation reveal that innovation fundsmayeventuallyplay arole in changing culture but the design of this particular Fundwas not optimal to thatpurpose. The main reason being that it is a project-based, resource-driven mechanismworkingonopen (toall thematicareas)and internal calls forproposals.The InnovationFund was designed to implement projects rather than to test solutions. Moreover, theoperative features of the Innovation Fund (timings, reporting systems) were notconducivetoapplyingtheinnovationprinciplesthatembedacultureofinnovationi.e.becollaborative,designwiththeuser,bedatadrivenanddesignforscale.

The functioning of the Innovation Fund made it also difficult to apply the failing-fastconceptandaccruelearningfromsuccessandfailure,whichwasatthecoreoftheculturalchange sought by the Innovation Initiative. In short, the functioning of the InnovationFundreflectedtheexistingcultureratherthanactingasagamechanger.TheInnovationFunddidgenerateasubstantialmomentumandchangedattitudestowardsinnovationbutitwasdesignedinawaythatmadeitdifficulttobringintherisk-taking,failureacceptanceandlearningoutcomesthatdefinethecultureofinnovationtheUNFPAwasseeking(box5).Overall, theassumption that the InnovationFundwould change culturedidnotholdentirelytrue.

Tacit(critical)assumptionsthathavenotholdtrue

1 Humanresourcearrangementstoimplementthevisionwereassumedtobeadequate

F For further details see section 3.1.2 on the functioning of the Innovation Fund and 3.1.6 ondevelopingnewpartnerships

TheSecretariatoftheInnovationFundandtheIDWGoninnovationwerethemainplayersin the implementationof the8-prongsof the Initiative. Itwaspresumed that theycoulddevoteenoughtimeandhavetherightincentivestorolloutthevisionembeddedinthe8-prongs(blueboxes).Thiswasanimplicitcriticalassumption.However,thatwasnotthecase and as a consequence the implementationof theworkplanhasbeenmodest,withlimited progress in key interventions (partnerships, consultations, labs). The mainrepercussionisthattheexpectedcontributionsoftheseactivitiestocreatinganenablingenvironmenthavenotoccurred.

2 Tacitinstitutionalfactorsrequiredtoempowerstafftoinnovatedidnotalwaysholdtrue

F Forfurtherdetailsseesection3.3.3onobstaclestoinnovationinUNFPA

8Itappearsonseveraloccasionsthroughoutthedocuments;thefirsttimeisonpage3oftheConceptPaper.

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

8

The enabling environment depicted in the theory of change includes a number of veryrelevant elements such as staff incentives and skills, innovation-friendly policies (e.g. inprocurement, partnerships) and a risk-controlled space to experiment (labs). However,thereareanumberof additional enabling factors thatalthoughnotmadeexplicit in thedocuments,were assumed to be in place. These factors are: seniormanagement buy-in,particularlyinfieldofficesandbusinessunits;staffbeinggivenenoughtimetodevotetoinnovation activities; and organizational incentives to carry out innovation tasks. Thesefactors are key to empower staff to innovate (outcome) and subsequently develop aculturethatnurtures innovation. Itwasan implicitassumptionhat thesethreeelementswouldbeinplace.

The findings of the evaluation reveal though that senior management buy-in, timeallowances for the staff to innovate andorganizational incentives to conduct innovation(suchastheneedtoreportonit)wereassumptionsthathavenotalwaysholdtrue,actingasanobstacletoempowerstafftoinnovate.

3 ThetheoryassumedthatstaffwouldactivelyaccesstheInnovationHub

F Forfurtherdetailsseesection3.1.2onthefunctioningoftheInnovationFundand3.1.5onlearningfrominnovation

The theory of change reflects a sequence whereby the results and experiences frominnovation projects would be generated and communicated in a way that nurtures acultureofinnovation.NurturingcultureofinnovationisdefinedinUNFPAasaculturethatlearns from success and failures. This assumed that applicants to the Innovation Fundwouldinteractwithoneanothertoshareexperiencesaswellastomakeuseofthewealthof information available on awarded and non-awarded project proposals and technicalfeedback provided to both. The InnovationHubwas expected to play a key role in thissequence as a reference platform facilitating these exchanges and knowledge transfers.Thiswasnotmadeexplicitasanassumptioninthedocumentsbutitwasindeedacrucialhypothesis.

AsexplainedinthemainfindingsofthereporttheuseoftheInnovationHubwaslimited(duetouneasinessofaccessandpoorawarenessonthesite).TheconsequencewasthattheHubcouldnotplayaroleasachanneltoturnsubmittedproposalsandfeedbackintolearning,andasaplatformtofosteracommunityofpracticearoundtheInnovationFund.

Outcomepathwaysthathavenotunfoldedasexpected

1 M&EmechanismswerenotadaptedtoinnovationF For further details see section 3.1.5 on learning from innovation (sub-sections on learning from

success and failure and on M&E and branding). See also the subsection about insights on thelimitationsoftheInnovationFundtonurtureacultureofinnovationundersection3.1.3

Having functioning M&E frameworks in place was the foundation for the measure,communicateandlearnlineofactioninthetheoryofchange.M&Esystemswereexpectedto feed into communication of results, branding and resource mobilization as well asfeeding into learning from success and failure. In this setting,M&E frameworkswere acritical building block in the theory of change and weaknesses in M&E systems havehinderedtheachievementofanumberofoutcomes.M&Esystemswereweakatproject,InnovationFundandInnovationInitiativelevel.TheInnovationInitiativedidnothaveanM&E framework; the Innovation Fund had one but it was not complete; and individualprojectshadaframeworkinplacebutbasedonmonitoringandreportingsystems.M&E

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

9

systems at project level were not data-driven, outcome-based, real-time monitoringsystems, and did not allow capturing unexpected outcomes, which are important tolearningwheninnovating.

Another consequence is that as presentlydesigned, these systemspresentdifficulties ingenerating thedata-driven success stories that are required to communicate innovationresultsexternally(right-handverticalarrowinFigure31).

2 Limitations of the Innovation Fund to generate innovative solutions that respond tochallenges

F For further details see sub-section about insights on the limitations of the Innovation Fund togenerateimpactsolutions,undersection3.1.4

AmainoutcomeofthetheoryofchangeisthatUNFPArespondstodiverse,complexandemergingdevelopmentchallengesthroughflexible(innovative)solutions.The Innovation Fund was established as a mechanism to generate a cultural shift,empoweringstafftotakerisksandlearnformfailureasameansto,inthefuture,generateimpact solutions. The foreseen sequencewas a Fund nurturing a culture of innovation,based on the creativity of the staff, leading “possibly (to) discovering new big impactsolutions”.9 However, the potential contribution10 of the Innovation Fund to generateinnovative (impact) solutions has been hindered as a consequence of design andoperational features. In spite of these limitations, the Innovation Fund has actuallymanagedtogenerateafewsuccessfulsolutionsofferinggoodprospectsforatransitiontoscale.

9Seepage2oftheInnovationConceptNote.10Thewidewhitearrowgoingfromculturetoflexiblesolutionssymbolizesthiscontribution.

F o r m a t i v e E v a l u a t i o n o f t h e I n n o v a t i o n I n i t i a t i v e

10