mdf fiji dced audit report 14jan19 · 2. summary of the programme and results measurement system...

Auditors’ReportMDFFiji/DCEDStandardforRM,14thJanuary2019

1

1. OverviewProgram MarketDevelopmentFacility–Fiji Auditvisitdates 13-23November2018 Overallfinalratings1 MUST 523/540=97% RECOMMENDED 223/245=91%Coverage ExportsandSupportServices,TourismandBusiness

AccelerationandExpansionSectorsExcluding2oldinterventions:Ben’sTradingLimitedandFarmboy

Allcontrolpointswerechecked. DCEDStandard VersionVIII,April2017 Signed:CountryRepresentative:Date/placeAuditors:PhitchaWanitphon Date/place ShibajiRoy Date/place:

1 An overall rating of 100% implies that the project meets the compliance criteria and has a strong measurement system of acceptable quality within the boundaries of what the programme has set itself to measure, not that it is has a perfect measurement system.

16.01.19

16.01.2019


2

Table of Contents

1. Overview................................................................................................................................1

2. SummaryoftheProgrammeandResultsMeasurementSystem..........................................3

3. SummaryoftheAuditProcess...............................................................................................5

4. SummaryofFindings..............................................................................................................8

5. Summaryofkeyareasforimprovement..............................................................................16

Annexes........................................................................................................................................16

Acronyms

AUD Australiandollar

BA BusinessAdvisor

DCED DonorCommitteeforEnterpriseDevelopment

DFAT DepartmentofForeignAffairsandTrade

ESIA Earlysignofimpactassessment

IA Impactassessment

IG Interventionguide

MDF

MFEE

MarketDevelopmentFacility

MDFFijiEnterpriseEngine

MRM

RBF

MonitoringandResultsMeasurement

ReserveBankofFiji

RC Resultschain

WEE Women’sEconomicEmpowerment


3

2. Summary of the Programme and Results Measurement System

2.1 Summary of MDF

TheMarket Development Facility (MDF) is amulti-countrymarket systems development programmefundedbytheAustralianDepartmentofForeignAffairsandTradeandimplementedfrom2011to2017forphaseIand2017to2022forphaseII.MDFcommencedinFijiin2011.

MDFsupportsprivateandpublicsectororganisationsto innovate, investand/orundertakereforms insuchamannerthatsmallfarmsandfirmsbenefitfrombetteraccesstoproductioninputs,servicesandendmarkets.This,inturn,isexpectedtomakethemmoreproductiveandgrowandultimatelycreatesjobsandincreasesincomeforpoorwomenandmen.

InFiji,MDFworksinthreesectors(orstrategicengagementareas),allofwhichareincludedintheaudit.

• Exports and Support Services: working with businesses that are interested in product andmarket diversification aswell as introducing improved support services such as transport andpackagingthatmakeiteasierforFijiancompaniestoexport.

• Tourism:workingwithbusinessesandgovernmenttoattracthighyieldtouriststoandaroundFiji,basedonFiji’scompetitiveadvantagesforkeytouristsegments,toincreasethenumbersofhigh-quality,locally-brandedproductsforthetouristmarketandtocreatemoreauthenticFijianservicesandactivities.

• Business Acceleration and Expansion: working with market players to develop businessincubator and business advisory services that will strengthen the management capacities ofMSMEsintheExportandTourismsectors.

Key features of the results measurement system

As per the original tender requirements, MDF’s results measurement system is guided by the DCEDResultsMeasurementStandard.MDF’sresultsmeasurementsystemhasbeendevelopedfortheFacility(whichcoversfivecountries),thenrolledoutatthecountrylevel.TheFacilityprovidesoverallguidanceand management of the results measurement system. There is flexibility to implement resultsmeasurementaccordingtotheinterventionneeds,althoughallinterventionstrackaggregatedcommonintermediateandimpactindicators.

Interventionguidesaredevelopedforeach interventionorpartnership.These include:aresultschain,resultsmeasurementplan;projectionandactuals(quantitativeandqualitative)informationalongwithassumptions.Morerecently,resultschainshavealsobeendraftedearlierbyMDFbusinessadvisorstoassistwiththepreparationofpartnershipjustificationdocuments.

Data relating to common indicators is compiled from each intervention to country-level aggregatedresults,whichistheninturncompiledtoMDF-wideresultsonanannualbasis.

Mostbaseline,monitoringandearlyimpactassessmentactivitiesareconductedin-houseinFiji.


4

In 2015/16,MDF also sought to integratewomen’s economic empowerment (WEE)more thoroughlyintotheresultsmeasurementapproach.InterventionguideswereupdatedtoidentifywhichofthefiveWomen’s Economic Empowerment parameters an intervention or partnership was expected tocontribute to. Subsequently, data collection tools were amended to collect information on thesedomains.

MDFalsodevelopedaframeworkforsystemicchange,withtheinitialthinkingcommencingin2014.Theframeworkconsistsofsixparameters:autonomy,sustainability,resilience,scale,inclusionandWEE.Itmapsa six-stagepathway frombeginning tohigh statesof systemic change.MDF’s role in facilitatingsystemic changediminishesover timewithheavier intervention in the initial and intermediate stagesand less at the advanced stage. The mature stage denotes that targeted changes have become thenorm.

ThemethodstoassessMDF’sresultsinpromotingsystemicchangehavenotyetbeenfullydeveloped,althoughaspectsofthemdrawonotherprocesses(suchastheWEEframeworkoutlinedabove).

Evolutionoftheresultsmeasurementsystem

TheMDFapproachtoresultsmeasurementisguidedbytheDCEDResultsMeasurementStandard.Theinitialsystemwasdeveloped in2011/2012andthe latestversionoftheResultsMeasurementmanualupdated in 2014. This manual is being replaced by a Strategic Guidance Note covering resultsmeasurement(currentlystillindraftform)initiatedin2017.

In2016/17,theMDFTeamLeadercommencedareviewoftheresultsmeasurementsysteminpractice,followingissuesthataroseinsomecountries.Insummary,MDFoverallfoundthattheimplementationand results measurement teams were not working together as intended but were rather operating,somewhat,inisolationofeachother.Theissuesidentifiedin2016included:

• User-unfriendlyresultschains,becauseimplementershadstoppedusingandupdatingthem,astheyhadbecometheprerogativeoftheresultsmeasurementspecialists;

• Changedefinitionsinresultschain‘boxes’foroutputsandoutcomesgotincreasinglymoregeneric(shorter,simpler,lessprecise)whilelonglistsofindicatorsweregenerated.Thelesspreciselydefinedchangemeantthatimplementersandresultsmeasurementspecialistsdivergedonthefocusofthemonitoringandimpactassessmentactivities.


5

• Rigid,clunkyanduntimelysurveyresearchthatgrewoutofthelonglistsofindicators,largersamplesizes,lackofqualitativeinformationtohelpunderstandthequantitativedata.

Changes were implemented, including simplifying results chains; a reconfiguration of roles so theimplementersfirstdrafttheresultschains,initiallyformulatingkeyquestionsanddevelopingindicatorsfromthese;andincreasingthefocusonqualitativeinformation.Thesix-monthlyreviewmeetingswerepartoftheMRMprocessfromthestartoftheprogramme.However,particularemphasisisnowgiventoensurethattheinformationcollectedfromthefieldthroughmonitoringvisits,fieldobservationsandpartner discussions is reflected in the strategicmeeting to steer the future of partnerships, and theportfolioingeneral.Inaddition,MDFFijihasbeenauditedaccordingtotheDCEDStandardin2014.

3. SummaryoftheAuditProcessMDFFijiwasauditedunderVersion8of theDCEDResultsMeasurementStandard,published inApril2017.MDFFijiworksinthreesectors:ExportsandSupportServices,Tourism,BusinessAccelerationandExpansion.NotallMDFFiji interventionshavebeen included in theportfolio fromwhich theauditorsselectedthesampletobeaudited.MDFFijioptedtoexcludetwoolderinterventionsthathadnotbeenupgraded to the new RM system. These are Farmboy (August 2012 to June 2013) and Ben’s TradingLimited(August2013toJuly2015).Thesampleselectionfortheaudit isatwo-stageprocess.Thefirststageistoselectthesectors.SinceMDF Fiji has only three sectors, to meet the minimum requirement of the Standard, all sectors arecovered. The audit reviewed a representative sample of all current and past interventions in thesesectors,omittingtheinterventionsthatwereexcluded,asdescribedabove.Thenumberofinterventionspersectorwereselectedasfollows:

• FromExportsandSupportingServices:25interventions.Thesquarerootof25=5.• FromTourism:24interventions.Thesquarerootof24=4.89,roundeddown=4.• FromBusinessAccelerationandExpansion:2interventions.Bothinterventionsareselectedtomeet

minimumrequirementoftheStandard.

ForExportsandTourismsectors,theinterventionsforwhichMDFhadconductedearlysignsofimpactassessmentswerepurposivelyselectedsothattheseassessmentscouldbeincludedintheaudit.Then,fromtherest, the intervention ineachsectorthathadthehighestoutreachnumberswaspurposivelyselected. The rest of the interventions in each sector were chosen at random, using a website thatgeneratesrandomnumbers.Theinterventionsselectedcanbeseeninthefollowingtable:

Table1:InterventionsSelectedfortheAudit

Sector Code InterventionnameTotalBudget(FJD)

Startdate

Expectedenddate Interventionstatus

ExportsandSupportServices

FJEPI07 StimulatinginvestmentinmoderntechnologytoallowICTbusinessestoimproveitsefficiencyinordertotap/deepennew/existingmarketsin

639,450 Aug-16 Dec-17 ActivitiesCompleted&MonitoringOn-going


6

theBusinessProcessOutsourcing(BPO)sector


FJOI07 Upgradingnurseryfacilityforproductionoflargervolumeandnewvarietiesofseedlingsonayear-roundbasis

339,705 May-13 Jun-16 ActivitiesCompleted&MonitoringOn-going


FJHOI12 Strengtheningsupplychainandreducingpost-harvestlossesthroughuseofinformationbrochuresandfarmgatepickup.

264,000 Dec-14 Jan-15 MeasurementCompleted


FJHOI16 UsingmechanisationtoaddresslabourshortagesinthesugarindustryinVanuaLevu

423,500 Jul-15 Dec-15 ActivitiesCompleted&MonitoringOn-going


FJHOI01,08,19,20

Improvingfarmers'yieldandincomebyimprovingsoilhealththroughavailability,promotionandawareness,andcorrectapplicationofAglime

435,540 Nov-12 Jul-17 ActivitiesCompleted&MonitoringOn-going.

Tourism FJTOI02 Upgradingchocolatemanufacturerfacilityanddevelopingbackwardlinkagestococoafarmersuppliers.

564,582 Jun-12 Dec-14 ActivitiesCompleted&MonitoringOn-going.

Tourism FJTOI11 Creatingawarenesstoreachandcaterforthegrowingandnicheinternationalwalkingmarketandworkingwithlocalcommunityactivitysitestoraisestandardofservice.

158,320 Mar-14 Jan-17 ActivitiesCompleted&MonitoringOn-going.

Tourism FJTOI22 ConductinganeedsassessmentfortheFijiMuseumtoimproveitscapacitytoattractandhostmoretouristswhilsttheyareinSuva.

39,542 Nov-15 Mar-17 ActivitiesCompleted&MonitoringOn-going

Tourism FJTOI23 Increasingthecapacityofcoffeeprocessorandproductivitylevelsoflocalcoffeebeansuppliers

114,850 Jan-16 Aug-16 ActivitiesCompleted&MonitoringOn-going

BAE - PilotingbusinessincubationsystemforSMEstoincreasetheimpactofentrepreneurialperformanceof

100,000 Sep-16 Jun-18 ActivitiesCompleted&MonitoringOn-going


7

participatingbusinesses

BAE - OutreachandsupportforthedevelopmentofMicro,SmallandMediumEnterprises(MSMEs)inFijithroughthecollectionandanalysisofMSMEdatatoinformsectordevelopmentandpolicy.

113,711 Apr-18 Aug-18 ActivitiesCompleted&MonitoringOn-going

ForFiji,theauditwasnotabletoassessandscoreallcontrolspointsbecausetheprogramhasnotimplementedallaspectsoftheresultsmeasurementsystem.Therefore,somecompliancecriteriaintheStandardwerenotscored.Thefollowingpointsdescribetheissuesarising:

• Impactassessmentsforallrelevantinterventionsauditedhavenotbeencarriedoutandthecompliancecriteria3.3.1,3.3.2and3.3.3cannotbeassessedandwerenotscored.MDFcontinuestomonitorinterventionswellafteractivitieshavebeencompletedandpartnershavestartedimplementingbehaviourchanges,typicallyfor2-4yearsbutlongerifchangesareexpectedtohappenquiteslowly.Monitoringincludesmonitoringvisitsandearlysignsofimpactassessments.Theseactivitiesincludecollectingdatatoassessattributableimpactsatthebeneficiarylevel,albeitnormallyusingsmallsamplesizesand/orlimitedcoverageofbeneficiaries.Therefore,datacollectedandanalysedformonitoringandearlyimpactassessmentsisconsideredlessrigorousthanthatneededforimpactassessments.Impactassessmentsareplannedverylateintotheprogramme,normallyattheendofthemonitoringperiods(typically2-4yearafteractivitieshavebeencompleted)oftheinterventions.Duetotheslowuptakeofsomebusinessmodels,adaptationofthebusinessmodelsand/ornegativeimpactsfromtropicalcyclones,themonitoringperiodshavebeenextendedformostinterventions.Asaresult,theimpactassessmentshavebeenpostponed.Thus,noimpactassessmentsforallinterventionsauditedhavebeenconducted.

• Inaddition,forsomeinterventions(18%ofinterventionsaudited),MDFplantocollectbaselineinformationduringtheimpactassessments.Dependingontheyearofadoption,establishingandassessingimpactmaybeverychallenging.Thebeneficiariesmayhavetorecallthebaselineinformationoveralongperiodoftime,whichcouldleadtopossiblehighlevelofrecallbiases.

• Theactualreportedimpactfigures(compliancecriteria6.1.1and6.1.2)arebasedonthemonitoringvisitsand/orearlysignsofimpactassessments.Thesefiguresaretypicallytriangulatedwithothersourcesofdatae.g.partners’informationandcomparedyear-on-yeartounderstandresults,butarenotbasedonsamplesizesthatwouldbeappropriateforimpactassessments.

• Compliancecriteria4.2.6and4.2.7werenotscoredbecauseMDFhasnotyetassessedtheresultsofsystemicchangeatthepathwayandbeneficiarylevel.

• Compliancecriteria6.3.1and6.3.2werenotscoredbecauseMDFhasnotassessedsystemicchangesandthereforecannotreportagainstit.


8

4. Summary of Findings MDFFijiscored97%(523outofapossible540points)for‘must’compliancecriteriaand91%(223outof possible 245 points) for ‘recommended’ compliance criteria. MDF Fiji has not yet carried out allaspectsofresultsmeasurement.Therefore,somecompliancecriteriaintheStandardwerenotscored.Hence,themaximum‘must’and‘recommended’scoreshavebeenadjustedtoexcludethecompliancecriteria thatwerenot scored.Table 2 summarises the scores for each sectionof theDCED Standard.DetailedscoresareoutlinedinAnnex1.

Table2:ScoresbyDCEDStandardSection(disaggregatedmandatoryandrecommendedcompliancecriteria)

Totalmaximum

Totalactual %

Section1:Articulatingtheresultschain

Must 80 78 98%Rec 15 15 100%

Section2:Definingindicatorsandotherinformationneeds

Must 80 78 97%Rec 50 49 98%

Section3:Measuringattributablechange

Must 145 137 94%Rec 80 77 96%

Section4:Measuringsystemicchange

Must - - -Rec 60 47 78%

Section5:Trackingcostsandimpact

Must 55 55 100%Rec 20 20 100%

Section6:Reportingresultsandcosts

Must 50 45 90%Rec 20 15 75%

Section7:Managingtheresultsmeasurementsystem

Must 130 130 100%Rec - - -

Totals Must 540 523 97% Recommended 245 223 91%Thefollowingsub-sectionsoutlinethescoresforeachcontrolpointandsummarisethefindingsaccordingtothestrengthsandweaknessesofeachsection.MoredetailedfindingsforeachsectorareoutlinedinAnnex2.


9

4.1 Section 1: Articulating the results chain Table3:Score:Articulatingtheresultschain

No. Controlpoints Must/Rec Stdmax.score Actualscore2

1.1Anappropriate,sufficientlydetailedandlogicalresultschain(s)isarticulatedexplicitlyforeachintervention.

M 20 19

1.2Eachinterventionresultschainissupportedbyadequateresearchandanalysis

M 15 15

1.3Midandseniorlevelprogrammestaffarefamiliarwiththeresultschain(s)andusethemtoguidetheiractivities.

M 25 24

1.4Theinterventionresultschain(s)areregularlyreviewedtoreflectchangesintheprogrammestrategy,externalplayersandtheprogrammecircumstances.

M 20 20

1.5Eachinterventionresultschainissupportedbyadequateresearchandanalysisongender.

R 5 5

1.6Eachresultschainissupportedbyresearchandanalysisthatconsiderstheriskofdisplacement.

R 10 10

Strengths WeaknessesAllinterventionshaveaninterventionresultschain.Resultschainaremostlylogicalandsufficientlydetailed.

Insomecases,resultschainshaveafewminordiscrepanciesintermsoflogicand/ordetails.

Partnershipjustificationsexplainthelogicoftheinterventions.Viabilityofthebusinessmodelhasalsobeenconsideredwhendevelopingthepartnershipjustification.Thepartnerjustificationsandinterventionguides(IGs)includemostoftheexternalrisks/assumptionsrelatedtothelogicoftheinterventions.Staffcanexplaincriticalexternalrisks/assumptionsrelatedtothelogicoftheinterventions.Theinterventionsaddresskeyconstraintsinthecountryengagementstrategy.

Staffhaveagoodunderstandingoftheinterventionresultschainsandusethemintheirwork,includingdraftingthemaspartofthepartnershipjustification,monitoringprogress,designingassessmentsanddiscussingduringthereviewmeetings.

Inafewcases,thereareafewminordiscrepanciesintheexplanationsoftheresultschains.

2Theactualscoreshavebeenroundedup.Ifthedeductionsarelessthan0.5points,thescoresareroundeduptofullpoints.


10

Regularreviewsareundertakenofresultschainsandtheyarerevisedwhennecessary.

MDFhasdevelopedaWEEframeworkandeachinterventionisassessedtodetermineitscontributiontofiveWEEdomainsofchange(e.g.economicadvancement,decision-making).ThesearenotedintheIG.RelevantboxesintheRCarealso‘tagged’asrelatingtoWEE.

Displacementhasbeenconsideredfortheallrelevantinterventions.

4.2 Section 2: Defining indicators of change and other information needs

Table4:Score:definingindicatorsofchangeandotherneeds

No. Controlpoints Must/Rec Stdmax.score Actualscore

2.1Thereisatleastonerelevantindicatorassociatedwitheachchangedescribedintheresultschain(s).

M 10 10

2.2Qualitativeinformationonhowandwhychangesareoccurringisdefinedforeachintervention.

M 30 28

2.3Asmallnumberofindicatorsattheimpactlevelcanbeaggregatedacrosstheprogramme.

M 20 20

2.4Therearespecificindicatorsthatenabletheassessmentofsustainabilityofresults.

M 10 10

2.5Midandseniorlevelprogrammestaffunderstandtheindicatorsandhowtheyillustrateprogrammeprogress.

M 10 10

2.6Therearespecificindicatorsthatenabletheassessmentofgenderdifferentiatedresults.

R 15 15

2.7Anticipatedimpactsarerealisticallyprojectedforkeyquantitativeindicatorstoappropriatedates.

R 35 34

Strengths Weaknesses

Allinterventionshaveindicatorslinkedtoeachboxintheresultschains.Theindicatorsaremostlyrelevant,specificandmeasurable.

Insomecases,afewindicatorsaremissingorarenotspecificenough.


11

Foreachintervention,questionshavealsobeendevelopedforquantitativeandqualitativeinformationneeds.3Allinterventionsincludequalitativeindicatorsorquestionstomeasurebehaviouralchangesofeachactor.Qualitativeindicatorsorquestionsaresufficienttoassesscharacteranddepthofchangesinmostcases.

Insomecases,qualitativeindicatorsorquestions(inthe“keyquestionstoask”column)onwhychangeshaveorhavenothappenedaremissing.

Commonimpactindicatorsaredefinedforallinterventions.

Allinterventionshaveindicatorstoassessthesustainabilityofresults.Theprimarysustainabilityindicatorforthebusinessesisprofit,and,whenitisincluded,itistrackedforthreeyearspastMDFsupport.Atbeneficiarylevel,toassesslikelihoodofsustainability,netincomeistrackedinmanyinterventions,andforthreeyearsagain.

Inafewcases(e.g.DHLandDeveshandBharos),sustainabilityindicatorsforpartnersincludesalesvolumeand/orvalue.However,otherkeyindicatorsrelatingtocosts,profitandbusinessviabilityorqualitativequestionsonthelikelihoodofsustainabilityarenotincluded.Thislimitstheabilitytoassessthelikelihoodofsustainabilityoftheseactors.

Forallinterventions,WEEquestionshavebeenidentifiedforrelevantchangesintheresultschains.

Projectionshavebeendevelopedforallrelevantinterventionsandforalmostallkeyindicators.Projectionhasbeenreviewedandupdatedatleastannually.

ForAdiChocolate,thebeneficiaryincomecalculationforprojectionisnotcorrect.ForFijiMuseum,projectionshavenotbeendeveloped.Inaddition,afewassumptionsunderpinningtheprojectionsarenotdocumentedinafewinterventions.However,staffcanexplainthem.

4.3 Section 3: Measuring attributable change

Table5:Score:Measuringattributablechange


3.1 Baselineinformationonallkeyindicatorsiscollected. M 60 60

3.2 Monitoringinformationonallkeyindicatorsiscollected. M 60 52

3.3Impactassessmentisconductedtoassessattributablechangesinallkeyindicatorsintheresultschainsusingmethodsthatconformtoestablishedgoodpractice.

M N/A N/A

3.4Theprogrammeimplementsprocessestouseinformationfrommonitoringandresultsmeasurementinmanagement

M 25 25

3 The development of questions is part of the MDF RM system to also capture the need to understand whychangesareoccurringornot.Developingquestionstoarticulateinformationneedsisnotaspecificrequirementofthe Standard. However, developing questions for some information needs (instead of indicators) is acceptablepractice.


12

ofinterventionsanddecisionmaking.

3.5Theprogrammehasasystemforassessingandunderstandingdifferentiatedresultsbygender.

R 60 60

3.6 Theprogrammemonitorstoidentifyunintendedeffects. R 20 17


Planstocollectbaselinedataexistforallinterventions.Whereappropriate,baselineinformationhasbeencollected.

RMplansarelargelysatisfactory.Attributionhasbeenconsideredatthepartnerandbeneficiarylevels.Quantitativeandqualitativeinformationhasbeencollectedaccordingtotheplanthroughregularfieldvisits,earlysignsofimpactassessment,telephonecallsandreportsfrompartners;theinformationallowsstafftoadequatelymonitorprogress.

Insomecases,thedatesplannedfordatacollectionforafewboxesintheresultschainsarenottimelyand/orspecific.ForAglime,theattributionstrategyisnotadequate.Insomecases,qualitativeinformationforafewchangesintheresultschainshasnotbeencollected.Inmanycases,qualitativeinformationhasnotbeencompiled,summarisedanddocumentedtoprovide“aconsolidatedview”ofthequalitativeaspectsofperformance/statusofrelatedchangesintheresultschains.

Staffusethemonitoringinformationtoinformtheirworkandimprovetheinterventions.

RMplansalsoincludedisaggregatingrelevantdatabysexandtheappropriateassessmentofrelevantWEEdomains.AssessmentsofrelevantWEEdomainhasbeencarriedoutwhenapplicable.

Whereunintendedeffectshavebeenidentified,theyarereportedanddiscussedinteammeetings;theyarealsoincorporatedintotheinterventionswhenappropriate.

Apartfrominstructionstolookoutforunintendedeffectsduringassessments,theoverallprocessforassessingunintendedeffectsisnotexplicitlydocumentedintheRMguidee.g.wheretheunintendedeffectsobservedshouldbedocumentedandwhentheirimplicationswillbeanalysed,discussedanddocumented.


13

4.4 Section 4: Capturing wider changes in the system or market Table6:Score:capturingwiderchangesinthesystemormarket


4.1Theprogrammehasanoverallplanforassessingsystemicchangesatprogrammelevel.

R 10 7

4.2Systemicchangesareassessedatmarketsystemslevelandbeneficiarylevelusingappropriatemethods.

R 50 404


Theprogrammehasasystemicchangeframework.Thesystemicchangepathwayshavebeendevelopedforallsectors.

Whilethemeasurementofmostparametersinthesystemicchangeframeworkdrawsonotherdatacollectionactivities,theprocessforconsolidatingdataandanalysingsystemicchangehasnotbeenfullydevelopedandthereforethesystemisnotyetfullyoperational.

4.5 Section 5: Tracking costs and impact

Table7:Score:Trackingcostsandimpact


5.1 Costsaretrackedannuallyandcumulatively. M 20 20

5.2Programme-wideimpactisclearlyandappropriatelyaggregated.

M 35 35

5.3 Costsareallocatedbymajorcomponentoftheprogramme. R 20 20


Costsaretrackedannuallyandcumulatively.Thedirectinterventioncostsareallocatedtoeachintervention.

Thesystemforaggregatingcommonimpactindicatorsisappropriate.Aggregatedimpactisestimatedannually.Overlapsincommonimpactindicatorshavebeencorrectedintheaggregationprocess.

4Compliancecriteria4.2.7and4.2.8havenotbeenscored.Theformalassessmentoftheresultsofsystemicchangesatthepathwayandbeneficiarylevelhasnotbeenconductedyet.


14

4.6 Section 6: Reporting costs and results Table8:Score:Reportingcostsandresults


6.1Theprogrammeproducesareportatleastannuallywhichdescribesresultstodate.

M 50 45

6.2 Resultsofgenderimpactarereported. R 10 10

6.3 Resultsofsystemicchangearereported. R N/A N/A6.4 Resultsarepublished. R 10 5


Impactsarereportedintheannualaggregationofresultsreport.Thereportprovidesinformationonresults,progressandqualitativeexplanationsonresultsandprogress.Costsarereportedseparatelyonamonthlybasis.

TheimpactassessmentshavenotbeencarriedoutyetasMDFhasdecidedtoconductthemlater.Actualreportedresultsaremainlybasedoninformationgatheredthroughmeetingsandreportsfrompartnersandrelativelysmallsamplemonitoringvisits.Whilethisisacknowledgedintheannualreportsthatreportedimpactfiguresarebasedonmonitoringvisits,thedefinitionofmonitoringvisithasnotbeenprovided.

Contributionsofotherprogrammesandactorsareacknowledgedinthereports.

ReportsonresultsarepublishedontheMDFwebsite. Costsarenotincludedinthepublishedreports.

4.7 Section 7: Managing the system for results measurement

Table9:Score:Managingthesystemforresultsmeasurement

No. Controlpoints Must/Recommended Stdmax.score Actualscore

7.1Theprogrammehasaclearsystemforusinginformationfromtheresultsmeasurementsysteminmanagementanddecision-making.

M 30 30

7.2Thesystemissupportedbysufficienthumanandfinancialresources.

M 50 50

7.3Thesystemiswellmanagedandintegratedwithprogrammemanagement.

M 50 50


15

Strengths WeaknessesMDFhasaclearsystemforusinginformationfromtheMRMsysteminmanagementanddecisionmaking.Therearesufficienthumanandfinancialresourcestomanagetheresultsmeasurementsystemforeachofthesectorsandtheprogrammeasawhole.Staffcanclearlyexplaintheirrolesandresponsibilities,andthoseofothers,relatedtoresultsmeasurement.Rolesandresponsibilitiesinresultsmeasurementareclearlydefinedandintegratedintojobdescriptions,stafforientationsandperformanceappraisal.

Staffhaveaccesstosufficientguidance(predominantlyprovidedbystaffratherthanwritten)onhowtoimplementtheresultsmeasurementsystem.

QualityassuranceprocessesfortheMRMofeachinterventionandforaggregationofcommonimpactindicatorsandDFATindicatorsareinplaceandfunctional.WrittenMRMoutputs,suchasanalysesandaggregationreports,arereviewed.


16

5. Summary of key areas for improvement Outlinedbelowarethekeyareasforimprovement:

• EnsurethatsufficientqualitativeinformationneedsonwhychangeshappenornotareincludedintheRMplan;

• Whenplanningtoassessattribution,ensurethatthecounterfactualsarecarefullyconsideredandreflectedintheexplanationsoftherationaleforchoosingparticularattributionstrategies;

• Ensurethatquantitativeandqualitativeinformationcollectediscompiledanddocumentedtoprovide“aconsolidatedview”ofperformance/statusofrelatedchangesintheresultschains;

• Operationalisetheapproachtoassessingprogressinpromotingsystemicchanges.

Annexes(providedasseparatedocuments)1.Overallandsub-sectorspecificratings2.Sub-sectorspecificfindings3.Listofdocumentsreviewed4.Listofinterviewsconducted

mdf fiji dced audit report 14jan19 · 2. summary of the programme and results measurement system...

Documents