independent evaluator proposal - kieren mccarthy · 2010. 6. 18. · icann’s accountability and...
TRANSCRIPT
INDEPENDENTEVALUATORPROPOSAL
AFFIRMATIONOFCOMMITMENTS,ACCOUNTABILITYANDTRANSPARENCYREVIEW
1
TABLEOFCONTENTS
EXECUTIVESUMMARY..............................................................................2ValueProposition....................................................................................... 2ProposedSolution...................................................................................... 2FirstProgressReport .....................................................................................................................2SecondProgressReport ................................................................................................................3FinalReport ........................................................................................................................................3
ErsoyluConsulting/YeastLogic’suniquequalifications ............................. 4PricingDetails ............................................................................................ 5ProfessionalServices......................................................................................................................5Travelcostsandexpenses............................................................................................................5BillingTerms......................................................................................................................................5
INDEPENDENTEVALUATORPROPOSAL.....................................................6Scope.......................................................................................................... 6Whattheevaluationcovers .........................................................................................................6Whattheevaluationdoesnotcover ........................................................................................7
Purpose ...................................................................................................... 8Timelineandconstraints............................................................................ 8Methodology.............................................................................................. 9Firstprogressreport ................................................................................................................... 10SecondProgressReport ............................................................................................................. 11FinalReport ..................................................................................................................................... 13Website.............................................................................................................................................. 14
Confidentiality.......................................................................................... 14Results...................................................................................................... 14TheEvaluators ......................................................................................... 15YeastLogic ....................................................................................................................................... 15ErsoyluConsulting........................................................................................................................ 15
References ............................................................................................... 16ErsoyluConsulting........................................................................................................................ 16YeastLogic ....................................................................................................................................... 16
2
EXECUTIVESUMMARY
ValueProposition
ICANN’sAccountabilityandTransparencyReviewTeam(ATRT)islookingforanindependentevaluatortoassesstheorganization’sprocessesandprocedurestoensuretheyare“designedandexecutedinamannerthatensuresaccountabilityandtransparencyandreflectstheinterestsofglobalInternetusers”.Thefocusoftheevaluator’sworkwillbeon“reviewingandassessingthequalityofthedecision‐makingasaresultoftheprocessesandprocedures”thatexistwithinICANN.TheATRThasproducedaquestionnaireandaskedmembersofthecommunitytosenditexamplesofwheretheybelieveICANN’sprocessesandprocedureshavenotworkedeffectively,aswellasputforwardsuggestionsforimprovement.IthasbeguntocompileinformationanddataaboutICANN’saccountabilityandtransparencyandwillusetheevaluator’sreportalongwithotherdocumentationandface‐to‐facemeetingswiththecommunitytocompleteitsreviewbeforetheendof2010.TheATRTwouldlikehelpfromanexperiencedevaluatorthatalsohasagoodunderstandingoftheuniquenatureofICANNinordertoreviewitsactivitiesanddeveloppracticalimprovements.ItisexpectingtheevaluatortoprovidetwoprogressreportsinJulyandAugustwithafinalreportbeforetheendofAugust2010.
ProposedSolution
FirstProgressReportErsoyluConsulting/YeastLogicwillworkwiththeATRT,ICANNstaffandcommunitymemberstodevelopalistofexamplesofdecision‐makingbytheorganizationthatmeritin‐depthreview.ThelistwillbedrawnupthroughreviewofresponsestotheATRTquestionnaireanddiscussionswithkeymembersofICANNstaffandcommunity.Eachexamplewillbeaccompaniedbypertinentdetailsanddatagleanedfrominitialinterviewsanddocumentationreview.Fromthelistofexamples,theATRTwillberequestedto
3
choose3‐4examplesofdecision‐makingthatitwouldliketohavereviewedforamorein‐depthcasestudy.Theexamplescanbedrawnfromacrossthesampleandbebasedonkeyvariables.Forinstance:
Onecasethatisbroadlyviewedasasuccess Onecasethatisbroadlyviewedtohavefallenshort Onecasewhoseoutcomeiscontentious/unclear
Adedicatedreviewwebsitewillseekinputfromthecommunityandencouragediscussioninordertopromotetransparencyandaccountabilityoftheprocessitself.
SecondProgressReportOneachofthechosencases,arangeofqualitativeandquantitativedatawillbedrawnfromextensivedocumentreviewandanalysis,stakeholdersurvey,keyinformantinterviewsandfocusgroups.Interviewswiththoseinvolvedinthecaseswillbesought(staff,community,andBoardwhereappropriate),documentationsurroundingeachcaseattainedandreviewed,andthereviewwebsitewillinvitebroaderinputandreview.Alldatawillthenbecompiled,shared,analyzedandinitialconclusionsdrawnout.AfeedbacksessionwiththeATRTandICANNstaffwillbecriticaltocheckassumptionsandanalysis,andbegintodrawconclusionsandlessonslearned.
FinalReportStartingfromfeedbacktothesecondprogressreportandworkingcloselywithICANNstaffandtheATRT,additionaldataandanalysiswillbecarriedout.Asecondroundoffeedbacksessionswillhelpdevelopinitialconclusionsintofinalconclusions.Thoseresponsibleforimplementinganysubsequentrecommendationswillbepresentedwiththeanalysisandencouragedtodeveloptheirownrecommendationsinfacilitatedfeedbacksessions,inordertoensurethatconclusionsandlessonsremaingroundedincontext.AfinalsetofconclusionsandrecommendationswillbeprovidedtoICANNstaffwitharequestforofficialresponse,withboththenprovidedtotheATRTinthecompletedfinalreport.
4
ErsoyluConsulting/YeastLogic’suniquequalifications
ErsoyluConsultingisasmall,woman‐ownedenterprisebasedinSouthernCalifornia.ErsoyluConsulting’slead,DrLeahErsoyluholdsadegreeinEnvironment&ResourceEconomicsandaPhDinPoliticalScience,withafocusonnonprofitorganizationsandpublicpolicy.BothqualificationsequipherwithacoreunderstandingofmanyoftheissuesattheheartofICANN’sprocessesandprocedures.DrErsoyluhasover10yearsofnon‐profitexperienceatboththeprogrammaticandmanagementlevels,andhasworkedwithnumerouscommunity‐basedorganizations.ShelecturesonNonprofitOrganizations,AmericanGovernment,UrbanPoliticsandPublicPolicyAnalysisatUniversityofCalifornia,IrvineandCaliforniaStateUniversity,LongBeach.Assuch,sheisideallypositionedtounderstandthedifferentculturesandapproachesthatexistwithintheICANNmodel.SarpErsoyluholdsaBachelorsdegreeinMechanicalEngineeringandaMastersinBusinessAdministration(MBA).Heisaregisteredcivilengineerandhasnearly10yearsofexperienceinmanaginglarge‐scaleinfrastructureprojects,fromdesigntodelivery.HeisanexpertinCriticalPathManagement(CPM)scheduling–ahighlyvaluedmethodofprojectmanagement–andhasextensivebudgetingandprojectmanagementexperience.MrErsoylubringswithhimanin‐depthunderstandingofthetechnicalandbusinesspressuresonICANN,aswellasexpertiseinmanagingcomplexmulti‐dimensionalprojects.YeastLogic’sfounder,KierenMcCarthyholdsaMastersdegreeinMechanicalEngineeringwithManagement.BetweenFebruary2007andNovember2009,heactedasICANN’sGeneralManagerforPublicParticipation.Asalong‐timeobserverandparticipantinICANN’sprocessesfromboththeoutsideandtheinside,hepossessesaninvaluableunderstandingofthecomplexenvironmentthatICANNoperateswithin,aswellasreal‐worldexperienceoftheorganization’sprocessesandprocedures.MrMcCarthywasamajorcontributortoeachofthethreepreviouseffortsaimedataddressingtransparencyandaccountabilitywithinICANN.Heactedasacontactpoint,staffresource,aswellasauthorandcopyeditortotheOneWorldTrust
5
reportin2007,AccountabilityandTransparencyFrameworksandPrinciplesin2008,andtheImprovingInstitutionalConfidenceconsultationin2008and2009.DrErsoyluandMrMcCarthyaremembersoftheAmericanEvaluationAssociation;MrMcCarthyisalsoamemberoftheEuropeanEvaluationSociety.AllevaluatorsarebasedinCalifornia,closetoICANN’sheadquartersinMarinadelRey.
PricingDetails
ProfessionalServices
Stage Weeks YLK ECL ECS Totalhours
Cost
Rate 175 175 175 1stReport 3 80 50 20 150 $26,2502ndReport 2 60 70 40 170 $29,750FinalReport 3 70 80 50 200 $35,000
Total 210 200 110 520 $91,000
TravelcostsandexpensesTheprojectforeseesfourtripstoICANN’sheadquartersinMarinadelRey,onetoWashingtonDCandonetoBrussels.Exactcostingwillvarydependingondates,butaninitialestimatebasedoncurrentratesis$8,000.Out‐of‐pocketexpenseswillbebilledatactualcostinadditionaltoprofessionalfees.
BillingTermsAninitialretainerof15percentwillbedueatcontractsigning.Thereafter,billingwillbetwicemonthlybasedonhoursworkedandanyexpensesincurred,exceptinthelastinvoicewhentheinitialretainerwillbeconsidered.IfICANNortheATRTrequestanychangesinthescopeoftheproject,theadditionalcostsandeffortwillbeestimated.Allchanges,includingthetimetoassesstheimpactandcostoftherequest,willbebilledonatimebasisattheprojectrates.
6
INDEPENDENTEVALUATORPROPOSALWeareproudtoputforwardourproposalforanindependentevaluationofICANN’sprocessesandprocedures.Thedetailscontainedthereinareaninitialapproachandweremainopentosuggestionsforimprovementorenhancement.
Scope
Oneofthemostcrucialaspectsinanyeffectiveevaluationisaclearlyidentifiedscopefortheproject.Inthiscase,wehavetakensectionsfromtwoparagraphsintheRequestforProposals1producedbytheAccountabilityandTransparencyReviewTeam(ATRT)asspecifyingpreciselywhatthatscopeis.TheRFPaffirmsthattheintentofthereviewisto“identifywhetherICANN’sprocessesandproceduresaredesignedandexecutedinamannerthatensuresaccountabilityandtransparencyandreflectstheinterestsofglobalInternetusers”.Lateron,theRFPspecificallynotesthat“theATRTisnotseekinganauditofwhetherprocessesandproceduresareinplace(i.e.,aSarbanes‐Oxleyaudit),butratherafocusonreviewingandassessingthequalityofthedecision‐makingasaresultoftheprocessesandprocedures.”Thedocumentthenreferencesapreviousstudy,seriesofadoptedframeworksandprinciples,andconsultationasearlierworkdonewithrespecttoICANN’saccountabilityandtransparency.Italsodrawsattentiontoacurrentpublicconsultation2,runbytheATRT,regardingICANN’scurrentpracticesandprocedures.
WhattheevaluationcoversTakingtheseindicationsintoaccount,weproposethatthisevaluationfocusonpracticalissues‐ratherthanideological,sociologicaloreconomic‐andsoconcernitselfwithevaluatingafinitenumberofdecision‐makingcaseswithinICANN.
1http://www.icann.org/en/reviews/affirmation/rpf‐atrt‐02jun10‐en.pdf2http://www.icann.org/en/public‐comment/public‐comment‐201007‐en.htm#atrt%20
7
Theseexampleswillthenbeevaluatedwithrespecttofivecriteria:1. Overallaccountabilityofthedecision‐makingprocess.Inparticular,howfar
theprocessesandproceduresfollowedconsideredandincorporatedtheexistingguidingdocuments,frameworksandprinciples
2. Overalltransparencyofthedecision‐makingprocess3. The“quality”oftheendresult.By“quality”wereferto:
ThedegreeinwhichtheinterestsofglobalInternetusershavebeenmetThedegreeinwhichtheBoardmemberswereinconsensuswiththedecisionThedegreetowhichstaffwassuccessfulinimplementingtheresultThedegreetowhichthedecisionhelpedtofurtherICANN’smissionThedegreetowhichthedecisionimpactedkeystakeholderorpartnerrelationships(eithernegativeorpositively)
Throughthisassessmentofcasestudies,theevaluationwillthenidentifycommonthreadsthatinhibitorotherwisedisruptthegoalofprovidingatransparentandaccountableprocessthatproducesaresultofacceptablequalitythatrestsintheoverallinterestofglobalInternetusers.Throughreviewanddiscussionofthosecommonthreads,theevaluationteamwillthenworkcloselywithICANN’sprimarydecision‐makerstoidentifyworkableimprovementsintheorganization’sprocessesandprocedures.
WhattheevaluationdoesnotcoverICANNplaysanintriguingandever‐changingroleinthebroadercontextofgovernanceoftheInternet,particularlywithregardtoits“multi‐stakeholder”natureandthelevelofautonomyitenjoysglobally.Theorganizationisalsoremarkablefortheongoingreviewstoitsownstructure.Neitherofthesecomponentswillbeconsideredbythisevaluation,whosefocuswillbepurelyonreviewingdecisionsmadebyICANNinthecontextoftheorganization’sownstatedgoalsandstandards.Theevaluationwillalsonot:
Evaluatetheperformanceofanyindividualorgroup Considertheintroductionofnewprocessesorprocedures Investigateanydisagreementsordisputesthatoccuroutsideofthe
organization’sownprocesses
8
Purpose
TheRFPstatesthatthepurposeoftheevaluationistoprovide“anassessmentofthedecision‐makingattheInternetCorporationforAssignedNamesandNumbers”.Inanyevaluation,however,thereareinherentquestionsrelatedtobestpracticeguidelinesthatmustconsidered:
Whatarethequestionsthattheclientwantsansweredthroughtheevaluation?
Oncegenerated,howarepeoplegoingtousethisinformation?Thecontextofthisevaluationisaspartofaseriesofnewreviewswrittenintoan“AffirmationofCommitments”signedbetweenICANNandtheUSgovernmentsinSeptemberlastyear.ThatAffirmationreplacedtheprevious“JointProjectAgreement”(JPA)betweenthetwoparties.TheAffirmationofCommitmentsgivesICANNgreaterautonomytoactwithoutdirectoversightoftheUnitedStatesgovernment.Aseriesoffourreviewsareintendedtoreplacethatoversightrole,providing“globalInternetusers”withameanstoensurethattheorganizationcontinuestoworkintheirbestinterests.Inthiscontext,thereareseveralimplicitelementstothisevaluation:
ThatitshouldhighlightthepositiveaspectsofICANN’sprocessesandproceduresaswellastheareasthatcouldbeimproved
ThatitshouldbeaccessibleandunderstandabletoallInternetusers ThatitshouldconsidertheneedsofallInternetusers,regardlessoftheir
engagementinexistingprocesses Thatitshouldprovideusefulandactionableinformationforimprovements
tocurrentprocessesandprocedures Thatitshouldprovideanobjectiveanalysisbyindependententities
Timelineandconstraints
ThetimelineoutlinedintheRequestforProposalsisveryaggressive.Withthecontractawarded,afirstprogressreportisexpectedlessthanthreeweekslater.Asecondprogressreportisthenrequestedtwoweeksafterthat,withthefinalreportdeliveredthreeweekssubsequenttothesecondreport.Consideringtheself‐acknowledgedcomplexityofICANNasanorganization,thefactthatthewindowfallsinthemiddleofthesummerholidaysformostofICANN’scommunity,andthebroadnatureoftheevaluationitself,thisisaverytighttimeframeinwhichtoprovideaworkablefinalreport.
9
Webelievethat,giventheextensiveknowledgeofICANNbyonememberoftheevaluationteam,agoodoverallresultcanstillbeachieved.Evenso,effectiveprojectmanagementwillstillrequireseveralconsiderationstobecloselyobservedbyboththeATRTandICANN:
Immediateprovisionofrelevantdocuments Timelyresponsestorequestsforinformation DirectaccesstoATRTmembersandallmembersofICANN’ssenior
management ProactiveoutreachbyboththeATRTandICANNtorelevantstakeholders
Weconsideritcriticalthataseparatereviewwebsitebesetupspecificallytoprovideinterestedpartieswithasimpleanddedicatedonlineresourcefortheevaluation.Wehaveincorporatedthedesignandstagingofthatsiteintoourworkestimates.Intermsofprovidingusefulandworkableresultsinthetimeframepresented,wehaveproposedatargetedcasestudymethodthatwillallowustorapidlydrawoutkeyissuesandlessonsaboutICANN’sdecision‐makingprocessesandprocedures.Byexploringalimitednumberofdiverse,specificdecisionsmadethroughtheICANNprocess,itwillbepossibletoanalyzeandevaluatetheorganizationfromapracticalandtime‐sensitiveperspectiveaswellasdrawoutconclusionsthatwillbeapplicabletofutureprocessesandprocedures.Duetothetimewindowforthereview,itwillbenotpossibletocarryoutacost‐effectivenessstudyoftheprocessesorprocedures,ortheadditionalcost‐benefitofmakingimprovementstothem.Itwillalsoonlybepossibletodoaverylimitedreviewofparticipatoryengagementprocedures.Wewillprovidecursorysuggestionsastowhethertheproceduresthemselvesareeffectivelyengagingthosethatshouldbeincluded,andwhatcouldbedonetomaximizetheopportunitiesforinvolvementinthosenotcurrentlyengagedintheprocesses.
Methodology
TheAccountabilityandTransparencyReviewTeamhassetthefocusforitsworkthroughobservingspecificexamplesofdecision‐makinginits“QuestionsfortheICANNCommunity”document,released18May20103.
3http://www.icann.org/en/reviews/affirmation/community‐questions‐18may10‐en.pdf
10
Mostofthequestionsaskcommunitymemberstoidentify,withprecision,examplesoftimeswhentheyfeelICANNhasfallenshortinitsaccountabilityandtransparencyprinciples,ortoidentifyspecificstepsorrecommendations.Wehavetakenthisdocumentasaproceduralguideindesigninganeffectiveevaluationmethodologygiventhetimeconstraints.Assuch,weproposetonotonlyfollowthisfocuson3‐4casestudiesbuttoalsousetheresultsofthequestionnaireitselfasastartingdatapoint.Wedifferfromthequestionnaire’sapproachinonekeyelementhowever:whereasthequestionnaireseekspreciseexamplesofwhereICANNisnotperformingadequatelyi.e.itisseekingtofindinformationwithwhichtoperformacriticalanalysis,webelievethatinachievingthegoalsoftheevaluationitwillbenecessarytoalsoreviewatleastonecasewheretheendresultandprocessareviewedfavorablybythebroadercommunity.Inthisway,itwillbepossibletocompareprocessesandproceduresthatareseentohavebeensuccessfulalongsidethosethatareseentohavefallenshort.ThiscomparativeanalysisislikelytoyieldresultsthataremorebroadlyapplicabletoICANNoverall.
FirstprogressreportThebroadoutlineforthefirstprogressreportwillbetoidentifyandsharewiththeATRTarangeofdecision‐makingcases(ideally7‐15cases)fromwhichtochoosealimitednumber(3‐4)forin‐depthanalysis.Thefirststepinthisprocesswillbetoreviewquestionnaireresultsinordertoidentifyanydecision‐makingcasesthatarerepeatedlyreferencedbymembersofthecommunityasexamplesofafailureinaccountabilityortransparencyonthepartoftheICANN.Itmaybethattheendresultofthesecriticizedcasesisacknowledgedasbeingpositive,howeveritcanbeexpectedthattheyarethefocusofcriticismbecausetherewasnotconsensusaboutthequalityoftheoutcome.ItwillbenecessarytoalsotalktoATRTmembers,ICANNstaffandcommunitymemberstoelicitcasesofpositivedecision‐making–wheretheprocessesandprocedureswereseentoworkandproduceaviableandpositiveresult.ItislikelythatICANN’sstaffwillbethebestsourceofsuchexamples,butitisalsopossiblethatcommunitymemberswishtopromoteexamplesofwhentheyfeelICANNprocesseshaveworked.Inamidtheseexamplesofgoodandpoorcases,therewilllikelybeamultitudethatgravitatesomewherebetweenthetwo.
11
Inthefirstprogressreport,theintentwillbetoprovidetheATRTwitharangeoftheseexamplesalongwithsufficientdetailtoallowforaninformeddecisionaboutwhich3‐4casestoexamineingreaterdetail.AmatrixwillbepreparedfortheATRTthatdescribeseachcaseusingalistofkeygovernanceandprocessvariablesincludingbutnotlimitedtothefollowing:
Consensusaboutthevalueoftheendresult(good,bad,mixed) Therelativeefficiencyandeffectivenessoftheproceduresandprocesses
usedtogettotheendpoint(fast,slow,toofast,tooslow) Relevantdatasets–anddatathatcanbeexpectedtobeforthcomingwith
greateranalysis Therelativesizeandimportanceoftheprocess(small,focusedprocesswith
limitednumberofaffectedstakeholders;orlargeprocesswithbroadappealandresponse)
AnyinformationthatmaketheexamplesaparticularlygoodorrelevantcasestudyintermsofabroaderreviewofICANN’sprocesses
Facedwiththislistofbetween7‐15cases,theATRTwillbeencouragedtopickjustthreeforin‐depthreviewandanalysis.Thediversityofthe3‐4casestudiescanbebasedonwhichevervariablestheATRTdeemsmostcriticalforanalysis.However,oursuggestionatthistimeistoconsiderchoosingcasesbasedonthefollowingcriteria:
Onecasethatisbroadlyviewedasasuccess Onecasethatisbroadlyviewedtohavefallenshort Onecasewhoseoutcomeiscontentious/unclear
Atthisstage,itisnotpossibletopredictwhichexamplesarelikelytoyieldlargeamountsofanalyzabledataand/orwidespreadcommunityinterest.Howeverasexamplesofthesortsofprocessesmayarise,hereisalistofpotentialcandidates:
RevisionoftheRegistrarAccreditationAgreement ChangestotheAddGracePeriodfordomainnames TrademarkProtectionsinthenewgTLDprocess Whois Redelegationrequests Inter‐registrartransferpolicy UniformDisputeResolutionProcess InternationalizedDomainNames
SecondProgressReport
12
Once3‐4representativecaseshavebeenchosen(itwillnotbepossibletoreviewmorethanthisgiventhetimeconstraints),theevaluationwillthenmoveontogatheringthecasestudydata.Forthis,arangeofqualitativeandquantitativedatadrawnfromextensivedocumentreviewandanalysis,stakeholdersurvey,keyinformantinterviewsandfocusgroupswillbeused.Eachapproachhasitsadvantagesandshortcomings,summarizedbrieflybelow.Ofparticularfocusfortheevaluationteamwillbeschedulinginterviewswithrelevantstakeholdersaswellasdesigningappropriateonlineresponsetoolstoallowforbroaderinformationgathering.
Method OverallPurpose Advantages Challenges
Questionnaires,surveys
Abletogatherlargeamountsofdatafromabroadcross‐sectionofpeople
‐Inexpensive‐Fast‐Easytoanalyze‐Breadthofinformationgathered‐Anonymity(ifdesired)
‐Broadfeedback‐Self‐selectingrespondents‐Designcancauseaccidentalbiases‐Impersonal‐Doesnotallowforin‐depthcontextualinsights‐Doesnotallowforiterativedialogue
Interviews&Focusgroups
Gaindeeperunderstandingofknowledge,perspectivesandexperiences;uncovercontextualfactorsandissuesnotdiscoverablebyothermeans
‐Rangeofdeep,contextualinformationandbackgroundonprocessesandlogic‐Cangetperspectivefromindividualskeytodecision‐makingprocess
‐Timeconsuming‐Stakeholdersmustbenotifiedthattheirinsightsareneededbytheevaluationteaminadvance.‐Personalbiasescaninfluencememoryofevents
Documentationreview
Seeoriginaldesignandintentionofprocesses;abletoreviewlargeamountsofdatawithoutdisruptingstaff
‐Comprehensiveandhistoricaloverview‐Informationalreadyexists
‐Timeconsuming‐Canmisshumanaspectofdecisions‐Needtobeclearaboutwhatlookingfor‐Restrictedtowhatexists
13
Byreviewingalldocumentationrelatedtoeachidentifiedcase,wewillidentifytherelevantstakeholdersthatwerecriticaltoeachcasestudy.Theseindividualswillbecontactedtoparticipateinin‐personinterviews.Theseindividualsmaybestaff,Boardorcommunitymembers.Inordertogainperspectiveonhowthosenotintimatelytiedtothatdecision‐makingprocessviewedthedecision,wewillconductasurveyofotherindividuals.ThiswillbedoneeitherthroughthewebsiteorthroughanotherforumthattheATRTdeemsappropriate—togarnertheiropinionsonthedecision‐makingprocess.Asummarydocumentofthisinformation(bearinginmindconsiderationsofconfidentiality‐seebelow)willbemadeavailableonline,allowingforbroaderpeerreviewanddiscussiononeachcase.Oncethedatafromthesummarydocument(andonlinecomment)hasbeencompiled,thesecondreportwillbesharedwithICANNstaff.TheICANNstaff,throughaseriesoffeedbacksessions,willhavetimetoprovideonemoreroundofanalysisandfeedbackonthecomparativecasestudythatwaspresented.Followingthefeedbacksessions,theevaluatorswillincorporatethesuggestionsandissuesraised,beginningtheanalysisthatwillcomprisethefinalreport.
FinalReportThesecondprogressreportwillprovideabaselinecomparativeanalysisofthe3‐4targetedcasestudies.OnceICANNstaffandATRTparticipateinfeedbacksessionsreviewingthiscomparisondocument,theprocessofidentifyinglessonslearnedanddrawingoutconclusionsforthefinalreportcanbegin.Atthisfinalroundofanalysis,theneedformoredataorfurtheranalysiswillbeclear.Usingthisfeedback,andworkingcloselywiththeprimarydecision‐makers–inthiscase,ICANNstaff,theATRTandcommunityleaders–itwillbepossibletoteasethisout.Inordertomaketheendresultaseffectiveaspossible,thoseresponsibleforimplementinganysubsequentrecommendationswillbepresentedwiththeanalysisandencouragedtodeveloptheirownrecommendationsinfacilitatedfeedbacksessions,withtheresultsfromthatfedbackintotheprocess.Byallowingthoseresponsibleforrunningtheprocesstohaveadirectsayinsuggestionsforchangingit,itispossibletoputrecommendationsintoacontextthatincludesvalues,experienceandlikelyresourcesi.e.achieveapragmaticgoal.Publicreviewontheaccompanyingwebsiteofthisprocessshouldhelpidentifyremainingdivergencesorareaswhereconsensusisabsent–whichcanthenbenotedinthereport.
14
Followingbestpractice,afinalsetofconclusionsandrecommendationswillthenbeprovidedtoICANNstaffwitharequestforanofficialresponse,whichcanthenbeappendedtothefinalreportandpresentedtotheATRTforconsideration.
WebsiteEvaluationpromotesdialogueandimprovesco‐operationbetweenparticipantsatalllevelsinthedevelopmentprocessthroughmutualsharingofexperiences.SinceICANNisanInternetorganizationitcanassumeahighlevelofonlinefluencyinitscommunitymembers.AndconsideringtheglobalnatureofICANN’swork,aswellasthefactthatthereviewconcernsitselfwithtransparencyandaccountability,itisvitalthatthereviewitselfconformtomodernInternetrequirements–thatinformationiseasilyaccessibleonlineatalltimesofday,andthatpeopleareabletocommentanddiscussthatinformationattheirownleisureandwithaminimalamountofadministration.Forthisreason,weproposesettingupadedicatedreviewwebsitethatisfocusedontwogoals:
1. Immediacyofinformationregardingthereview,and2. Simple,openmethodsofprovidinginputanddiscussion
Thewebsitewillbeanintrinsicpartofthereview,withcommentsanddiscussiononitgivenasimilarweightingtootherformsofinputgathering.Thosethatwishtointeractwiththereview,eventotheextentofinterpretingthegathereddatawillbeencouragedtodoso.
Confidentiality
Althoughthereviewisfocusedonaccountabilityandtransparency,andtheevaluationteamadvocatespublishingitsowndataandanalysisinordertogatherongoingfeedback,itisalsonecessarytorecognizethatsomeinformationgatheredduringthecourseoftheprocesswillbeconfidentialinnature.Whereitispossible,thismaterialwillbemadeavailable,althoughhonoringrespondentconfidentiality.
Results
Asdiscussedearlier,theintentoftheevaluationistoprovidetheATRTwithworkable,verifiableandusefulinformationthroughwhichitcanunderstandICANN’sprocessesandproceduresbetterandwhichpointtopossibleimprovementsinthoseprocessesandproceduresinthewiderinterestsofglobalInternetusers.Althoughitistoosoontospecifywhatsortofconclusionsandrecommendationswillresult,itisworthaddressingtheissueofpost‐evaluationreviewnow,sincethisisacrucialfactorinmakinganyevaluationeffective.
15
Inordertohavingalastingimpact,itisimperativethatanyevaluationisitselfreviewedsometimeperiodafterithasbeendelivered–typically12months–inordertomeasurethedegreetowhichithasbeenadopted(ornotadopted),andtheimpactsubsequentchangeshavehad.Inthisway,anorganizationisabletolearnandadapttoitsownblindspotsandculturalshortcomings.Converselywithoutsuchareview,anorganizationmayfinditrepeatsthesamemistakesseveraltimesover.
TheEvaluators
YeastLogicYeastLogicprovidescommunication,strategic,evaluativeandsocialmediaadviceandservicesforcompaniesintheInternetarena.Wespecializeinengagementofonlinegroups,aswellasexpertadviceonadvocacy,media,strategiccampaignsandissuessurroundingInternetgovernance.OnlineparticipationWithmorecustomers,communitymembersandcompetitorsengagingwithorganizationsonline,weofferadviceandtrainingonhowtoengagewithpeopleandgroupsonlineandwellasencouragingparticipationandinteractionthroughonlinetools.StrategicadviceWhetherseekingtopersuaderegulators,stakeholdersorexternalevaluatorsofaparticularcasewithadefinitegoalinmind,wecanprovidestrategicguidanceandadviceonhowbesttoachievethataim,includingtheuseofthemedia,onlinecampaigningandface‐to‐facenegotiation.
ErsoyluConsultingErsoyluConsultingprovidespolicyresearch&analysis,programdesign&evaluationandstrategicprogrammanagementservicesfornon‐profitorganizations,publicagenciesandotherinstitutions.Wespecializeinhelpingourclientsaccuratelyresearch&evaluateissues,makeeffectivepolicydecisionsandattaintheirprogramgoals.Wesetachievableandrealisticoutcomemeasurementswhenconductingprogramanalysis,work‐plansordynamiclogicmodels.Ourexpertiseisinensuringfullparticipationofstakeholdersinprogramdesign,interpretationofresearchandevaluationfindingsandpolicyanalysis.
16
PolicyResearch&AnalysisWeconductcarefulresearch,analysisandevaluationofgovernanceandpolicymakingprocessestosupportcommunityefforts.Weusebothquantitativeandqualitativemethodstoproducetimely,relevantdatafororganizations.ProgramDesign&EvaluationWebringvariousanalyticaltoolstobearonorganization’sprogramsandprojects.Wehaveexperienceinbothqualitativeandquantitativemethodssuchasfocusgroupfacilitation,surveydata,processandoutcomeevaluation,communityassessments,movementandcoalitionbuildingandgeneraltechnicalsupporttoadvocacyefforts.PlanningServicesStrategicPlanning,OrganizationalAnalysisandCharrettesareallwaysthatlocalgovernmentsandagenciescanensurethattheyarebest‐usingtheirlimitedresourcesinefficientandeffectiveways.Weareabletoprovidetheseservicesonvariousscales.Governanceandaccountabilityarekeyissuesthatsuccessfulorganizationsmusthaveconsensusandclarityoninordertohaveamaximumimpact.Communityparticipationisacriticalelementtothis;wehelpnon‐profitorganizationsandpublicinstitutionsbestrespondtotheseneedsinasophisticatedandtimelymanner,ensuringmaximumstakeholderparticipationintheprocess.
References
ErsoyluConsultingDr.ElisaNicholasExecutiveDirectorTheChildren’sClinic(MemorialHospital)[email protected]&RecreationAgencygmouet@santa‐ana.org
YeastLogicPaulLevinsFormerVicePresidentofCorporateAffairs,[email protected],[email protected]