better measurement for performance improvement in low‐ and

Original Investigation

Better Measurement for PerformanceImprovement in Low- and Middle-Income

Countries: The Primary Health CarePerformance Initiative (PHCPI) Experience of

Conceptual Framework Development andIndicator Selection

JEREMY VEILLARD, ∗,† KRYCIA COWLING, ‡,∗ASAF BITTON, §,‖ HANNAH RATCLIFFE , §,‖

MEREDITH K I MBAL L , ‖,# SHANNON BARKLEY, ∗∗LAURE MERCEREAU, ∗ ETHAN WONG, ††

CHELSEA TAYLOR, # L I S A R . H I R S C H H O R N , ‡‡a n d H O N G WA N G ††

∗The World Bank Group; †Institute of Health Policy, Management, andEvaluation, University of Toronto; ‡Bloomberg School of Public Health, Johns

Hopkins University; §Ariadne Labs, Brigham and Women’s Hospital;‖Harvard T.H. Chan School of Public Health; #Results for Development;

∗∗The World Health Organization; ††The Bill & Melinda Gates Foundation;‡‡Feinberg School of Medicine, Northwestern University

Policy Points:

� Strengthening accountability through better measurement and report-ing is vital to ensure progress in improving quality primary health care(PHC) systems and achieving universal health coverage (UHC).

� The Primary Health Care Performance Initiative (PHCPI) providesnational decision makers and global stakeholders with opportunitiesto benchmark and accelerate performance improvement through betterperformance measurement.

� Results from the initial PHC performance assessments in low- andmiddle-income countries (LMICs) are helping guide PHC reforms

The Milbank Quarterly, Vol. 95, No. 4, 2017 (pp. 836-883)c© 2017 The Authors The Milbank Quarterly published by Wiley Periodicals, Inc. on

behalf of The Millbank Memorial Fund

This is an open access article under the terms of the Creative Commons AttributionLicense, which permits use, distribution and reproduction in any medium, provided theoriginal work is properly cited.

836

http://creativecommons.org/licenses/by/4.0/

The Primary Health Care Performance Initiative 837

and investments and improve the PHCPI’s instruments and indica-tors. Findings from future assessment activities will further amplifycross-country comparisons and peer learning to improve PHC.

� New indicators and sources of data are needed to better understandPHC system performance in LMICs.

Context: The Primary Health Care Performance Initiative (PHCPI), a collabo-ration between the Bill and Melinda Gates Foundation, The World Bank, andthe World Health Organization, in partnership with Ariadne Labs and Resultsfor Development, was launched in 2015 with the aim of catalyzing improve-ments in primary health care (PHC) systems in 135 low- and middle-incomecountries (LMICs), in order to accelerate progress toward universal health cov-erage. Through more comprehensive and actionable measurement of qualityPHC, the PHCPI stimulates peer learning among LMICs and informs decisionmakers to guide PHC investments and reforms. Instruments for performanceassessment and improvement are in development; to date, a conceptual frame-work and 2 sets of performance indicators have been released.

Methods: The PHCPI team developed the conceptual framework through lit-erature reviews and consultations with an advisory committee of internationalexperts. We generated 2 sets of performance indicators selected from a literaturereview of relevant indicators, cross-referenced against indicators available frominternational sources, and evaluated through 2 separate modified Delphi pro-cesses, consisting of online surveys and in-person facilitated discussions withexperts.

Findings: The PHCPI conceptual framework builds on the current under-standing of PHC system performance through an expanded emphasis on therole of service delivery. The first set of performance indicators, 36 Vital Signs,facilitates comparisons across countries and over time. The second set, 56 Diag-nostic Indicators, elucidates underlying drivers of performance. Key challengesinclude a lack of available data for several indicators and a lack of validatedindicators for important dimensions of quality PHC.

Conclusions: The availability of data is critical to assessing PHC performance,particularly patient experience and quality of care. The PHCPI will continueto develop and test additional performance assessment instruments, includ-ing composite indices and national performance dashboards. Through countryengagement, the PHCPI will further refine its instruments and engage withgovernments to better design and finance primary health care reforms.

Keywords: primary health care, measurement, health systems, performanceassessment.

838 J. Veillard et al.

T he international community has embarked on theagenda of the Sustainable Development Goals, with primaryhealth care (PHC) a critical platform for achieving universal

health coverage (UHC) efficiently, effectively, and equitably.1 Despiteglobal political commitment to investing in and improving PHC, manycountries have not prioritized strengthening PHC. Furthermore, the in-terventions driving PHC system performance improvement over time inlow- and middle-income countries (LMICs) are still poorly understood.2

Thus, even when countries decide to prioritize PHC, they often lackthe necessary information to pinpoint weaknesses, identify strengths,and improve their PHC systems.3,4 Better measurement of PHC systemperformance is a critical first step to identifying areas for reforms thatwill strengthen performance, as well as evaluating the impacts of thesereforms to guide continued performance improvement.

Filling the performance measurement gap in PHC is one of theprimary objectives of the Primary Health Care Performance Initiative(PHCPI), a collaboration of the Bill and Melinda Gates Foundation, theWorld Bank, and the World Health Organization, in partnership withAriadne Labs and Results for Development.4 Launched in September2015, the PHCPI’s aim is to catalyze improvements in PHC systems inLMICs and accelerate progress toward achieving UHC through better,more comprehensive, and actionable measurement of quality PHC andbetter insights into interventions that effectively address numerous per-formance gaps.4,5 The initiative is supported by a grant from the GatesFoundation over the period 2016-2020.

Through a series of novel conceptual and methodological develop-ments that distill new lessons from existing literature and provide newtools for performance assessment, the PHCPI is improving measurementand advancing understanding of the determinants of PHC system per-formance in LMICs. The first tool, which was introduced and describedby Bitton and colleagues, is a new conceptual framework of PHC systemperformance that emphasizes service delivery, improving on the existingframeworks’ traditional focus on inputs and outputs.4 The second toolis the compilation of 2 sets of indicators, “Vital Signs” and “DiagnosticIndicators,” to measure performance. These indicators reflect the prin-cipal dimensions of the conceptual framework, identify PHC systems’specific strengths and weaknesses, and are highly relevant at global,national, subnational, and, in some cases, facility levels. The third toolis the creation of a set of composite indices that provide more holistic


measures of PHC systems to assess relative performance across countriesand broad trajectories of performance over time, which can guide strate-gic and investment priorities of national and international financers ofPHC. The last tool is national PHC performance dashboards that providecomprehensive and multidimensional snapshots of each country’s PHCsystem performance, which will likely be used to inform policymakers,advocates, and diverse audiences about the strengths and weaknesses ofPHC systems. Companion tools are also being developed, such as anassessment guide that outlines the steps to comprehensively evaluatePHC system performance with participation from diverse stakeholdersas well as a solution strategies model that identifies typologies of PHCsystems’ strengths and weaknesses to help in selecting reforms to im-prove performance. With input from a broad range of global expertsand policy practitioners and based on testing in and engagement withpartner countries, each of these instruments is continually being refined.

In this article, we review the PHCPI experience with the conceptualframework development and the selection of core performance indicators,addressing the question of how to measure and understand key aspectsof PHC systems’ performance in LMICs. We also describe ongoing workon composite indices and performance dashboards and articulate anagenda for future research to develop better indicators of quality PHC inLMICs. We conclude with a discussion of possible strategies to improvedata availability, increase capacity to measure quality PHC in LMICs,and operationalize the measurement agenda to improve performance inLMICs.

Methods

Defining PHC

A substantial literature has debated the precise definitions of and dis-tinctions between primary care and primary health care.6 According toMuldoon and colleagues, primary care “describes a narrower conceptof ‘family doctor-type’ services delivered to individuals,” whereas pri-mary health care is broader, encompassing “both services delivered toindividuals (primary care services) and population-level ‘public health-type’ functions.”7 The PHCPI’s conception of primary health carefollows this broader definition, incorporating prevention and healthpromotion at the societal, community, and individual levels, in addition


to the provision of primary care services. The PHCPI endorses a vision ofa well-functioning PHC system as one that can manage and fulfill eachof these roles, and it is this conception of PHC that guides its work.

International Experts Advisory Committee

The development of each of the PHCPI instruments reflects substantialinput from a group of approximately 30 international experts, represent-ing a broad range of expertise in measurement, health systems research,and primary health care reform implementation in low-, middle-, andhigh-income countries, including leaders from various LMICs’ ministriesof health. Experts were identified through the authors and references ofseminal publications in the field of health systems performance assess-ment and leading institutions in the fields of PHC and health systemsstrengthening, and were selected to provide a diversity of perspectivesfrom a range of professional backgrounds with global representation,including policymaking at the national level. In future iterations of itsadvisory committee, the PHCPI seeks to further improve this range ofperspectives by including more experts from currently underrepresentedworld regions and by incorporating more experts with a specializedknowledge of primary health care delivery, such as family medicine. Todate, the PHCPI has convened 3 major international advisory meetings.More than 50 country-level, NGO, World Bank, and academic expertsattended the initial global input meeting in July 2014. Two subsequentmeetings of the PHCPI international expert advisory committee wereattended by 13 experts in July 2015 and by 16 experts in November2016. In addition, we conducted online surveys to help refine the selec-tion of the indicators before the meetings. Appendix 1 lists the expertswho participated in each of the 2 expert advisory committee meetingsand responded to each of the online surveys.

Country Engagement

Working in LMICs is fundamental to the PHCPI’s objectives, and coun-try engagement will progressively expand as the PHCPI assessmenttoolkit is further developed and finalized. To date, the PHCPI teamhas tested early versions of its various tools in a select group of coun-tries, chosen opportunistically to combine this work with ongoing re-lated activities by one or more partner organizations. The PHCPI’s


conceptual framework and indicators were first applied in Cameroon,helping elucidate subnational disparities in PHC performance acrossregions. Teams in Mexico used the PHCPI’s indicators to systemati-cally identify available data across each of the conceptual framework’sdomains, and the PHCPI conceptual framework was used in Rwandato guide the development of a performance management scorecard forutilization by the Ministry of Health. These early experiences providedvaluable feedback on the relevance of the conceptual framework to dif-ferent PHC systems and the feasibility of compiling sufficient data toestimate the PHCPI’s indicators and generate meaningful conclusionsabout PHC performance. As a next phase for country engagement con-tributing to tool development, the PHCPI is identifying countries andmaking plans to test its assessment guide. As this work expands to amore diverse set of countries, this will also help identify any issues aris-ing from the generalizability of the tools to different epidemiologicaland health systems contexts.

Conceptual Framework

Underlying the development of the PHCPI’s conceptual framework is anunderstanding that the organization and management of service deliveryfor better population health management as well as the quality of healthcare services are paramount aspects of performance. This emphasis addsa level of analysis to existing models and frameworks predominantlyfocused on system inputs and outputs. Historically, the focus of routinehealth systems measurement, as well as the majority of facility surveys,in LMICs has been on inputs, such as the total number of health work-ers, medicines, and supplies available, and outputs and outcomes, suchas intervention coverage and mortality rates.8,9 Policymakers, however,often lack data on the service delivery processes by which these inputsproduce the desired outputs and outcomes. Furthermore, the experienceof patients who receive care, constraints faced by health workers whoprovide care, and barriers encountered by people who may not interactwith the system are rarely measured. For example, systematic and com-parable data on how often health workers are present at health centersand the accuracy of their diagnoses are collected in very few LMICs.10,11

In other words, many LMICs lack measures of quality service delivery.12

In the first step of developing a conceptual framework, the PHCPIteam reviewed the literature on key characteristics and determinants of


high-performing PHC systems. This included a search of peer-revieweddatabases, as well as reports and other publications from multilateralorganizations. In the second step, the team reviewed approximately 40different frameworks and measurement platforms on PHC, health sys-tems, and health scorecards to identify their strengths, limitations, andcommon features (Appendix 2). During the initial convening meetingswith country consultants at the World Bank in July 2014, the PHCPIcollaborators gathered feedback from experts on the emerging concep-tual framework.

The PHCPI team then used this background research to develop adraft framework that was reviewed and discussed at the initial meeting,in July 2015, of the PHCPI’s international expert advisory committee.Based on this discussion, the team revised the draft framework and usedthe first version as the context for performance reporting in the initialrelease of the PHCPI’s website.4 During the following year, additionalinputs from a range of experts and partners and the experience of thePHCPI team in working with the framework informed a set of proposedupdates. In particular, initial PHC performance assessments with theteams in Cameroon and Mexico allowed us to explore the applicabil-ity of the theoretical framework to real-world settings. We presentedand discussed these tentative changes at the second advisory commit-tee meeting, in November 2016, and incorporated them in the secondversion of the framework, currently in use and supporting measurementactivities.

Performance Indicators

The first set of PHCPI performance indicators, the “Vital Signs,” consistsof internationally comparable indicators for diagnosing the level andtrend of PHC system performance. The second set, the “DiagnosticIndicators,” is an expanded set of indicators to understand the underlyingdrivers of core performance. In developing these two indicator sets, ourobjective was to measure strengths and weaknesses across all dimensionsof the conceptual framework and to measure the provision of key PHCfunctions through the lens of the patient, the provider, and the healthsystem.

Differing criteria guided the selection of Vital Signs and DiagnosticIndicators, which were chosen through coinciding processes of identi-fying candidate indicators, consulting experts, and selecting the final


sets. Vital Signs were selected for their ability to assess how an overallPHC system is performing. In order to facilitate cross-country compar-isons and trend analysis, indicators included as Vital Signs had to bevalidated, reliable, and available from as many LMICs as possible. Diag-nostic Indicators were chosen to provide insight into why performanceand outcomes vary between countries and how these can be improved.They were designed to be used in concert with the Vital Signs as part ofassessing performance at national, subnational, and facility levels. Whilevalidity and reliability were also priorities for Diagnostic Indicators, in-dicators with less internationally available data (eg, treatment accuracy,high blood pressure diagnosed and receiving treatment) were permittedin this set, as the goal of the Diagnostic Indicators was focused less oninternational comparability than on local applicability.

Figure 1 summarizes the process for selecting the Vital Signs andDiagnostic Indicators. First, we identified through a literature reviewthose indicators relevant to all domains of the conceptual framework.We searched specifically for indicators in the following relatively newareas of measurement, which reflect performance dimensions not tradi-tionally included in health systems assessments in LMICs: first-contactaccess, coordination, comprehensiveness, continuity, safety, and orga-nization and management. We then cross-referenced these indicatorsagainst existing indicators available globally, taken from the followinginternational sources:

� Health facility surveys: Service Delivery Indicators (SDI), ServiceAvailability and Readiness Assessment (SARA), Service ProvisionAssessment (SPA).

� Household surveys: Demographic and Health Survey (DHS), Mul-tiple Indicator Cluster Survey (MICS), World Health Sur-vey (WHS), Performance Monitoring and Accountability 2020(PMA2020).

� Global databases: Global Health Observatory (GHO), UNICEFData: Monitoring the Situation of Children and Women, WorldBank World Development Indicators (WDI).

� National Health Accounts (NHA), System of Health Accounts2011 (SHA 2011).

We found more than 600 indicators, which we narrowed down to a pri-ority list of approximately 100 indicators by subject-based experts from


Fig

ure

1.P

HC

PI

Indi

cato

rSe

lect

ion

Pro

cess


the PHCPI’s partner organizations, based on the criteria of importanceand relevance, and mapped to the PHCPI’s conceptual framework.4 Forthis set of potential indicators, we compiled definitions of numeratorsand denominators; exclusion criteria; information on validity and relia-bility; existing data collection mechanisms; and references to literaturesupporting the use of the indicator.

After receiving feedback from 10 independent measurement experts,we chose the initial set of 25 Vital Signs from this list of indicators. Wedid not use a more formal expert review process for the initial selectionof Vital Signs because these indicators were selected from establishedinternationally valid and comparable indicators. In 2016, the Vital Signswere updated using a modified Delphi approach to review existing andproposed new indicators.

Our selection of the Diagnostic Indicators was guided by a modi-fied Delphi approach,13 consisting of an online survey and an in-personmeeting, both drawing on the input of the experts advisory committee.Experts (n = 33) were approached via email to determine their inter-est and willingness to participate in the survey. All those consentingwere given a web link to the survey and provided with supplementarymaterials, including the PHCPI’s conceptual framework, a glossary ofkey terms, and reference sheets for every proposed indicator listing thesource, definition, rationale for inclusion, and limitations. The surveywas open for 3 weeks, and the participants received up to 3 weekly re-minder emails if they had not yet responded. The survey was completedby 23 experts (response rate of 70%).

In the e-Delphi survey, respondents were requested to complete 3activities. First, they were asked to rate each indicator on a scale from1 (not at all) to 5 (very strong) on the 5 dimensions of relevance, va-lidity, actionability, reliability, and feasibility. Second, they were askedwhether each indicator should be “included,” “included with adapta-tion,” or “excluded” from the Diagnostic Indicators. Third, we asked forrecommendations for additional indicators. The results from the onlinesurvey were summarized for the in-person meeting, during which ex-perts were asked to provide additional feedback on the indicators. Theaim of the modified Delphi approach was not to generate a final indi-cator list with complete group consensus but to gather feedback on theidentified indicators to inform the PHCPI’s measurement strategy andresearch and development agenda. The literature review, survey results,


and expert input were then consolidated to generate a final set of 56Diagnostic Indicators.

Updated Vital Signs

In 2016, after approximately 1 year of use, including application of theVital Signs for PHC assessment in Cameroon and Rwanda, we reviewedand expanded the 25 original Vital Signs to better cover measurementgaps in the performance dimensions of the revised conceptual frame-work. The PHCPI team selected an additional 15 indicators for possibleinclusion as Vital Signs from a range of sources. We then compiledPHC-relevant indicators from the indicators lists for Universal HealthCoverage,14 the Sustainable Development Goals,15 the World BankGlobal Financing Facility in support of Every Woman, Every Child,16

the WHO 100 Core Indicators17 from a WHO Eastern MediterraneanRegional Office working group on quality of care in PHC,18 and theWorld Bank Service Delivery Indicators.19 We selected those indicatorsmeeting the selection criteria earlier and available for more than 10countries as potential new Vital Signs. In addition, the definitions of4 existing Vital Signs indicators were modified. The 15 proposed newindicators, along with the 25 original or slightly amended Vital Signs,were then scored and considered through a modified Delphi approach,again entailing an online survey followed by an in-person meeting.

The online survey was sent to 23 experts, of whom 17 providedcomplete responses (response rate of 74%). The survey methods andquestions closely resembled the preceding Diagnostic Indicators survey,with minor changes in the phrasing of questions and response options.Respondents were requested to rate each indicator on a 7-point Likertscale against 4 criteria—relevance and importance, reliability, valid-ity, and actionability (Table 1)—and to indicate whether they believedan indicator “must be included” or “should not be included” as a Vi-tal Sign. Experts were also invited to suggest additional indicators forconsideration.

We summarized the survey results and shared them with partici-pants prior to the second expert advisory committee meeting, to in-form the in-person discussion. To obtain further input, in January 2017we also shared the list of 40 proposed indicators with participantsat a meeting of the Joint Learning Network PHC Measurement for


Table 1. Updated Vital Signs Indicator Selection Criteria

Category Description Operational Definition

Relevance andimportance

The indicator reflectsimportant aspects ofPHC systemperformance.

� Consistent withconceptualframework

� Amenable tointervention byPHC systems

� Aligned with otherglobal initiatives

Reliability The indicator producesconsistent results.

� Minimized standarderror

Validity The indicator is anaccurate reflection ofthe dimension of PHCsystems performancethat it is intended toassess.

� Minimizedmeasurement erroras compared withtrue value

Actionability The indicator is useful forPHC systemperformanceimprovement purposes.

� Indicator resultspoint to tangibleinterventions forperformanceimprovement,ideally supported bystrong evidence ofeffectiveness.

Improvement Collaborative,20 an international community of leadersfrom LMIC health systems, focused on knowledge sharing and collab-orative tool development for PHC performance improvement. Finally,we used correlation analysis to explore suspected collinearity betweenselected indicators that capture closely related information. After consid-ering the survey results, input from the various discussions, and resultsof the correlation analysis, we proposed a revised set of 36 Vital Signs tothe PHCPI’s steering committee (composed of the principals from the5 organizations), which they adopted. They also agreed that a processwould be carried out to revise the Vital Signs indicators on an annualbasis.


Results

Conceptual Framework

From the literature review of the characteristics and determinants ofhigh-performing PHC systems, we identified several key elementsof strong PHC. More than a package of services, quality PHC has4 core functions: comprehensiveness of promotive, preventive, curative,and palliative care services; continuity across the life cycle; coordinationacross service providers and levels of the health care system; and a pointof first contact access for the majority of patients’ health needs.21 Thesecore functions, first articulated by Starfield, have been broadly acceptedby and included in all PHC frameworks. The influential work on qualityof care by Donabedian, which emphasizes interactions among patients,providers, and communities as fundamental to quality,22 is also reflectedin many models of health system performance. Recent work in LMICsto assess the quality of primary care is also particularly informative;for example, a study in Brazil included an assessment of management,organization, and working conditions.23

In existing frameworks, domains such as efficiency are often depictedas cross-cutting attributes, as opposed to goals of a strong system.24,25

Dimensions reflecting “hardware” inputs (funds, human resources, sup-plies, facilities, and information systems) are prevalent,8 and those in-corporating “software” inputs (financing mechanisms, provider paymentincentives, regulations, and market structures) are less prevalent. LMICsdisproportionately measure hardware inputs, compared with HICs, pro-viding an opportunity for a new focus on holistic system functioningin LMICs, rather than only on the availability of inputs. In additionto the overreliance on hardware inputs, very few frameworks emphasizeintegrating PHC with other health care settings. Moreover, most frame-works pay little attention to the perspective of the people involved inthe system, that is, the providers, families, communities, and individualpatients that interact with the PHC system.

Of the existing frameworks for assessing PHC performance, the Pri-macy Care Assessment Tool (PCAT)26 and the Primary Health Care Ac-tivity Monitor for Europe (PHAMEU)27 are arguably the most closelyaligned with the measurement approach advocated by the PHCPI, al-though they were developed for the context of higher-income healthsystems. Previous applications of the PCAT tools in LMICs suggest that


these are adequate but require some adaptation.28,29 The PHCPI’s frame-work retains the principal aspects of the PCAT and the PHAMEU (eg,first contact access, coordination, and comprehensiveness) but divergesin its unique applicability to low- and middle-income settings. Forexample, the PHCPI indicators include the availability of basic equip-ment and access to treatment for infectious diseases that are not highlyrelevant to high-income settings. Addressing the aforementioned gapsin existing approaches to performance assessment also drives WHO’semphasis on integrated people-centered health services (IPCHS),30 thenew guiding framework for service delivery reforms to expand access tohealth services and achieve UHC. While IPCHS strategies span the en-tire health system, IPCHS requires prioritization of high-quality PHCas its cornerstone.

Figure 2 shows the PHCPI’s revised conceptual framework (the firstversion is provided in Appendix 3). It has 5 domains: System, Inputs,Service Delivery, Outputs, and Outcomes, and each has 5 or more subdo-mains and sub-subdomains. This conceptual framework draws on severalimportant earlier system frameworks, such as the logic model,31 ControlKnobs Framework,32 Health Systems Performance Assessment,9 eco-nomic models of supply and demand, and predefined key characteristicsof high-performing PHC systems.21,33,34 The PHCPI’s framework de-scribes the key inputs, functionalities, and desired goals of an effectivePHC system and focuses on the intersection between service deliveryand the core functions of PHC (first contact access, continuity, coordina-tion, comprehensiveness, and person-centeredness) as the main driversof performance variation. It also highlights people- and community-centered care, supply and demand functions, system responsiveness,resilience, and integrated service delivery through effective organizationand management.4

The framework structure is similar to the commonly used input-process-output-outcome logic model, indicating logical relationshipsbetween constructs. We placed a System-Level Determinants domainbefore the Inputs domain to indicate the importance of the modifiablePHC system structure emphasized in the Control Knobs Framework.32

In addition, we more clearly defined process as the various critical sub-domains of Service Delivery. The framework exhibits an overall direc-tionality of influence, where the System domain influences the Inputsdomain, which affects the complex interplay within the Service Deliverydomain, contributing to effective Outputs, which subsequently affect


Fig

ure

2.P

HC

PI

Con

cept

ualF

ram

ewor

k,as

Rev

ised

Nov

embe

r20

16[C

olor

figu

reca

nbe

view

edat

wil

eyon

line

libr

ary.

com

]

Abb

revi

atio

ns:P

HC

,pri

mar

yhe

alth

care

;RM

NC

H,r

epro

duct

ive,

mat

erna

l,ne

wbo

rn,a

ndch

ild

heal

th;N

CD

,non

com

mu-

nica

ble

dise

ase.


Outcomes. This framework also incorporates the health system goals forthe Outcomes domain—health status, responsiveness, equity, efficiency,and resilience—as articulated by numerous health systems performanceassessment frameworks. Finally, in accordance with a biopsychosocial,as opposed to a strictly biomedical, model of disease causation,35 weacknowledge that PHC performance lies within a larger health system,which itself lies within wider political, cultural, demographic, and so-cioeconomic contexts.

The revised version of the conceptual framework, finalized after thesecond experts advisory committee meeting, did not alter the 5 per-formance domains of System, Inputs, Service Delivery, Outputs, andOutcomes arranged in a logic model; however, the content of the Sys-tem, Service Delivery, and Outcomes domains were altered. We mademinor changes to the Outcomes domain to remove the components of“Reduced Mortality” and “Reduced Morbidity” from the subdomainsof Health Status to better match the level of specificity provided in theother subdomains of Outcomes. Additionally, the name of the subdomainof “Health System Responsiveness” was changed to “Responsiveness toPeople” to better reflect the span of responsibility of PHC systems, andthe System domain was revised to move the concept of “CommunityEngagement” from the subdomain of “Social Accountability and Com-munity Engagement” to the Service Delivery domain. The concept of“Payment Systems” had been missing from the subdomain of “HealthFinancing” in the first version and was added.

In the revised framework, we made the greatest number of changes inthe Service Delivery domain. Because we recognized that the “Organiza-tion and Management” subdomain was 2 different types of management,population health management and facility organization and manage-ment, we separated them and clarified the lower-level components ofeach subdomain. We reframed the “People-Centered Care” subdomainas “High-Quality Primary Health Care,” reflecting conceptual shiftstoward thinking of “person-centeredness” as a core function of high-quality PHC of equal importance to the 4 functions first laid out byStarfield21 and others at WHO.34 Alterations to “Availability of Effec-tive PHC Services” included moving “Safety” to this subdomain; adding“Patient-Provider Respect and Trust” to explicitly recognize this con-cept; and changing “Provider Absence Rate” to “Provider Availability.”Finally, we reorganized the 5 subdomains in Service Delivery to betterreflect the logical flow of service delivery, in keeping with the overall


flow of the conceptual framework and recognizing interdependenciesamong the subdomains.

Vital Signs

The original set of 25 Vital Signs (provided in Appendix 4) covered all5 domains of the conceptual framework, but many subdomains werestill not reflected in this indicator set. The Vital Signs revision in 2016resulted in 36 Vital Signs, 24 from the original 25 and 11 new indicators.Table 2 lists the 36 updated Vital Signs and the results of the experts’evaluation of each indicator in the e-Delphi survey. The average totalrating reflects the mean of the experts’ summed scores (from 0 to 6) oneach of the 4 criteria (relevance and importance, reliability, validity, andactionability). The number reflecting the qualitative recommendationswas constructed by counting each response to definitely exclude anindicator as −1 and each response to definitely include an indicatoras 1, and taking the mean across all responses. Providing this typeof feedback was optional for each indicator, so values closer to 0 mayindicate polarization in expert opinions or a lack of strong opinionsin either direction. The last column in the table also identifies thenumber of LMICs with available data for each indicator; the numbersin parentheses reflect the number of countries expected to have dataavailable soon as a result of data collection efforts currently under way.

Diagnostic Indicators

The modified Delphi approach we used in July 2015 resulted in theselection of 62 Diagnostic Indicators, which cover most of the originalconceptual framework’s subdomains; we did not find appropriate indica-tors for 7 subdomains. Table 3 gives the full list of Diagnostic Indicatorsand the average score for each indicator from the e-Delphi survey. Inthe update to the Vital Signs, 1 Diagnostic Indicator was reclassifiedas a Vital Sign (cervical cancer screening rate); there currently are 56Diagnostic Indicators. Many of the Diagnostics Indicators, especiallythose representing the Service Delivery domain, were validated in high-income settings but were recognized as requiring further work in orderto be adapted to LMICs. Similarly, the PHCPI team recognized that forsome areas, existing indicators poorly reflect the attribute of interest (eg,


Tab

le2.

Upd

ated

Vit

alSi

gns

bySu

bdom

ain

ofth

eP

HC

PI

Con

cept

ual

Fram

ewor

k,W

ith

Res

ults

ofth

ee-

Del

phi

Surv

ey(N

ovem

ber

2016

),an

dN

umbe

rof

Cou

ntri

esW

ith

Ava

ilab

leD

ata

Dom

ain

/Su

bd

omai

nIn

dic

ator

Ave

rage

Tot

alR

atin

g(O

ut

of24

)

Qu

alit

ativ

eR

ec-

omm

end

atio

n(-

1[D

rop

]to

1[K

eep

])

Nu

mb

erof

LM

ICs

Wit

hA

vail

able

Dat

a:C

urr

ent

(Exp

ecte

d)

A2.

Hea

lth

Fina

ncin

gP

erca

pita

PH

Cex

pend

itur

e18

.33

0.47

31(5

0)%

ofcu

rren

tgo

vern

men

the

alth

expe

ndit

ure

devo

ted

toP

HC

17.0

10.

3531

(50)

Gov

ernm

ent

PH

Cex

pend

itur

eas

%of

allP

HC

expe

ndit

ure

17.1

60.

2431

(50)

OO

PP

HC

expe

ndit

ure

as%

ofal

lPH

Cex

pend

itur

e17

.86

0.35

31(5

0)

B1.

Dru

gsan

dSu

ppli

esB

asic

equi

pmen

tav

aila

bili

ty19

.00

0.47

15

Ava

ilab

ilit

yof

esse

ntia

ldru

gs20

.72

0.65

12A

vail

abil

ity

ofva

ccin

es20

.81

0.47

8(1

0)Fa

cili

ties

wit

hcl

ean

wat

er,

elec

tric

ity,

sani

tati

on19

.75

0.24

7(9

) Con

tinu

ed


Tab

le2.

Con

tinu

ed

Dom

ain

/Su

bd

omai

nIn

dic

ator

Ave

rage

Tot

alR

atin

g(O

ut

of24

)

Qu

alit

ativ

eR

ec-

omm

end

atio

n(-

1[D

rop

]to

1[K

eep

])

Nu

mb

erof

LM

ICs

Wit

hA

vail

able

Dat

a:C

urr

ent

(Exp

ecte

d)

B2.

Faci

lity

Infr

astr

uctu

reH

ealt

hce

nter

and

heal

thpo

stde

nsit

y18

.73

0.35

76

B4.

Wor

kfor

ceC

omm

unit

yhe

alth

wor

ker,

nurs

e,an

dm

idw

ife

dens

ity

18.1

90.

4153

C3.

Acc

ess

Acc

ess

barr

iers

due

totr

eatm

ent

cost

s17

.71

0.53

56

Acc

ess

barr

iers

due

todi

stan

ce18

.06

0.47

56C

4.A

vail

abil

ity

ofE

ffec

tive

PH

CSe

rvic

es

Pro

vide

rab

senc

era

te17

.24

0.06

7(9

)D

iagn

osti

cac

cura

cy17

.06

0.18

6(8

)A

dher

ence

tocl

inic

algu

idel

ines

17.0

00.

297

(9)

Cas

eloa

dpe

rpr

ovid

er(d

aily

)15

.63

0.18

6(8

) Con

tinu

ed


Tab

le2.

Con

tinu

ed

Dom

ain

/Su

bd

omai

nIn

dic

ator

Ave

rage

Tot

alR

atin

g(O

ut

of24

)

Qu

alit

ativ

eR

ec-

omm

end

atio

n(-

1[D

rop

]to

1[K

eep

])

Nu

mb

erof

LM

ICs

Wit

hA

vail

able

Dat

a:C

urr

ent

(Exp

ecte

d)

C5.

Hig

h-Q

uali

tyP

HC

Dro

pout

rate

1st

to3r

dD

TP

3va

ccin

atio

n19

.06

0.24

132

Dro

pout

rate

1st

to4t

han

tena

talv

isit

18.6

00.

2411

9

Trea

tmen

tsu

cces

sra

tefo

rne

wT

Bca

ses

16.7

30.

1213

1

Car

e-se

ekin

gfo

rsy

mpt

oms

ofpn

eum

onia

15.9

30.

1212

8

D1.

Eff

ecti

veSe

rvic

eC

over

age

Dem

and

for

fam

ily

plan

ning

sati

sfie

dw

ith

mod

ern

met

hod

19.1

30.

4712

8

Ant

enat

alca

reco

vera

ge19

.66

0.29

121

Skil

led

birt

hat

tend

ance

19.2

60.

4712

8D

TP

3im

mun

izat

ion

cove

rage

21.9

60.

5913

2

Con

tinu

ed


Tab

le2.

Con

tinu

ed

Dom

ain

/Su

bd

omai

nIn

dic

ator

Ave

rage

Tot

alR

atin

g(O

ut

of24

)

Qu

alit

ativ

eR

ec-

omm

end

atio

n(-

1[D

rop

]to

1[K

eep

])

Nu

mb

erof

LM

ICs

Wit

hA

vail

able

Dat

a:C

urr

ent

(Exp

ecte

d)

Chi

ldre

nw

ith

diar

rhea

rece

ivin

gap

prop

riat

etr

eatm

ent

19.9

30.

3511

0

TB

case

sde

tect

edan

dcu

red

18.7

90.

2412

8P

eopl

eli

ving

wit

hH

IVre

ceiv

ing

AR

T18

.47

0.18

98

ITN

cove

rage

for

mal

aria

prev

enti

on17

.95

0.06

40

Cer

vica

lcan

cer

scre

enin

gra

te18

.53

0.18

�30

Hyp

erte

nsio

nco

ntro

l20

.14

0.47

�40

-50

Dia

bete

sm

elli

tus

cont

rol

19.9

80.

47�

40-5

0E

1.H

ealt

hSt

atus

Mat

erna

lmor

tali

tyra

tio

19.0

80.

2412

8A

dult

mor

tali

tyfr

omN

CD

s15

.87

0.18

119

Und

er-5

mor

tali

tyra

te19

.58

0.24

132

Neo

nata

lmor

tali

tyra

te19

.65

0.24

135

E3.

Equ

ity

Dif

fere

nce

betw

een

1st

and

5th

wea

lth

quin

tile

sfo

run

der-

5m

orta

lity

17.7

30.

1861

Abb

revi

atio

ns:P

HC

,pri

mar

yhe

alth

care

;OO

P,ou

t-of

-poc

ket;

DT

P3,

diph

ther

ia,t

etan

us,p

ertu

ssis

;AR

T,an

tire

trov

iral

ther

apy;

ITN

,ins

ecti

cide

-tre

ated

bed

net;

NC

D,n

onco

mm

unic

able

dise

ase.


Table 3. Diagnostic Indicators With Results From the e-Delphi Survey(July 2015)

Domain/Subdomain Indicator

AverageTotal Rating(Out of 25)

A2. HealthFinancing

Per capita PHC expenditure 19.7General government health expenditure as a

percent of total health expenditure17.8

Public sector tax revenue (percentage of grossdomestic product [GDP])

17.3

Out-of-pocket expenditures as percentage oftotal health expenditure

19.2

B2. FacilityInfrastructure

Health center density 18.9Total density per 100,000 population: health

posts19.2

Total density per 100,000 population:district + rural hospitals

19.7

B4. Workforce Community health worker (CHW) densityper 1,000 population

18.3

Physician density per 1,000 population 20Nursing and midwifery personnel density

per 1,000 population20.1

Total density (physicians + CHWs + nurses+ midwives) per 100,000 population

19.4

B5. Funds Provider has financing to renew andmaintain building/equipment (eg,maintenance and/or spare parts budget)

18.8

Percent of revenue from user’s charge 17.1Average cash amount for operation support

per facility17.8

Community attendance at managementmeetings

17.9

Health facilities providing supervision andsupport to community health workers

18.3

Regular management meetings 18.9

Continued


Table 3. Continued



Facility participates in national/facilityservice level accreditation/certificationprogram and is currently certified

18.8

Supportive management: formal training 18.6Supportive management: supervision 19Quality assurance processes 19.8Presence of client feedback system 19.7System for eliciting and reviewing client

opinion19.8

Average user’s charge per visit 18.7Prices (paid by patient) for key priority

services, such as maternal and childhealth services

19

Cost-related access: are there transportationcosts/barriers to your receiving care?

18

Cost-related access: did you not fill aprescription; skipped a recommendedmedical test, treatment, or follow-up; orhave a medical problem but did not visitthe doctor or clinic in the past yearbecause of cost?

19.3

Cost-related access: did you have seriousproblems paying for the visit, or wereunable to pay medical bills?

18.5

Timeliness: When the facility is open andyou get sick, would someone see you thesame day?

18.1

Timeliness: Is it very or somewhat difficultto get medical care in the evening,weekend, or on a holiday without goingto the emergency room?

18.2

Timeliness: Waiting time for being seen inemergency care need was 2 hours ormore

18.9

Continued


Table 3. Continued



C4. Availability ofEffective PHCServices

Management of maternal/neonatalcomplications

19.1

Treatment accuracy 17.9Provider burnout 15.3Standard precautions for infection

prevention and control18.2

C5. High-QualityPHC

First-contact access: Is it difficult foryou to get medical care at theprimary health care facility whenyou think you need it?

18.9

First-contact access: is it easy to getan appointment for a routineconcern?

16.1

First-contact access: When theprimary health care facility isclosed, is there a phone numberyou can call when you get sick?

18.5

First-contact access: When you havea new health problem, do you goto your regular primary health carefacility before going somewhereelse?

17.5

First-contact access: How far do youregularly travel to receive primarycare?

18.7

Relational continuity: When you goto your primary health carefacility, do you see the same healthcare provider each time?

16.3

Relational continuity: How confidentare you that your health careprovider at the primary health carefacility will look after you, nomatter what happens in the futureto your health?

13.5

Continued


Table 3. Continued



Informational continuity: At yourprimary health care facility, doesyour regular health care provideralways or often know importantinformation about your medicalhistory?

17.5

Informational continuity: At yourprimary health care facility, werethere times when the health careprovider you were seeing did nothave access to your most recenttest or exam results?

17.3

Informational continuity: At yourprimary health care facility, isthere one unique health recordthat follows you over time and is itaccessible when needed?

19.3

Management continuity: Thinkingabout all the persons you saw indifferent places, is there one personwho ensures follow-up of yourhealth care?

17.3

Management continuity: Is theperson who ensures your follow-upaware of health care you receivefrom others?

15.9

Management continuity: Is theperson who ensures your follow-upin contact with other providersabout your health care?

13.5

Formal system for referring patientsand/or accepting patients

18.3

Does your regular health careprovider know when you havevisited a specialist?

17.7

Continued


Table 3. Continued



Does your regular health careprovider help coordinate referralsto a specialist?

18.5

Does your regular health careprovider get a report from thespecialist about the visit?

18.4

Have you often or always felt thatyour care was well coordinatedamong different providers?

17.5

D1. EffectiveServiceCoverage

Tobacco use among adults 16.6Diabetes and raised blood glucose 18.5

E3. Equity Differential rate ratio of Q1-Q5maternal mortality ratio

18.7

provider caseload as a measure of provider motivation) and thus are areaswhere further research and development are needed.

Our in-person discussions about the Diagnostic Indicators validatedthe results of the scoping exercise and online survey. The partici-pants debated the merits of some of the proposed indicators but hadfew recommendations for including additional indicators not iden-tified in the initial review. They also discussed the appropriatenessof many of the proposed Diagnostic Indicators for low-income coun-tries. Even though the relevant core domains of PHC were con-sistent across settings, there was consensus among participants thatmany of the existing indicators do not adequately capture these do-mains in low-income settings (eg, percentage of births with a skilledbirth attendant as a measure of effective service coverage). The ex-perts also agreed that the Diagnostic Indicators should be capturedand reported at the subnational level in order to understand varia-tion in performance across countries and to support local improvementefforts.


Discussion

The results of the 2 modified Delphi processes confirmed that the indica-tors selected as Vital Signs and Diagnostic Indicators are strong existingmeasures that are validated, that are relevant to quality PHC, and forwhich many LMICs have data. The experts also agreed that these sets ofindicators, though likely composed of the best available indicators, arenot fully adequate for measuring the core components of quality PHCservice delivery in LMICs and that substantial future work is needed toimprove the indicators and increase data availability for these improvedindicators. Currently, several subdomains of the conceptual framework(A1, A2a, A3a, A3c, B3, C1a-c, C2a, C2c, D1g, E2, and E5) are notreflected in the Vital Signs and Diagnostic Indicators, owing to the lim-itations of validated and globally used measures. Furthermore, we needto consider issues salient to measuring quality PHC, such as governance,trust in the system, resilience, patient experience, communication andautonomy, pandemic preparedness, and mental health services. Morescoping is needed to determine whether we have adequate indicatorsfor these domains or whether further research and development of newindicators are needed.

The expert meeting participants agreed as well that the measurementand data collection strategies for all the proposed indicators shouldfocus on strengthening national and subnational data systems in orderto enable local leadership, governance, ownership, and use of data ina sustainable manner. Experts also agreed that the goal of any newmeasurement and data collection effort should be to generate not morebut better measures that reduce the use of irrelevant or not actionableindicators to improve performance.

Ongoing Work

Composite Indices. One priority area for ongoing methodological de-velopment is the creation of composite indices. The PHCPI initiallydeveloped 1 composite index as part of the original 25 Vital Signs, andwe are now working to develop more comprehensive indices for each of 4of the 5 domains of the conceptual framework: Inputs, Service Delivery,Outputs, and Outcomes. For the System domain, we will likely developseveral composites instead of a single composite, because the diver-sity of subdomain topics (governance and leadership, health financing,


adjustment to population health needs) makes aggregation into a singlemetric problematic.

The original composite Vital Sign was the service coverage index, whichcombined data from 3 Vital Signs in the Outputs domain: antenatalcare coverage (4 or more visits), percentage of children under age 5with diarrhea receiving oral rehydration therapy and continued feed-ing, and DTP3 (diphtheria-tetanus-pertussis) vaccine coverage. Weconstructed this as the mean scaled residual of the 3 indicators andrescaled to range from 0 to 1; we made no imputation or adjustment formissing data.

Our initial work on the new Outputs domain composite index seeksto incorporate additional primary health care services for a broader rangeof conditions relevant to a wider population, including family plan-ning, pneumonia, tuberculosis, HIV, and malaria. The objective is alsoto closely align the indicators and methodology with the WHO in-dex on the coverage of essential health services, which will be used tomonitor progress toward UHC. Our current methodological challengesinclude choosing an approach to fill missing data (using the most re-cent year of available data for a country, the regional average, or datafor a highly correlated indicator) and whether to construct the indexas the mean of all services or as a compilation of subgroup means (eg,reproductive, maternal, newborn, and child health, infectious diseasecare).

PHC Performance Dashboards. The PHCPI team is also currently de-veloping national-level performance dashboards of PHC systems. Theobjective is to produce a uniform set of visualizations for as many LMICsas possible that effectively communicate summary performance infor-mation about each country’s PHC system. Our first step in designingthe dashboard template was to review existing scorecards to generateideas for data visualizations and layouts. Using ideas from this reviewand initial decisions about what information to convey, we developed aprototype through an iterative process.

The prototype was initially developed for Uganda, which had the mostcomplete Vital Signs data availability of any country. We created variouslayouts and graphics to present contextual information (eg, population,gross domestic product per capita), each of the Vital Signs, and selecteddata over time to convey trends. Through several iterations, we generateddifferent variations of the dashboard, which we shared with the PHCPIteam, experts, and partners for feedback, which was integrated into


subsequent versions. We are currently amending the prototype to reflectrecent updates to the Vital Signs and methodological developments inthe production of the composite indices.

Limitations

The PHCPI’s work to date has important limitations, primarily arisingfrom the lack of available data for many performance indicators frommost LMICs, in addition to the absence of validated and reliable indi-cators for measuring key aspects of PHC, even when data are available.While the PHCPI has articulated a comprehensive ideal of the com-ponents of PHC performance assessment in the form of the conceptualframework, substantial advancements in developing and validating in-dicators and collecting data are required to populate the full scope ofthis framework with indicators, mainly for the multiple subdomainsthat currently have no indicators. In other domains, the optimal meansof collecting accurate data for relatively new indicators (eg, diagnos-tic accuracy, caseload per provider) are still being explored, leading tovariability in the types of data available across countries. While challeng-ing, these limitations are the primary factors informing and motivatingplanned activities to improve data availability and guide future research,which will address these gaps over time. There are additional limitationsarising from the methods that the PHCPI has used to develop its con-ceptual framework and indicator sets. These tools reflect findings fromthe literature review and the opinions of the PHCPI team and expertscommittee. Although we have attempted to incorporate a diversity ofsources and perspectives, additional or alternative input would likelylead to slight differences in the framework and indicators. The PHCPIwelcomes continued feedback on and refinement of its tools and seeksto improve them over time.

Improving Data Availability for Vital SignsGlobally

Even for the current set of Vital Signs and Diagnostic Indicators, dataavailability is highly variable: ranging from a low of 7 to 9 countries(indicators on provider competence, absenteeism, caseload) to a high of100 to 130 countries (coverage indicators, mortality rates). The reasonis that some aspects of performance are inherently difficult to measure


(notably, measuring the effectiveness and quality of care is difficult inany setting), but also that many LMICs lack reliable data-collectionsystems and tools.

In discussion with the international experts’ advisory committee, thePHCPI team identified 4 major strategies for filling these informationgaps:

� Influencing internationally comparable sources of information inglobal initiatives such as the Health Data Collaborative and otherefforts to standardize health information systems and measure-ment approaches (eg, System of Health Accounts, SHA 2011)

� Working with and helping to strengthen countries’ routinehealth information systems to generate comparable performanceinformation

� Identifying similar (acceptable substitute) indicators from routineinformation systems to feed into country-level PHC dashboards

� Imputing data when information is missing

Other organizations and initiatives provide useful lessons for maxi-mizing the benefits of a combination of these strategies. For example,the Organisation for Economic Co-operation and Development (OECD)has worked extensively with member countries to develop compara-ble indicators, by working with national statistical institutes to adoptstandardized measures. This process combines 2 approaches: bottom-upwhen more than 20% of countries already have an indicator for a domainand top-down when fewer than 20% of countries already have definedindicators. WHO’s regional offices have also used a bottom-up process,similar to the OECD’s, to develop and validate indicators on qualityof care with countries. This requires time and commitment to encour-age countries to develop ways of measuring and to build capacity andsystems to collect data.

Some experts expressed concern that countries might see the PHCPI’sefforts as measurement for judgment, rather than measurement for im-provement. For this reason, a bottom-up approach for developing newindicators that emphasizes the data a country needs to improve perfor-mance rather than the data a country needs to be globally compara-ble may be preferable for the PHCPI. Because the PHCPI works withcountries, trust also is critical, particularly when considering ways tohandle missing data and observing the limitations of using imputation


methods in order to understand performance. Preserving the integrityand credibility of the indicators published and the conclusions based onindicators remains the initiative’s top priority. The PHCPI is commit-ted to involving countries in the production of indicators and compositeindices before making any public release.

Agenda for Future Research

As noted earlier, there was a general consensus within the expert advisorycommittee that key aspects of quality PHC currently lack validated,broadly used indicators and that existing measures for coordination ofcare, first-contact access, continuity of care, and comprehensiveness ofcare must be refined so as to be more applicable to LMIC settings. We alsoneed to increase the measurement of some validated indicators that arenot currently widely collected (eg, provider competence). Developingnew indicators and adapting existing indicators used in high-incomesettings to fill these gaps are the priorities of the PHCPI’s research anddevelopment agenda. Using facility and household surveys, research anddevelopment activities are currently under way in Ghana for indicatorsof organizational and managerial performance, as well as for experientialquality of care.

Additional topics requiring research and development of new indica-tors include

� Patient trust in the system and provider� Provider motivation� Timeliness of health information systems� Avoidable hospitalizations� Avoidable mortality amenable to primary health care

interventions� Palliative care coverage� Out-of-pocket expenses and financial protection� User-reported access barriers� Organization and management, including referral systems, infor-

mation systems, and performance incentive systems� Mental health care coverage� Informal health workforce� Governance at the facility level� Community engagement


To develop new and improved indicators for aspects of quality PHCthat have been neglected by existing measures, the PHCPI is partner-ing with academic researchers, survey programs, and other initiatives.WHO is currently selecting indicators for monitoring integrated people-centered health services (IPCHS), which will cover several of the topicsmentioned earlier, for example, patient experience, referral systems, andcommunity engagement. The PHCPI and IPCHS indicators will bealigned when appropriate. The PHCPI is also a member of the HealthData Collaborative, which strives to improve the coordination of datacollection activities across global health agencies and donors, such asthrough the creation of harmonized survey instruments for householdand facility surveys. Through these partnerships, the PHCPI and oth-ers are developing and testing new indicators in household and facilitysurvey tools with the goal of identifying improved indicators of PHCperformance and integrating these indicators into the global measure-ment agenda.

Designing and testing survey questions and then evaluating the valid-ity and reliability of indicators generated from survey data, particularlyfor LMICs’ many health systems, takes a long time. As a result, themultiyear PHCPI research and development agenda must be harmo-nized with efforts to increase the availability of data for establishedPHC performance indicators and to enable health information systemsto incorporate data collection for any additional measures.

Operationalizing the Measurement Agenda toImprove Performance

The applicability of the PHCPI’s approach to assessing performanceis most meaningfully tested and refined by engaging with countriesworking to improve their PHC systems. The PHCPI is engaging withthese trailblazer countries to help them apply and strengthen perfor-mance measurement and improvement instruments by linking themwith other initiatives in the PHCPI’s partner institutions, includingthe Global Financing Facility (GFF) in support of Every Woman, Ev-ery Child; the Joint Learning Network (JLN); the World Bank ServiceDelivery Indicators; and countries’ results-based financing initiatives.An example of country engagement was using the PHCPI’s conceptualframework and Vital Signs to diagnose PHC system performance bottle-necks in Cameroon, as part of the development of the GFF investment


case. This engagement resulted in a better understanding of service de-livery bottlenecks and changes in the prioritization of interventions asproposed by the government of Cameroon and its partners. Throughalignment with these and other ongoing activities in countries, PHCperformance indicators can provide valuable insights for partners andsimultaneously generate new data and feedback on tools for the PHCPI.The World Bank will seek to engage with a minimum of 14 countriesthrough its investment operations from 2017 to 2019.

The PHCPI has identified 5 key mechanisms for country engagement.First, in partnership with other members of the initiative, the PHCPIcan support the development of new partner investments and engage-ments, including World Bank operations and GFF investment cases.Through its national PHC performance dashboards and other diagnos-tic instruments, countries can assess their PHC systems’ strengths andweaknesses in order to set policy and investment priorities. Second, thePHCPI is useful for supporting the implementation of new investments.The assessment activities help to identify implementation bottlenecksand improve capacity for strategically using performance information.Third, data gaps identified by the PHCPI can guide investments ininformation systems. Key indicators for which countries have no data orno recent data highlight areas where new or improved measurement ismost needed. Fourth, through its global network, the PHCPI facilitatesknowledge sharing and mutual learning on innovative service deliverymodels and the use of performance measurement for improvement. ThePHCPI is already supporting these activities through the JLN PHCMeasurement for Improvement Collaborative and establishing a com-munity of practice in the World Bank. Finally, through engagement overtime, the PHCPI will develop expertise to guide innovative financingapproaches for strengthening PHC systems. As the PHCPI’s work incountries expands, the range of experiences can inspire and inform newways of financing and incentivizing PHC performance improvement.

Conclusions

Over the past 2 years, the PHCPI has been developing a suite ofperformance assessment tools to provide in-depth and comprehensiveunderstanding of PHC systems in low- and middle-income countries.The foundation of the PHCPI’s approach to performance assessment


is the conceptual framework, which is unique among health systemassessment frameworks for its emphasis on service delivery and qualityof care. Two sets of performance indicators, selected with substantialinput from a diverse and global group of experts, provide the backbonefor measuring system performance in alignment with the conceptualframework. Additional instruments are still being developed, such ascountry-level performance dashboards, a national and subnational perfor-mance assessment tool, and performance improvement pathways. Thesewill improve countries’ abilities to design and implement an assessmentof PHC system performance; communicate national and internationalfindings with diverse audiences; evaluate changes in performance overtime; and identify and select appropriate policy reforms for performanceimprovement.

The PHCPI’s main challenges are the lack of available data for manyindicators, particularly for measuring the PHC’s core functions, and alack of validated indicators for other critical dimensions of quality PHC.The PHCPI is working with partner organizations and other initiativesto develop and test new indicators and improve data collection systemsto generate better PHC performance information.

Through substantial country engagement and continual expert over-sight, the PHCPI will continue to improve each of its performanceassessment tools; develop new and improved indicators; and increase theavailability of data, particularly for aspects of PHC that are not wellunderstood in LMICs. Each of these activities catalyzes the dissemina-tion of new knowledge and best practices for measuring and improvingPHC system performance, supporting the achievement of UHC, andimproving quality of care globally.

References

1. Pettigrew LM, Maeseneer JD, Anderson M-IP, Essuman A, KiddMR, Haines A. Primary health care and the sustainable develop-ment goals. Lancet. 2015;386(10009):2119-2121.

2. Coarasa J, Das J, Gummerson E, Bitton A. A systematic tale of twodiffering reviews: evaluating the evidence on public and privatesector quality of primary care in low- and middle-income countries.Global Health. 2017;13:24.

3. Rohde J, Cousens S, Chopra M, et al. 30 years after Alma-Ata: has primary health care worked in countries? Lancet.2008;372(9642):950-961.


4. Bitton A, Ratcliffe HL, Veillard JH, et al. Primary health care as afoundation for strengthening health systems in low- and middle-income countries. J Gen Intern Med. 2017;32(5):566-571.

5. Primary Health Care Performance Initiative website. 2017.http://phcperformanceinitiative.org. Accessed May 5, 2017.

6. World Health Organization. The World Health Report 2008—Primary Health Care (Now More Than Ever). Geneva, Swi-itzerland: World Health Organization; 2008. http://www.who.int/whr/2008/en/. Accessed September 12, 2017.

7. Muldoon LK, Hogg WE, Levitt M. Primary care (PC) and primaryhealth care (PHC). What is the difference? Can J Public Health Rev.2006;97(5):409-411.

8. World Health Organization. Everybody’s Business: StrengtheningHealth Systems to Improve Health Outcomes: WHO’s Framework forAction. 2007. Geneva, Switzerland: World Health Organization;2007.

9. Murray CJL, Frenk J. A framework for assessing the performanceof health systems. Bull World Health Organ. 2000;78(6):717-731.

10. Chaudhury N, Hammer J, Kremer M, Muralidharan K, RogersFH. Missing in action: teacher and health worker absence in de-veloping countries. J Econ Perspect. 2006;20(1):91-116.

11. Das J, Hammer J, Leonard K. The quality of medical advice inlow-income countries. J Econ Perspect. 2008;22(2):93-114.

12. Peabody JW, Taguiwalo MM, Robalino DA, Frenk J. Improv-ing the quality of care in developing countries. In: Jamison DT,Breman JG, Measham AR, et al., eds. Disease Control Priorities inDeveloping Countries. 2nd ed. Washington, DC: World Bank; 2006.http://www.ncbi.nlm.nih.gov/books/NBK11790/. Accessed May5, 2017.

13. Hsu CC, Sandford, BA. The Delphi technique: making sense ofconsensus. Pract Assess Res Eval. 2007;12(10).

14. Hogan D, Hosseinpoor AR, Boerma T. Developing an Index for theCoverage of Essential Health Services. Geneva, Switzerland: WorldHealth Organization, Department of Evidence, Information, andResearch; May 2016.

15. United Nations Department of Economic and Social Affairs,Statistics Division. SDG indicators—revised list of global sus-tainable development goal indicators. https://unstats.un.org/sdgs/indicators/indicators-list/. Published 2017. Accessed May 5, 2017.

16. Global Financing Facility. Consultation paper in advance of thethird GFF Investors Group meeting: results measurement. 2016.

17. World Health Organization. Global Reference List of 100 CoreHealth Indicators, 2015: Metadata. Geneva, Switzerland: World

http://phcperformanceinitiative.org

http://www.who.int/whr/2008/en/

http://www.who.int/whr/2008/en/

http://www.ncbi.nlm.nih.gov/books/NBK11790/

https://unstats.un.org/sdgs/indicators/indicators-list/

https://unstats.un.org/sdgs/indicators/indicators-list/


Health Organization; 2015. http://www.who.int/healthinfo/indicators/2015/metadata/en/. Accessed May 5, 2017.

18. WHO EMRO Working Group on Quality of Care in PHC. Listof core indicators of quality in primary care. 2016.

19. Service delivery indicators (SDI). World Bank Group website.http://datatopics.worldbank.org/sdi/. Accessed May 6, 2017.

20. Joint Learning Network. Primary Health Care for Improve-ment Collaborative website. http://www.jointlearningnetwork.org/primary-health-care-measurement. Accessed May 5, 2017.

21. Starfield B. Is primary care essential? Lancet. 1994;344:1129-1133.22. Donabedian A. The quality of care: how can it be assessed? JAMA.

1988;260(12):1743-1748.23. Mello LA, Tonini T, Sousa da Silva A, Dutt-Ross S, de Souza

Velasque L. Evaluation of primary health care units in the Rio deJaneiro city according to the results of PMAQ 2012. J Ambul CareManage. 2017;40(Suppl. 2):S71-S82.

24. Arah OA, Westert GP, Hurst J, Klazinga NS. A conceptual frame-work for the OECD health care quality indicators project. Int J QualHealth Care. 2006;18(Suppl. 1):5-13.

25. Arah OA, Klazinga NS, Delnoij DMJ, Asbroek AHAT, CustersT. Conceptual frameworks for health systems performance: a questfor effectiveness, quality, and improvement. Int J Qual Health Care.2003;15(5):377-398.

26. Shi L, Starfield B, Xu J. Validating the adult primary care assess-ment tool. J Fam Pract. 2001;50(2):161.

27. Kringos DS, Boerma WG, Bourgueil Y, et al. The European pri-mary care monitor: structure, process and outcome indicators.BMC Fam Pract. 2010;11:81.

28. Harzheim E, Starfield B, Rajmil L, Alvarez-Dardet C, Stein AT.Internal consistency and reliability of primary care assessmenttool (PCATool-Brasil) for child health services. Cad Saude Publica.2006;22(8):1649-1659.

29. Pongpirul K, Starfield B, Srivanichakorn S, Pannarunothai S. Pol-icy characteristics facilitating primary health care in Thailand: apilot study in transitional country. Int J Equity Health. 2009;8:8.

30. WHO framework on integrated people-centred health services.World Health Organization website. 2017. http://www.who.int/servicedeliverysafety/areas/people-centred-care/en/. AccessedMay 18, 2017.

31. W.K. Kellogg Foundation Logic Model Development Guide.Detroit, MI: WKKF; 2004. https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide. Accessed May 6, 2017.

http://www.who.int/healthinfo/indicators/2015/metadata/en/

http://www.who.int/healthinfo/indicators/2015/metadata/en/

http://datatopics.worldbank.org/sdi/

http://www.jointlearningnetwork.org/primary-health-care-measurement

http://www.jointlearningnetwork.org/primary-health-care-measurement

http://www.who.int/servicedeliverysafety/areas/people-centred-care/en/

http://www.who.int/servicedeliverysafety/areas/people-centred-care/en/

https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide




32. Roberts MJ, Hsiao WC, Berman P. Getting Health Reform Right: AGuide to Improving Performance and Equity. New York, NY: OxfordUniversity Press; 2008.

33. Maeseneer JD, Moosa S, Pongsupap Y, Kaufman A. Primary healthcare in a changing world. Br J Gen Pract. 2008;58(556):806-809.

34. De Maeseneer J, Willems S, De Sutter A, Van de Geuchte I,Billings M. Primary Health Care as a Strategy for Achieving EquitableCare: A Literature Review Commissioned by the Health SystemsKnowledge Network. Geneva, Switzerland: World Health Organi-zation; March 2007. http://www.who.int/social_determinants/resources/csdh_media/primary_health_care_2007_en.pdf. Ac-cessed September 22, 2017.

35. Engel GL. The need for a new medical model: a challenge forbiomedicine. Holist Med. 1989;4(1):37-53.

Funding/Support: Funding for the Primary Health Care Performance Initiativeis provided by the Bill & Melinda Gates Foundation.

Conflict of Interest Disclosures: All authors have completed and submitted theICMJE Form for Disclosure of Potential Conflicts of Interest. No conflicts werereported.

Acknowledgments: We acknowledge current and past members of the PHCPIsteering committee and working group for all of their efforts in moving thework forward, as well as the many stakeholders consulted in the developmentof the initiative so far. We also thank Daniel Fox for his guidance and support.

Address correspondence to: Jeremy Veillard, The World Bank Group, 1776 G StreetNW, Washington, DC 20433 (email: [email protected]).

Appendix 1

International Experts Advisory CommitteeMembers

Participants at International Experts AdvisoryCommittee Meeting, July 2015

Name InstitutionSelected RelevantAreas of Expertise

Irene Agyepong Ghana Health Service Health systems, healthpolicy

Continued

http://www.who.int/social_determinants/resources/csdh_media/primary_health_care_2007_en.pdf

http://www.who.int/social_determinants/resources/csdh_media/primary_health_care_2007_en.pdf


Appendix. Continued


Sara Bennett Johns HopkinsBloomberg School ofPublic Health

Health systemsresearch, healthgovernance andpolitics

Peter Berman Harvard T.H. ChanSchool of PublicHealth

Health economics andhealth financing,health systemperformance

Ian Forde OECD Quality of careEmmanuela

GakidouInstitute for Health

Metrics andEvaluation

Measurement,evaluation, healthinequalities

Jeannie Haggerty McGill University Primary health care,quality of care

Niek Klazinga OECD, University ofAmsterdam

Quality of care, healthservices research

Margaret Kruk Harvard T.H. ChanSchool of PublicHealth

Quality of care, healthsystem performance

HernanMontenegro

World HealthOrganization

Health systems,integrated servicedelivery

Henry Perry Johns HopkinsBloomberg School ofPublic Health

Primary health care,community health

Cristian HerreraRiquelme

Ministry of Health,Chile

Health systems,primary health care,politics

Leiyu Shi Johns HopkinsPrimary Care PolicyCenter

Primary health care,health inequalities

Stephane Verguet Harvard T.H. ChanSchool of PublicHealth

Health decisionscience, healtheconomics


Participants at International Experts AdvisoryCommittee Meeting, November 2016




Lola Dare Centre for HealthSciences Training,Research andDevelopment

Community medicine,epidemiology

Jean Paul Dossou Institute of TropicalMedicine, Antwerp,Belgium, Centre deRecherche enReproductionHumaine et enDemographie,Cotonou, Benin

Integrated servicedelivery, infectiousdiseases,epidemiology

Joseph Dieleman Institute for HealthMetrics andEvaluation

Health financing,measurement,economics

Fadi El-Jardali Faculty of HealthSciences; AmericanUniversity of Beirut;WHO CollaboratingCenter forEvidence-InformedPolicy and Practice

Health policy, healthsystems

Ian Forde OECD Quality of careBasile Keugoung COP Service Delivery,

Yaounde, CameroonIntegrated service

delivery, infectiousdiseases, healthsystems



Continued


Appendix. Continued


Sheila Leatherman(via Webex)

UNC Gillings Schoolof Global PublicHealth, Departmentof Health Policy andManagement

Quality of care, healthsystems

Mondher Letaief Patient Safety andQuality Programme,World HealthOrganization;University Hospitalof Monastir

Health services, qualityof care

KamaliahMohamad

Family HealthDevelopmentDivision, Ministryof Health, Malaysia

Primary health care

Mercy Mwangangi Ministry of Health,Kenya

Quality of care,measurement andevaluation

Henry Perry John Hopkins Schoolof Public Health




Health systems,primary health care,politics

Stephane Verguet Harvard University Health decisionscience, healtheconomics

Respondents to Online Survey (DiagnosticIndicators), July 2015


Irene Agyepong Ghana Health Service Health systems, healthpolicy

Continued


Appendix. Continued


Janet Angbazo Primary Health CareDevelopmentAgency, Nigeria

Primary health care

Abul Azad Ministry of Health andFamily Welfare,Bangladesh

Health informationsystems, healthpolicy

Sara Bennett Johns HopkinsBloomberg School ofPublic Health

Health systems, healthgovernance andpolitics



Abel Bicao IDRC Primary health careIan Forde OECD Quality of careEmmanuela Gakidou Institute for Health



Niek Klazinga OECD, University ofAmsterdam

Quality of care, healthservices research

Dionne Kringos European Primary CareMonitor

Health systemperformance,primary health care



RamananLakshminarayanan

PMNCH Health economics,infectious diseases

Muhammed Lecky Health ReformFoundation ofNigeria

Health systems,demography

Jean FredericLevesque

New South WalesBureau of HealthInformation,Australia

Health systemperformance

Rosina Lipyoga Ministry of Health andSocial Welfare,Tanzania

Primary health care

Continued


Appendix. Continued


Isabel Maina Ministry of Health,Kenya

Monitoring andevaluation, healthinformation systems

Rashad Massoud University ResearchCorporation

Quality of care

HernanMontenegro

World HealthOrganization

Health systems

FlorenceMuli-Muliisme

Daystar University,Kenya

Health services

Andrea Nove Integrare Human resources forhealth, monitoringand evaluation



Health systems

Leiyu Shi Johns Hopkins PrimaryCare Policy Center


NanaTwum-Danso

MAZA, Ghana Quality of care, healthpolicy

Respondents to Online Survey (Revised VitalSigns), November 2016




Lola Dare Centre for HealthSciences Training,Research andDevelopment

Community medicine,epidemiology, civilsociety

Jishnu Das The World Bank Group Quality of careJoseph Dieleman Institute for Health


Health financing,economics

Continued


Appendix. Continued


Jean Paul Dossou Institute of TropicalMedicine, Antwerp,Belgium; Centre deRecherche enReproductionHumaine et enDemographie,Cotonou, Benin

Infectious diseases,epidemiology

Ian Forde OECD Quality of careEmmanuela

GakidouInstitute for Health



Niek Klazinga OECD Quality of care, healthservices research



Sheila Leatherman UNC Gillings Schoolof Global PublicHealth, Departmentof Health Policy andManagement

Quality of care, healthsystems

Mondher Letaief Patient Safety andQuality Programme,World HealthOrganization;University Hospitalof Monastir

Health services, qualityof care

Jean-FredericLevesque

Centre for PrimaryHealth Care andEquity, Universityof New South Wales

Health systemperformance

KamaliahMohamad

Family HealthDevelopmentDivision, Ministryof Health, Malaysia

Primary health care

Continued


Appendix. Continued


Henry Perry John Hopkins Schoolof Public Health




Health systems

Leiyu Shi Johns HopkinsPrimary Care PolicyCenter


Stephane Verguet Harvard University Health decisionscience, healtheconomics

Appendix 2

Health System Frameworks and ScorecardsReviewed

Afghanistan Health SectorBalanced Scorecard*

African Leaders Malaria Alliance(ALMA) Scorecard*

Balanced ScorecardBaldrige FrameworkCGD Commitment to

Development IndexCommonwealth Fund Health

Systems ScorecardsControl Knobs FrameworkCountdown to 2015*Dartmouth AtlasDemographic and Health Surveys

(DHS)*

Doing Business (and BizCLIR)Every Woman Every ChildHealthy Partnerships InitiativeHunger and Nutrition

Commitment IndexIbrahim Index of African

GovernanceInternational Budget PartnershipInstitute for Health Improvement

Triple AimKellogg Foundation Logic ModelLearning from Effective

Ambulatory Practices (LEAP)Project

Learning Metrics Task Force(LMTF)

Continued


Appendix. Continued

Living Standards MeasurementStudy (LSMS)

Medicare Hospital ReadmissionsReduction Program

Millennium Development Goals(MDGs)*

Multi Indicator Cluster Survey(MICS)*

NHS Star Rating SystemOpen Health InitiativeOECD Health Care Quality

Indicators Project FrameworkPrimary Care Assessment Tools

(PCAT)*PHC Activity Monitor for Europe

(PHAMEU)Programme for International

Student Assessment (PISA)PHC Activity Monitor for Europe

(PHAMEU)Partnership for Maternal,

Newborn & Child Health(PMNCH)

RMNCH Country Scorecards*

RWJ/TARSC Primary CarePractice Case Studies

Service Availability and ReadinessAssessment (SARA)

Service Delivery Indicators (SDI)Service Provision Assessment

(SPA)Starfield’s Characteristics of

Primary Health CareSystems Approach for Better

Education Results (SABER)Trends in International

Mathematics and Science Study(TIMSS)

UHC Monitoring FrameworkUNICAT Readiness AssessmentUSAID Measuring Results of

Health Sector Reform forSystem Performance

WHO Global Strategy onIntegrated People-CenteredHealth Services (IPCHS)

WHO Health System BuildingBlocks

*Frameworks that were explicitly used in the development of the PHCPI Conceptual Framework.


Ap

pen

dix

3

PH

CP

IC

once

ptua

lFra

mew

ork,

Ver

sion

1,Se

ptem

ber

2015


Appendix 4

Original 25 Vital Signs

Subdomain Indicator

A2. Health Financing Per capita PHC expenditurePercent of government health spending

dedicated to PHCB1. Drugs & Supplies Basic equipment availability

Availability of essential drugsAvailability of vaccines

B2. FacilityInfrastructure

Health center and health post densityper 100,000 population

B4. Workforce Community health worker, nurse andmidwife density, per 1,000 population

C1. Access Access barriers due to treatment costsC2. Availability of

Effective PHCServices

Provider absence rateDiagnostic accuracyCaseload per provider (daily)

C3. People-centeredCare

Dropout rate between 1st and 3rd DTPvaccination

Dropout rate between 1st and 4thantenatal (ANC) visits

Treatment success rate for new TB casesD1. Effective Service

CoverageCoverage IndexDTP3 immunization coverageAntenatal care coverage (4+ visits)Contraceptive prevalence rate (modern

methods)Percent of births taking place in a health

care facilityPercent of children under 5 with

diarrhea receiving oral rehydrationand continued feeding

E1. Health Status Maternal mortality ratioUnder-5 mortality rateAdult mortality for noncommunicable

diseasesE3. Equity Under-5 mortality: difference between

1st and 5th wealth quintilesE4. Efficiency Under-5 mortality relative to per capita

PHC expenditure


Supplementary Material

Additional supporting information may be found in the online ver-sion of this article at http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1468-0009:

PHCPI Vital Signs and Diagnostic Indicators

better measurement for performance improvement in low‐ and

Documents