. . - .

INFORMATION AND TECHNOLOGY TO ENABLE YOUR BUSINESS

I T @ W B S T A F F W O R K I N G P A P E R S

Impact Assessment Study of Computerized Services Delivery Projects from India and Chile

Primary author:

Subhash Bhatnagar Professor, Indian Institute of Management, Ahmedabad

Contributors to report:

Jiro Tominaga Senior lnformation Officer, ISG E-Government Practice

Shirin Madon lnformation Systems & Innovations Group, Dept. of Management, London School of Economics

Deepak Bhatia Manager, ISG E-Government Practice

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

The findings, interpretations, and conclusions expressed in this book are entirely those of the authors and should not be attributed in any manner to the World Bank, to its affiliated organizations, or to members of its Board of Executive Directors or the countries they represent. The World Bank does not guarantee the accuracy of the data included in this publication and accepts no responsibility for any consequence of their use. The boundaries, colors, denominations, and other information shown on any map in this volume do not imply on the part of the World Bank Group any judgment on the legal status of any territory or tho nnrlnrrnmnnt n r =rrnnt=nrn nf cl orh hnnonrl=r;nr

IMPACT ASSESSMENT STUDY

OF

COMPUTERIZED SERVICE DELIVERY PRO JEClTS

FROM INDIA AND CHILE

June 2007

Study Coordinator:

Task Managers:

Study Team:

Indian Institute of Management, Ahmedabad (LIMA)

London School of Economics

Partners:

Subhash Bhatnagar, Adjunct Professor, Indian Institute of Management, Ahmedabad (IIMA)

Deepak Bhatia, Manager, e-Government Practice, Information Solutions Group (ISGEG) Jiro Tominaga, Senior Information Officer, ISGEG

T. P. Rama Rao, Professor Nupur Singh, In-charge, Center for Electronic Governance Ranjan Vaidya, Research Associate, Center for Electronic Governance Mousumi Mandal, Research Assistant, Center for Electronic Governance

Shirrn Madon, Senior Lecturer, Information Systems and Innovation Group, Department of Management Matthew Smith, Doctoral Student, Information Systems and Innovation Group, Department of Management

The World Bank: ISGEG IIM A Ministry of Communications and Information Technology, Government of India

ABBREVIATIONS AND ACRONYMS

AMC AP

B2C CARD

C-DAC

DFID eGEP EU

G2C ICT

r n A IT KAVERI

LSE

MIS OECD PRM

RTC SII

UN UNCTAD

LTNDP UNPAN

Ahmedabad Municipal Corporation Andhra Pradesh

Business to citizen Computer Aided Administration of Registration Department

Centre for Development of Advanced Computing

Department for International Development

eGovernment Economics Project European Union Government to citizen Information and Communications Technologies

Indian Institute of Management, Ahmedabad

Information Technology Karnataka Valuation and e-Registration

London School of Economics and Political Science Management Information Systems

Organization for Economic Co-operation and Development Performance Reference Model Records of Rights, Tenancy and Crop Inspection Register

Servicio de Impuestos Internos

United Nations United Nations Conference on Trade and Development

United Nations Development Programme United Nations Online Network in Public Administration and Finance

This research is the product of a collaborative effort on the part of the faculty of the Indian Institute of Management, Ahmedabad (IIMA) and the London School of Economics, as well as the staff of the Information Solutions Group of the World Bank. It benefited greatly from a series of consultations with internal and external experts. A number of versions of the concept note for the study were circulated to solicit input and comments. The authors would like to extend special thanks to: Bruno Laporte; David Wheeler; Pierre Guislain; Philippe Dongier, Christine Qiang, Randeep Sudan, and Isabel Neto; Mark Dutz; Markus Goldstein and Sumeet Bhatiti; Soren Gigler; Gregory Kisunko; Michael Kubzansky; Jens Kristensen; and Edwin Lau (OECD e-Government Project) for their valuable suggestions.

This study would not have been possible without the support of the management of Information Solutions Group, in particular, Rakesh Asthma, senior manager with ISGEC. Deepak Bhatia, in addition to task management, played a key role in harnessing support for the study from different units in the Bank.

The entire project team from IIMA, London School of Economics, and World Bank contributed to the conceptualization of the study and development of the assessment framework that has been proposed in the report. The team based at IIMA designed the survey instruments and sampling methodology, and carried out the impact assessment for five projects in India. The survey of clients in India was done by TNS India, one of the large market research agencies in the country. The survey in Chile was conducted by Matthew Smith under the guidance of Shirin Madon.

This report is primarily authored by Subhash Bhatnagar. Nupur Singh and Ranjan Vaidya provided the data analysis support and generated all the tables presented in the report. Shirin Madon did a separate paper on literature search, parts of which have been used in the section that discusses efforts to assess public sector projects. Jiro Tominaga and Deepak Bhatia worked closely with Subhash Bhatnagar from the initial concept stage and made significant contributions in developing the structure and content of the report.

The research team would like to acknowledge the help provided in data collection from the agencies and users by all the staff of the implementing agencies of the projects that were assessed. In particular, the team is grateful to the following persons: Mr. Rajeev Chawla, Secretary (e-Governance), Department of Personnel and Administrative Reforms, Government of Karnataka (GOK); Mr. Gurumurthy, Deputy Tahsildar, Bhoomi Monitoring Cell, GOK; Mr. S. M. Bhavikatti, Assistant Inspector General of Registration (Computer), GOK; Mr. V. N. Thippanagouda, District Registrar and Deputy Commissioner of Stamps, Bangalore (Urban); Mr. P. Raghuveer, Additional Secretary, Government of Andhra Pradesh (GOAP); Mr. A. Raveendranath, Assistant Inspector General (CARD), Department of Registration and Stamps, GOAP; Dr. N. Sailaja, Special Officer, Department of IT and Communications, GOAP; Mr. Bireshwar Das, Deputy Director (Technical), eSeva; Mr. Anil Mukim, Commissioner, Ahmedabad Municipal

Corporation (AMC); Capt. Dilip Kumar Mahajan, Deputy Municipal Commissioner (Finance and e-Governance), AMC; and Mr. A. P. Patel, EDP Officer, Computer Department, AMC.

The team would also like to thank: Alejandra Barraza and Mathias Klingenberg, both working with the Directorate of Public Procurement Contracting; Fernando Barraza and Victor Hugo Lopez at Servicio de Impuestos Internos (SII); Viktor Aguila, Mabel Verdi, and Juan Enrique Hinostroza from the Universidad de la Frontera in Temuco; and Carolina Seward.

In extended consultations, a number of versions of the concept note for the study were shared with relevant World Bank staff members since November 2005. The concept note was also reviewed at the Concept Review Meeting held on January 23,2006.

Originally, the study was to cover five projects in India, two in Chile and five projects from other countries where assistance from the World Bank field staff might be sought. Potential projects were identified on the basis of publicly available evaluation data and documents. A number of World Bank staffers were contacted to explore if additional data (client surveys) could be collected. However, the effort required appeared at that point to be more than what could be handled. The study team, therefore, decided to limit the scope of this study to seven projects in India and Chile, leaving more comprehensive scope for future studies.

The field work was conducted from April 2006 to June 2006. In May 2006, interim results from the study were discussed with Bank staff that had provided input on the concept note in meetings held in Washington, D.C. On May 14, 2006, the interim findings were presented in a public forum organized by Information Solutions Group.

Although with limited scope, this study contains analyses of diverse computerization projects implemented by public sector organizations to improve service delivery in two developing countries. The study demonstrates that it is possible to estimate direct and indirect benefits to clients with reasonable accuracy using the proposed framework. Our analyses are based on in-depth surveys of clients (citizens and businesses that use the systems) and other stakeholders involved in these projects. While there is room for further refinement, the study makes a modest contribution in understanding the potential and limitation of the power of information and communications technologies (ICT) in improving delivery of services by public sector organizations. The results were both encouraging and sobering. The potential of ICT to help reduce cost of accessing services by clients, improve convenience, further accountability, and promote good governance in a broad sense is difficult to ignore. However, for those of us in the field of development assistance, the study also guides us to look beyond the hype of "e-government." Lack of effective project design, inadequate infrastructure, and a lack of willingness to reengineer processes can result in wasted opportunities, with the computerized systems showing hardly any improvements over the manual systems that are replaced. Familiar principles of project design and project management continue to be important for generating results.

The authors would like to extend sincere gratitude to all those who supported the completion of this study. It is our hope that this study becomes a starting point for many new studies to come.

STUDY TEAM ............................................................................................................................................... I1 ................................................................................................... ABBREVIATIONS AND ACRONYMS I11

ACKNOWLEDGMENTS ........................................................................................................................... IV PREFACE ............................................................................................................................................... VI EXECUTIVE SUMMARY ........................................................................................................................... 1 INTRODUCTION AND CONTEXT ............................................................................................................ 5

................................................ CHAPTER 1: ASSESSMENT OF PUBLIC SECTOR ICT PROJECTS 8 CHAPTER 2: MEASUREMENT FRAMEWORK AND DATA ANALYSIS ....................................... 12

.................................................................................. MEASUREME~T F'RAMEWORK AND IV~ETHODOLOGY 12 ANALYSIS OF SURVEY DATA .................................................................................................................. 16

................................................................................................................. 1 . Pro$le of respondents 16 ................................................................................................ 2 . Dimensions of impacts on clients 16

.................................................................................... 3 . Analysis of data collected from agencies 17 CHAPTER 3: ASSESSMENT RESULTS FOR INDIAN AND CHILEAN PROJECTS ..................... 18

..................................................... A COMPARATIVE ANALYSIS OF PROJECTS FROM CLIENT PERSPECTIVE 18 1 . Cost to users .............................................................................................................................. 18

.............................................................................................. 2 . User perception of service quality 20 .......................................................................... 3 . User perception of governance and corruption 22

4 . Comparison of fve Indian projects on overall client impact ...................................................... 24 .................................. 5 . Variability of client impact across different delivery centers of a project 28

IMPACT ON AGENCIES ................................................................................................................................ 29 IMPACT ON SOCIETY .............................................................................................................................. 34

CHAPTER 4: ASSESSMENT OF MEASUREMENT FRAMEWORK ................................................ 36 ............................................................................................... CONCLUSIONS AND FUTURE ACTION 38

..................................................................................................................... SUMMARY OF KEY FINDINGS 38 L m r r ~ n o ~ s OF THE STUDY AND AREAS FOR FURTHER RESEARCH .......................................................... 40

ANNEX I PRELIMINARY MEASUREMENT FRAMEWORK ...................................................... 42 ANNEX I1 LIST OF ICT PROJECTS COMPLIED FOR WHICH EVALUATION DOCUMENTS WERE ANALYZED .......................................................................................................... 48 ANNEX I11 SURVEY AMONG TARGETED CLIENTS. AHMEDABAD MUNICIPAL CORPORATION .......................................................................................................................................... 55 ANNEX IV PROFILE OF RESPONDENTS .......................................................................................... 61 ANNEX V IMPACT ON CLLENTS OF PROJECTS IN INDIA ........................................................ 62 ANNEX VI FACT SHEETS ON PROJECTS STUDIED ...................................................................... 63

........ ANNEX VII RATING OF CLIENT IMPACT (AFTER ELIMINATION OF SCALE BIAS) 102 ANNEX VIII LOCATIONWISE ASSESSMENT ON DIFFERENT DIMENSIONS ......................... 103 ANNEX IX A METHODOLOGY FOR POSTEVALUATION ......................................... ........... 105

vii

. ............... TABLE I TYPES OF ICT APPLICATIONS DEPLOYED IN THE PUBLIC SECTOR 6 TABLE I1 . KEY OUTCOME DIMENSIONS ...................................................................................... 13 TABLE III . LIST OF SEVEN PROJECTS FOR DETAILED STUDY .............................................. 15

. ................... TABLE IV WAGE LOSS (RS) FOR MANUAL AND COMPUTERIZED SYSTEMS 20 TABLE V . PERCEPTION OF COSTITIME REDUCTION: TAX ONLINE AND CHILECOMPRA ......................................................................................................................................... 20 TABLE VI . AMOUNT PAID TO FUNCTIONARIES AND INTERMEDIARIESIAGENTS (RS).23

. ............... TABLE VII DESCENDING ORDER OF IMPROVEMENT IN COMPOSITE SCORES 25 . TABLE VIII TOP FOUR DESIRED ATTRIBUTIONS OF SERVICES ............................................. 25

TABLE M . KEY INDICATORS FOR ASSESSING IMPACT ON CLIENT .................................... 28 TABLE X . IMPACT ON AGENCY ...................................................................................................... 30 TABLE XI . SAVINGS IN COST TO CUSTOMERS-ESTIMATES FOR ENTIRE CLIENT POPTJLATION .............................................................................................................................................. 34

............................ . TABLE XI1 ATTITUDE TOWARD E-GOVERNMENT-INDIAN PROJECTS 35 TABLE XD[I . DETERMINATION OF SAMPLE SIZE ........................................................................ 107

.................................. . FIGURE I E-GOVERNMENT PROJECT SUCCESS APPRAISAL MODEL 9 FIGURE I1 . NUMBER OF TRIPS SAVED ............................................................................................ 18 FIGURE 111 . TRAVEL COST SAVED(RS) ............................................................................................ 19 FIGURE IV . WAITING TIME SAVED (MINUTES) ........................................................................... 19 FIGURE V . DIFFERENCE IN OVERALL SERVICE QUALITY SCORE ...................................... 21 FIGURE VI . ERROR RATE (PERCENTAGE) ...................................................................................... 21 FIGURE VII . PROPORTION PAYING BRIBES (PERCENTAGE) ..................................................... 22 FIGURE VIII.DIFFERENCE IN OVERALL GOVERNANCE SCORE ........................................ 23 FIGURE M . RADIAL CHARTS REPRESENTING CLIENT IMPACT ............................................ 26

..................................... . FIGURE X INVESTMENTS AND OPERATING COSTS OF PROJECTS 31 FIGURE XI . PERCEPTION OF SUPERVISORS OF IMPACT ON AGENCY (PERCENTAGE) . 33 FIGURE XI1 . PREFERENCE FOR COMPUTERIZATION (PERCENTAGE) .................................. 38 F1GU.R E XIII.INTERACTION TREE-* REFERENCE FOR COMPUTERIZEDMANUAL SYSTEM BASED ON USER PROFILE .................................................................................................... 69

viii

An increasing number of governments are investing in information and communications technology (ICT) as an integral part of their strategies to improve public sector administration and services. The efforts to automate back-office functions and to introduce management information systems started decades ago. With the advent of the Internet and rapid development in technologies, ICT has made its way to front-end service delivery through such means as a government portal. While ICT has become more common in contemporary governments, evidence to date on the outcomes of ICT investments in the public sector has not been clearly established. This is particularly valid for ICT investments in developing countries. On one hand, a number of success stories in developing countries are widely lauded as best practices, but on the other hand, a reported high rate of failure based on expert opinions and case studies suggests high risks involved.

This study was launched to make a modest beginning for systematic assessmenit of ICT investments in the public sector in developing countries. The effort was to go beyond the anecdotal assessments that were commonly available. The objectives of the study include: (i) to develop and present a framework to conduct analysis on the innpact of public sector investments in ICT applications; (ii) to assess the effect of ICT investments on performance measures for a selected set of projects and identify the key determinants of economic, organizational, and social impact, based on analysis of sample cases; and (iii) to propose guidelines and a methodology that can be used by subsequent studies to work with more comprehensive data sets and expanded metrics of outcome to assess impact of future projects.

Existing literature on impact assessment methodology was reviewed to clefine a measurement framework. The proposed framework focuses on the idea of measuring the total value delivered by a project to different stakeholders. It recognizes that somle part of the value for each stakeholder can be monetized and other parts need to be ;assessed qualitatively. The aim is to measure performance of the manual and computerizetl system on the following five dimensions: (a) cost of access to clients, (b) client perception of quality of service, (c) client perception on quality of governance, (d) agency cost and revenue, and (e) employee perceptions about process changes. Difference between the computerized and the manual system was analyzed for each dimension, and statistical significance of the difference was evaluated.

The framework was used to make detailed assessments of five mature wide-scope service delivery projects from three states in India and two projects from Chile. In the Indian projects, online operators deliver services at designated service centers. In these projects, data was collected from 240 randomly selected respondents from eight stratified locations. The responses encapsulated the experience of clients with the use of the computerized delivery system as well as the manual system. The survey covered 120

questions grouped under five dimensions on which impact was being assessed.

In the two Chilean projects, services are accessed through a portal. In each of these, projects data was collected through an Internet survey of more than 600 users of the computerized system, and through focus group discussions with employees of the implementing agencies and citizens. Data was also collected from agencies and employees through structured formats and surveys.

Key findings from the study are primarily based on the detailed impact assessment of five Indian projects where data collection was comprehensive. Results from the two Chilean projects confirm many of the findings from the Indian projects, even though the nature of application and country context were quite different. Although the sample size was relatively small, the estimates of many different types of impacts were found to be significant at 99 percent levels. The study points to the following key features of the evaluated projects.

Survey respondents who have used both the manual and computerized systems in the five projects in India indicated an overwhelming preference for computerized service delivery. Their preference was supported by specific areas where concrete benefits have accrued to them. In most cases, the cost of accessing service for sample users was reduced because the number of trips required to be made to the concerned offices were reduced significantly and waiting time came down by nearly 50 percent. The quality of service delivery and of governance was also perceived to have improved with computerization.

There is considerable variability in the overall composite scores across five projects. A composite score that could be compared across projects was computed from a rating of each project on a list of 16 to 18 common attributes of an e-government application. Respondents were also asked to select the three most important attributes. Applying user-assigned importance to different attributes, a weighted composite score was calculated for each project for the computerized as well as the manual system. The difference in the composite score between the two systems indicated the degree of perceived impact. The differences in scores for manual and computerized systems range from 0.15 to 1.6 on a five-point scale, while a difference of 1.6 points indicates a significant difference, and 0.15 represents virtually no improvement. In general, these ratings corroborate well with data on cost savings to customers, perception of quality of governance, and quality of service collected through an independent set of questions. The distinct preference for computerized systems by the clients seems to suggest that even small gains for the users can trigger a significant positive change in perception about the overall quality of service delivery systems.

Although the outcome is mixed in reducing corruption, ICT seems to have the potential for significant reduction in corruption in service delivery, as indicated by one project. Less than one percent of surveyed users of one of the three projects where significant corruption was reported in the manual system responded that they paid bribes to officials after the services were computerized.

In the other two projects the impact was marginal, as corruption continued at a significant level in the computerized systems. In the two Chilean projects, the perceived impact on reducing corruption was marginal.

Estimates of incremental costs for agencies (including amortized investment) per transaction in all the Indian projects are less than the reduction in direct cost of accessing the service reported by the clients. This indicates that the idea of a user fee that covers the incremental costs to make the applications economically viable may not face political resistance. Any resistance to ithe levy of a user fee comes from the lack of credible information on costs and benefits delivered by projects. If projects can become self-sustaining through rlevenues from user fees, private sector investment is one area that can be tapped. Three of the five Indian projects have private partners, and the expansion of the fourth project in rural areas is being planned through private partnership.

Significant variability of impact across different delivery centers in a project was observed. This variability should be a cause for concern for delivery models in which physical service centers are created. Part of this variability is explained by poor infrastructure (power and connectivity), particularly in rural areas;. Given the large variation in activity levels at different centers, it is often difficult to match the capacity to the demand at each of these centers. Portal-based delivery accessed via the Internet can be a solution; however, unequal access to Internet can put some users at a disadvantage in such systems.

In all cases the ability to cope with growth in transactions was perceived to have been enhanced. In some cases, computerization helped in the griowth of transactions. Computerization was partially responsible for improved tax collections by some agencies. While the staff in these agencies did not perceive that cost had been reduced, they did feel that there was a significant positive impact on the quality of governance.

Surveyed citizens, irrespective of their rural or urban location or educational background, strongly supported the idea that more agencies should be computerized.

More projects need to be evaluated from different contexts before generalizing the conclusions of the positive impact and economic viability of electronic service delivery projects. The selected projects have operated for at least three years or more, implying successful implementation. All the projects except for one are serving predominantly urban clients. Projects that serve rural clients could have a very different cost structure and demand pattern.

The methodology used in this study has produced estimates of direct benefits for the five Indian projects and an assessment of perceived change in costs in the Chilean projects. It may be used in other contexts for similar retrospective assessments. Key elements of the proposed methodology are: selecting a project; constructing a project profile; identifying key dimensions of impact; defining measurement indicators; design of survey instrument,

including design of questions, sampling plan, and a data validation scheme; and data analysis.

The results of the retrospective evaluation of the seven projects carried out in this study highlight the relevance of such an exercise for government officials preparing a plan for an ICT project and development institutions such as the World Bank. The results provide a cue for investigating the reasons why a certain project may not be delivering on a potential area of service improvement. They also provide a preliminary benchmark for comparing other ongoing projects and those that will be implemented in the future. The approach enables an assessment of project viability in terms of cost and benefits-both in directly measurable and qualitative terms. The relevance of the proposed approach for practical use is demonstrated by the fact that the Government of India has adopted this framework for carrying out an impact assessment of a large number of projects implemented at the national, state, and local levels in India.

INTRODUCTION AND CONTEXT

It has become the norm rather than an exception to utilize hformatilon and communications technologies (ICT) in a broad range of public sector operations in contemporary governments. The term ICT encompasses diverse technologies such as computing, networking, hardware, software, telecommunications infrastructure (landline, mobile, and broadband fiber networks), and broadcasting devices. The Internet symbolizes this definition of ICT, as it utilizes many of these technology components. Investments in ICT are made to create the underlying infrastructure and also to build applications that utilize the computing power and connectivity. The rapid technological development in recent times was accompanied by a significant decline in the prices of hardware, software, and related auxiliary services. Deregulation and liberalization of the telecommunications sector resulted in diversification of information transmission modes, making the delivery of online services much easier and, most importantly, cheaper. In recent years an increasing number of projects have been launched under the label of "e- government," and several governments have identified this as a key policy priority.

E-government projects deploy ICT applications in the public sector for different purposes. This portfolio of applications has been described as a continuum in terms of "stages of growth" from e-administration to e-services to e-participation (Ranerup 1999; Heeks 2001). Many governments started using ICT to improve efficiency and overall performance through automation of back-office functions and introduc~tion of management information systems (MIS). Following the trend in the private sector, governments then expanded the scope of ICT projects to areas requiring direct int~eraction with citizens. Informational Web sites provided the entry point for citizen services. As the technologies matured, the level of interactivity and integration increased. Currently, attempts are being made to use ICT to induce wider public participation in policy-.making processes. Through such means as electronic voting and virtual discussion forums, the intersections between ICT and governments are no longer limited to the traditional public administration arena. ICT is beginning to demonstrate its effect in the fundamental process of policy making, government-citizen relationships, and, ultimately, how democracy functions. Table I below illustrates the typical areas where ICT is being deployed.

Table I. Types of ICT Applications Deployed in the Public Sector

Back-office automation1 / Accounting SystemsFinancial I Increased efficiency in public Types

operation support I Management Systems I administration

Examples I Intended Outcomes

I Human Resource Management I Transparency in public financial

I

I Systems ( management I Facility Control Systems

Data gathering and

Electronic delivery of services 1 .

Sector-specific Management Information Systems

Web sitesltransaction portals Integrated online delivery centers and rural telecenters Online customs, tax, issuance of certificates, and municipal services

E-discussion

Improved expenditure control

Effective performance monitoring Improved decision making

Reduced costs for transacting with government for citizens and business Enhanced revenue, greater transparency, and reduced corruption

Citizen engagement

While an increasing number of governments are investing in ICT, evidence to date on the outcomes of ICT investments in the public sector has not been fully established. This is particularly valid for ICT investment projects in developing countries, some of which are financed by development institutions such as the World Bank. On one hand, the recent success of a few projects in improving the investment climate, improving delivery of services, and creating a market for information in rural areas has reinforced the idea of ICT for development. On the other hand, evidence of failed projects has drawn attention to the level of risk involved in implementation. A failure rate of more than 50 percent based on the opinions of 15 experts and student submissions is widely cited1 in this context. A recent study commissioned by the Department for International Development (DFID) is researching firm-level impact of ICT in developing c~untr ies ,~ but similar studies have not been undertaken for the public sector. A report by the United Nations (2003) laments the fact that documented research on the social or economic impact of e-government development is virtually n~nexistent.~ While multilateral organizations such as the World Bank fund nearly seven billion dollars of ICT investment as part of their lending programs and grant assistance to client countries, evidence to date has been ad hoc, highlighting the need for systematic assessment of the impact of these applications.4

' See Heeks 2003. "Success and Failure Rates of E-government in Developing/Transitional Countries: Overview." IDPM, University of Manchester, U.K., e-government for development Web site: http:llwww.e-devexchange.orgleGov1sfoverview.htm

An ongoing study sponsored by DFID at the London Business School that is collecting data from firms in India and Brazil to establish a relationship between ICT investments, profitability, growth and productivity at the firm level. Degree of collateral organizational changes will be studied as a mediating variable.

United Nations 2003. "World Public Sector Report 2003: E-Government at the Crossroads," p. 61. New York.

The report quotes: "On the World Bank side, the largest, yet the least monitored, investments in ICT are IT components of projects in other sectors." See World Bank Group. April 2002. "Information and Communication Technologies: A World Bank Group Strategy," p. 36. Washington, D.C.

This study aimed to ascertain the extent to which the intended outcomes from ICT applications in the public service delivery have been achieved. It is an attempt to make a modest beginning to go beyond the anecdotal evidence currently available on the impact of ICT investments in the public sector in developing countries. More specifically, it aims to:

Develop and present an analytical framework to conduct analysis on the innpact of public sector investments in ICT applications. Assess the effect of ICT investments on performance measures for a selected set of projects and identify the key determinants of economic, organizational, and social impact, based on analysis of sample cases. Propose guidelines and a methodology that can be used by subsequent studies to work with more comprehensive data sets and expanded metrics of outcome to assess impact of future projects.

To this end, it assessed the extent to which a number of sample projects have resulted in direct and indrect value for: (i) clients (citizens and businesses), (ii) the agencies that implement systems, and (iii) the broader society. Based on the findings, this report presents a systematic framework and methodology to analyze the impact of future public sector investments in ICT applications.

The study focuses on the use of ICT in the delivery of public services. The j.ntended outcomes from the four broad types of applications indicated in Table I are important components of governance reform leading to the attainment of key social and economic development goals. The focus on the service delivery aspect was chosen because ithis area directly affects the nature of interactions between governments and citizens. It is, therefore, the area where the change in the quality of governance is most visible to citizens and businesses.

This report is organized under six major headings. The Introduction discusses the scope and objectives of the study. In Chapter 1, the review of literature on existing frameworks and methodologies for impact assessment of e-government projects is presented. Chapter 2 describes the evaluation framework and methodology as applied to the assessment of seven projects from India and Chile. The types of analyses that were carried out on the survey data collected from users and agency staff for the projiects are also presented. Results from the analysis are presented in Chapter 3. Chapter 4L briefly discusses the assessment framework used in this report. Finally, Conclusions ant1 Future Actions summarizes the key findings and discusses the usefulness of the study for developing countries.

CHAPTER 1: ASSESSMENT OF PUBLIC SECTOR ICT PROJECTS

Many ICT projects have been launched in different countries that were initially dubbed "success stories," but which eventually failed to make any significant impact. For example, a recent EU report (2003) based on a study of over 100 projects found that success with realizing concrete and measurable economic benefits as a result of e-government initiatives remains minimal. In the context of developing countries, many projects launched have not led to any significant improvement in the delivery of government services (Benjamin 200 1, Heeks 2002, Kanungo 2003).

While a significant amount of academic and policy studies have been conducted, these have so far proved inadequate in providing policy makers with a systematic framework for evaluation. One group of studies provides macro-level estimations of e- government activity using appraisal indices focusing on supply-side, quantifiable measures such as Web presence of government, network coverage, institutional and regulatory support, and human capital provision.5 Such factor-based assessments of e- readiness do not have immediately obvious and tractable policy implications. They tend to focus almost exclusively on measurement of physical access to certain types of ICT without incorporating issues such as affordability, appropriateness, ICT capacity and training, and the regulatory and macroeconomic en~ironment.~ The second group of studies, which have occurred in a largely anecdotal, piecemeal fashion, provide project- level evaluations with little prospect for synthesis from past approaches. A small number of studies have focused on the impact of projects in terms of how they affect both citizens and government agencies, but findings have seldom been linked to broader national, regional, or global trends; for example, in terms of changes in the approach to development and governance policy (Madon and Kiran 2002, Grant 2005, Heeks 2003).

Recent research in information systems, on the other hand, focuses on understanding underlying processes of change as a necessary input toward impact assessment. Of importance here is the evolving historical and institutional background within which an ICT project is implemented. Such organizational background has shown to influence the process by which resources allocated for an ICT project leads to a successful or unsuccessful system (Avgerou and Madon 2004).

The focal unit of analysis in the growing body of literature on enterprise information systems is the project in which the information systems success model presented by Delone and McLean (1992) is widely cited. The model is based on an analysis of 180 papers and proposes six major variables of impact which are interlinked with each other-systems quality, information quality, information systems use, user satisfaction,

Examples of such frameworks are (1) UNPAN (2004) E-Government Readiness Report, www.unpan.org and (2) Brown University (2004) Global E-Government Report, available on www.insidepolitics.orpr/e~ovtO5int.pdf

www.bridges.org (2005, May). "E-ready for What? E-readiness in Developing Countries: Current Status and Prospects Toward the Millennium Development Goals." http://www.infodev.orpr/fiIes/2O49 file InfoDev E Rdnss Rpt revllMay05.pdf

individual impact, and organizational impact. Impact on the individual was later broken down into economic benefit and social benefit (Seddon and Kiew 1996).

The enterprise information systems success model provides a generic approach to analyzing informatization in organizations in terms of the system, usage of th.e system, and its effect. The goal of a public sector organization is, however, more complicated than an enterprise information system, as its function includes both internal management and public service delivery. While enterprises view economic benefits as their strategic goal, the government needs to take both economic and social benefits into account. The development of a framework for appraising the impact of ICT applications in Ithe public sector, therefore, depends on the formation and sustenance of a supporting institutional, infrastructural, and policy environment. The e-Government project Success Appraisal Model (Figure I) incorporates a variety of objectives being pursued by governments. It aims to serve as a basic reference framework and important building block for evaluating e-government projects. Rather than impose a rigid template for conducting the appraisal exercise, the model is intended to be a guide to improve the impact of systems, acknowledging that different countries may have different contextual conditions and

- -

motives for implementing e-government applications.

Figure I. E-Government Project Success Appraisal Model

Foundation and environment of e-government I Source: Hu, Y , Xiao, J. Pang, J. and Xie, K. (2005). A Research on the Appraisal Framework of e- Government Project Success. Proceedings of the 7th International Conference on Electronic C~xnmerce.

These discussions have been incorporated in assessment frameworks for IC?' projects developed by different public sector agencies and The U.S. govt:mment7s Performance Reference Model (PRM), for example, is a widely cited model that aims to

' The indicative items are based on a review of the following documents: i) Performance Reference Model of the U.S. Federal enterprise architecture framework used by the Office of Management and Budget in U.S. federal government; ii) European Commission, eGovernment Economics Project (eGEP), Measurement Framework Interim Version Deliverable D.2.2, June 27, 2005; iii) Rarna Rao, T. P., Rao, V. Venkata, Bhatnagar, S. C., and Satyanarayana, J. (2004). "E-Governance Assessment Fraimeworks." Available on http://egov.mit.gov.in; iv) Lau, E. (2005). Electronic Government and the Drive for Growth and Equity. OECD; and v) Lanvin, Bruno (2005, November). "METER E-Strategies Monitoring and Evaluation Toolkit." E-Development: From Excitement to Effectiveness, ed. Robert Schware, prepared for the World Summit on the Information Society, Tunis, Tunisia. World Bank.

measure value creation in the public sector. The PRM is a tool to evaluate the impact of IT investment across federal government agencies and serves as the overall driver for a set of interrelated reference models developed under the U.S. Federal Enterprise Architecture Program. The model is used to facilitate cross-agency identification and analysis of duplicative investment, gaps, and opportunities for collaboration and integration across agencies, drawing on the best of existing approaches to performance measurement in the public and private sectors and harnessing various tools and methodologies frequently used in the private ~ e c t o r . ~ The framework builds from value chain and program logic models to reflect how value is created as inputs (such as technology) are used to create outputs (through processes and activities), which in turn impact outcomes (such as mission, business, and customer results). Guiding the entire PRM framework are strategic outcomes representing broad policy priorities that drive the direction of government.

The national e-government frameworks developed by the Australian (Commonwealth of Australia 2004) and German (Federal Ministry of the Interior 2006) governments focus broadly on the same dimensions as the PRM but with an emphasis on overall agency impact. These methodologies offer two levels of impact assessment: first, in terms of how the project provides a business case justification of expenditure and policy and whether it meets the targets set for it, and second, in terms of how the project meets the goals of the agency concerned and, in turn, how this helps in meeting wider government strategies.

As many e-government initiatives worldwide are struggling to demonstrate the expected benefits, there is a growing demand to understand the 'processes' that deliver a return on investment and generate value (EU 2003). Common best practice factors for value creation in e-government projects have been well-documented in several reports by governments and consultant firms (Accenture 2002, Gartner 2002, EU 2003). An example is the study conducted jointly by the Danish Ministry of Finance and Accenture that identifies two primary dimensions of value: (i) value to investors in terms of tangible financial benefits, cost savings, cost avoidance, and increased revenue; and (ii) value to users in terms of improved services, reduced cost and time savings to citizens, and reduced administrative burden to businesses. Investment in government employees and technological infrastructure are identified as key enablers for creating value through e-government. The focus has been on using business case methodologies from the private sector to demonstrate the economic value of e-government projects.

The calculation of time and money spent in finding and using public information are the most direct and measurable benefit of e-government applications to users, as shown in recent Organization for Economic Co-operation and Development (OECD) and European Union (EU) studies (Lau 2005, eGEP 2005). These reports identify improved revenue collection, lower costs caused by efficient processing of transactions, and a reduction of administrative burdens caused by simplification or elimination of procedures as some direct impacts that can be measured. Financial savings to users in terms of time and money spent in finding and using public information are the most direct and measurable

Some of the main techniques included in the PRh4 are strategic planning and analysis, business cases, value chain, applied information economics, and the balanced scorecard.

10

benefits of e-government applications for clients. A significant amount of work has also been done on studying users' perceptions of quality in terms of attributes such as accessibility, attitude of staff, cost of service, provision of information, .procedural fairness, and convenience.

While the benefits of process improvements in the public sector are acknovvledged in many studies, the links to increased productivity of government services are not straightforward. The expected links are not easily inferred through the expc:riences of ICT implementation in the business sector. The distinction between measuring productivity increases in the manufacturing and services sectors has been articulated in a recent McKinsey report in which the authors document how service businesses around the world are trying to boost their productivity (Harmon et al. 2006). Unlike consumer goods, which are relatively similar and therefore comparable, the report describes how there is often great divergence in the way services are recorded and monitored among groups even in the same organization. Service companies, therefore, need to develop internal benchmarks to find their own overall best practices and to see where and how they can be achieved. These internal benchmarks may differ from company to company highlighting the importance of context in impact assessment studies. The same would apply for measuring improvement in government services.

Calculation of net economic benefit of an IT project has proven to be challenging, as outcomes are multidimensional and composed of both quantitative and qualitative indicators. Further attention is required to be given to the linkages among issues of quality of service, governance, wider impacts on society, and ICT investment. Newer frameworks are evolving around a notion that an exclusive focus on financial costs and benefits for the government ignores many important noneconomic benefikg One such benefit relates to improvements in various aspects of governance activity. Another noneconomic benefit relates to the addressing of wider policy priorities, which, in the context of developing countries, could constitute the priorities as articulated in the United Nations Development Programme (UNDP) Millennium Development Goals.

This study builds on these recent studies on the assessment of ICT projects in the public sector. These studies indicate the value of ICT to be its contribut:ing to the achievement of organizational priorities. They also call for an assessment framework applicable for different ICT projects, so that systematic evaluation of the role of ICT can be undertaken. Following these discussions, this study focuses on understanding and documenting how inputs lead to outcomes in particular projects, given their contextual conditions. The assessment framework focuses on the idea of measuring the total value delivered by a project to different stakeholders and takes a balanced approach between case-study and quantitative analysis. It recognizes that some part of the value for each stakeholder can be monetized and other parts needs to be assessed qualitatively.. The next section provides a description of the measurement framework and the assessment methodology of this research.

Lau, Edwin (2005). Electronic Government and the Drive for Growth and Equity. OECD.

11

CHAPTER 2: MEASUREMENT FRAMEWORK AND DATA ANALYSIS

MEASUREMENT FRAMEWORK AND METHODOLOGY

Most of the assessment models proposed in the literature have not been used in the context of a developing country and many have not yet been applied in practice even in developed countries. Nor do they account for a variety of delivery models used in developing countries such as common service centers and franchised outlets that can retail e-services offered by the government. Practical issues of paucity of data have not been taken into account, particularly in case of a developing country where baseline surveys are not done and monitoring and evaluation systems are weak. Recognizing that adequate data for quantitative assessment is not available for most of the projects, collection of such data using a standard measurement framework for a sufficiently large number of projects is the first task that needs to be undertaken. Since impact assessment is the key objective, establishing counterfactuals is an important element of the proposed measurement framework.

A number of empirical studies suggest that ICT has had an impact in improving the performance of private sector organizations, particularly in developed countries. However, regarding ICT investments by the public sector in developing countries, many researchers have noted that past evaluation studies have not used a common framework or methodology and that rates of success or failure have been declared based on purposive samples.'0 Since ICT is introduced mostly in the context of governance reform to improve efficiency, effectiveness, and transparency of governments, a crucial first stage is to ascertain to what extent these intended outcomes from ICT applications in the public sector have been achieved.

A common measurement framework evolved on the basis of existing frameworks has been used in this study. An e-service delivery project impacts three groups of stakeholders: (i) clients receiving the service; (ii) agency (including several partners) that delivers the service; and (iii) the larger society consisting of citizens, businesses, government as a whole, and civil society. The impact can be assessed in terms of a variety of outcomes experienced by each type of stakeholder. Table I1 lists key dimensions of outcomes for each type of stakeholder. The complete assessment framework is provided in Annex I.

10 Peters, Rob M., Janssen, Marijn, and van Engers, Tom M. (2004, August). "Measuring E-Government Impact: Existing Practices and Shortcomings." Proceedings of the 7" International Conference on Electronic Commerce.

Table 11. Key Outcome Dimensions

Client -

Agency (including partners in

Stakeholders Economic (direct and indirect) Governance (corruption, accountability, transparency, participation) Quality of service (decency, fairness, convenience, etc.) I Economic Governance (direct (corruption, and indirect) accountability, transparency, participation)

Key Dimension of Impact

implementation) I Performance on key noneconomic objectives

~ o v e i m e n t as a whole responsiveness) Civil society Development goals

Society

I ~t t i tude toward computerization of public service delivery

Process improvements

Governance (corruption, accountability, transparency, participation,

The generators of these outcomes are inputs and business processes. 'Therefore, assessment needs to incorporate project information such as inputs (technology, human capital, etc.); process outcomes (reengineered processes, shortened cycle time, improved access to data and analysis, flexibility in reports); and organizational processes (institutional arrangements, organizational structure, and other reform initiatives of the government that might have influenced the outcome of the ICT project)."

As a first step to apply the framework to actual projects, a list of projects from developing countries intended for public sector operations that have some level of impact evaluation conducted and documented was compiled. Projects included in the list had a comprehensive scope in terms of functions that had been automated in an agency or were projects that had a statewide or region-wide coverage in large countries and a national coverage in small countries. The investment in such projects was generally upward of $2 million. Thus, pilot projects and projects with a small scale and scope were excluded. Different types of documentation were collected on these projects, especially tlocuments that were evaluative. The list of about 75 such projects is presented in Annex 11. Some observations were made based on this sample:

Evaluative studies had been done to serve a variety of purposes. Some studies looked at implementation success in terms of whether the systems were functioning as they were designed to, or the degree to which the specified outcomes were achieved. Some studies looked at long-term sustainability and scope for replication; l2 some measured the benefits that were delivered to agencies.13 A few focused on benefits to the clients.I4 Hardly any comprehensive study assessed the impact on all the stakeholders and covered both shont-term and long-term direct and indirect impacts. A few studies had carried out a cost-benefit analysis. Often, evaluation studies had been done by agencies that wer~e likely to

l 1 Hu, Yong, Xiao, Jing Hua, Pang, Jia Feng, and Xie, Kang (2005, August). "A Research on the Appraisal Framework of E-Government Project Success." Proceedings of the 7th International Conference on Electronic Commerce. 12 Government of India, Ministry of IT, has commissioned quick assessment of 29 projects. l3 Korea's eprocurement agency has evaluated the impact on dfferent government agencies using the system. l4 Global Knowledge Sharing Program had four Indian projects evaluated where clients were surveyed.

have an interest in showing a positive outcome. A variety of approaches had been used for evaluation. These included surveys, expert opinion, ethnographic studies, and internal assessments produced by - lending agencies. Different studies of the same project showed very different outcomes, thus indicating a lack of credibility of results.15 Part of the reason for different outcomes was the use of very small samples and a lack of rigor in sampling and collecting data from clients of the systems. The results could therefore not be easily generalized over the entire population of clients. The studies evaluated the functioning of the computerized system but were not able to assess the difference made by ICT use, as the need for counterfactuals (evaluation of systems as they worked before computerization) was ignored. Often, the impact of ICT use was not separated from other interventions that were made simultaneously with the computerization effort. Since different studies did not use a standard methodology, it was difficult to compare the outcome of one project with that of others.

From the compiled list of 75 projects, a sample of seven mature, wide-scope and -scale projects were selected for further study (Table 111). The list includes Bhoomi, Karnataka valuation and e-registration (KAVERI), Computer-aided administration of Registration Department (CARD), eSeva, and Civic centers of Ahmedabad Municipal Corporation (AMC) projects from India; and Tax Online and ChileCompra projects from Chile. They were chosen also to represent the countries in different stages of overall IT readiness and evolution.

l5 For example, the Bhoomi project of issuing copies of land titles has been evaluated by the Public Affairs Council, reporting significant positive outcomes, including reduction in bribes. Recent studies by a team from the Massachusetts Institute of Technology and the Indian Institute of Information Technology Bangalore found that corruption had not declined and major benefits were derived by land sharks.

Table III. List of Seven Projects for Detailed Study

r (RTC)'~ and filing of a

administration of

eSeva

, state, and local governments and public utilities and are now used

Corporation (AMC) I property tax, issue of birth and death certificates, and issue of shop licenses. Chile Tax Online Online tax filing was implemented by the Servicio de Impuestos Internos (SII),

which launched its portal in 1998. Attained maturity by May 2002 when 55 percent of Chilean citizens were filing their tax declarations online. By 2004, half

I of all tax returns were prepared automatically and 20 percent partially. In 2006,

I platform in 2003. Provides for online publication of procurement notices and e- ChileCompra

I mail alert services for registered businesses. Over 90 percent of private- and

97.4 percent of all declarations were conducted online.

Portal was launched in 2000, but came into full form as an e-procurement

I ~ublic-sector organizations have availed themselves of the Drocurement svstem.

For each project, the measurement framework was converted into a set of data collection instruments, including: i) a profile of the project identifying services, clients, and other stakeholders; ii) agency-level data on activity levels, investments, and operating costs; iii) a client survey questionnaire covering direct cost of access, quality of seirvice and governance; overall satisfaction, and perception of ICT projects based on exposure to different ICT applications; and iv) an employee survey for perceived impact on work, efficiency, and effectiveness. Measurement instruments were translated into local languages. A sample client questionnaire for AMC civic centers is shown in Annex III.

The survey conducted in India tried to assess both the manual system and the computerized system that replaced it on all the indicators to the extent possible. An analysis of the differences between the old and the new system was done to provide a measure of impact. A sampling scheme for survey of clients through structured

l6 Besides its other uses, this document is required for availing crop loans from banks.

CHAPTER 3: ASSESSMENT RESULTS FOR INDIAN AND CHILEAN PROJECTS

This section presents a comparative analysis of the Indian and Chilean projects from the perspective of the clients. Impacts for the key dimensions of cost, quality, governance, and overall satisfaction are analyzed. A profile of each project, detailed analysis of impact on clients and agency, and perception of employees is presented as a case study in Annex VI.

A COMPARATIVE ANALYSIS OF PROJECTS FROM CLIENT PERSPECTIVE

1. Cost to users

The use of ICT in public service delivery in India has reduced the number of trips surveyed users have to make to complete a transaction (Figure 11). The survey shows that in all five projects, there were savings in the number of trips citizens made to service facilities where public services are delivered (see Annex V for more detail). l8

Figure 11. Number of Trips Saved Greater formalization of processes after process reform, quicker retrieval of data from computerized databases, and automation of tasks such as writing and copying documents, as compared with manual methods, has reduced the total processing time of a service request in all applications.19 This has resulted in increased predictability in service delivery and reduced the number of trips.

Along with reduction in the number of trips, the travel costs per transaction have generally declined (Figure 111). The only exception is the Bhoomi project in which travel costs for obtaining RTC

l8 A paired t-test was applied to assess whether the differences between the means of the number of trips for the computerized and manual systems were significantly different. For details of the test statistic, refer to Annex V and project fact sheets in Annex VI. l9 In the case of AMC, the time elapsed has been reduced from 9.8 days to 5.3 days for certain types of transactions, while in CARD, the total time required for registrations has come down from 4.69 to 1.82 days. For details, refer to project fact sheets in Annex VI.

have increased by Rs 79.96 per transaction and for mutations by Rs 81.38. This is primarily caused by the change in the location of service offices. Under the manual system the RTC was issued in a village, but with computerization users need to travel longer to the higher-level government office (taluka headquarters) to receive the same service (more detail in Annex V). The opportunity cost for users, which is represented by wage loss because of travel, is, therefore, larger.

Figure 111. Travel Cost Saved(Rs) , The increase in travel costs for

Bhoom'- Bhoond- KAVERI CARD eSeva AMC RTC Mutation

expertise to support elaborate ICT applications. To planning to expand access in rural areas by allowing I database for issue of RTCs.

Waiting time at the service centers was reduced in all the projects. 20 Among those projects in which the reduction in waiting time were statistically significant 21 , the savings ranged from 16.16 minutes (AMC) to 96.24 minutes (CARD), amounting to a nearly 50 percent reduction from the waiting time in the manual system (Figure IV). Reduction in waiting time would have direct impact on

users of the Bhoomi project demonstrates one of the key challenges of service delivery using ICT. The change in the point of service delivery frlom the village office to the taluka headquarters was necessitated to ensure the availability of robust information infrastructure and technical support to operate the system. Many parts; of developing countries, partilcularly those outside major cities, may lack infrastructure and technical

address this problem, Bhoomi is ~rivately owned kiosks to access its

Figure IV. Waiting Time Saved (minut

e

s)

Bhoom- Bhoomi- KAVWl CARD eSeva A M : RTC Mutation

20 A paired t-test was applied to assess whether the differences between the means of the waiting time saved for the computerized and manual systems were significantly different. Refer to Annex V and fact sheets in Annex VI for details of the test statistic. 2 1 Only Bhoomi RTC was not statistically significant. See Annex V for more detail.

user costs through drop in foregone wages, as indicated in Table IV.

Table IV. Wage Loss (Rs) for Manual and Computerized Systems

N=sample size; S.E=standard error

While the difference between manual and computerized systems on all the cost indicators are statistically significant (in most cases at a confidence level of 99 percent), the accuracy of the estimate of the difference in direct costs can be improved by using larger samples.

In the two Chilean projects actual changes in cost of access were not measured directly. Instead, the impact on costs and time reduction was measured as a perception on a five- point scale. While citizen and business users of the Tax Online system reported a - significant perceptional reduction in the cost and time, in ChileCompra the perceived improvements were relatively marginal with a higher proportion of respondents perceiving no change.

Table V. Perception of Costrrime Reduction: Tax Online and ChileCompra

Costs required to search for business 3.28 1 Change in the economic cost to process 4.02 opportunities (0.07) taxes (0.07) Costs required to participate in 3.23

hid din^ (0.07) I - . .. - . . . I \ - . - ' I

Time Reduction Time required to search for business

Any change in the total amount of work 3.40

4.47 opportunities time and effort required to process

(0.07) (0.05) Time required to participate in taxes? 3.44

bidding (0.08) Note: Standard error in ( ).

2. User perception of service quality

Users rated the quality of services delivered using computerized systems as better than that by manual systems. For all five projects, users rated the computerized

22 services higher in overall quality (Figure V). The differences were statistically

22 A paired t-test was applied to assess whether the differences between the overall service quality scores for the computerized and manual systems were significantly different. For details of the test statistic, refer to Annex V and project fact sheets in Annex VI.

significant. The rate of error, which is an important measure of quality, was also1 lower for all the projects, except the issue of RTCs (Figure VI). Bhoomi also shows very significant improvement in service quality, among the highest in the five Indian projects assessed in the study.

Figure V. Difference in Overall Service Quality Score (five-point scale)

Bhoom- Bhoomi- KAVERl CARD eSeva AMC RTC Mutation

Figure VI. Error Rate (percentage)

RTC Mtation

Overall improvement in service quality was reported by users in the two projects in Chile as well. Both projects received favorable assessment for their user-friendliness and convenience of services. The average ratings on the quality of service of the tax online system were high, with many items being rated around 4.5 in five-point scale. On an average, users of the tax online system can complete tax filing in 30 minutes. Abolut one- third of business users and 3.6 percent of citizen users, however, had to make an appeal to SII regarding a tax decision. The system was also rated poorly on responsiveiness to feedback. For suppliers using the ChileCompra portal, the interviews revealed a marked increase in efficiency and effectiveness. While earlier a supplier had to constantly refer to newspapers for knowledge of purchase intent declared by the government, the ChileCompra portal provides such information at a single place. In terms of complaint

handling, about half of public sector users surveyed had to appeal to a ChileCompra decision and around one-third of user organizations had noticed errors in ChileCompra documentation. Focus group discussions and interviews revealed that many providers were disgruntled with procurement decisions made through ChileCompra. Thus, transparency of information appears to have led to increased complaints from suppliers.

3. User perception of governance and corruption

The proportion of surveyed users paying bribes has generally declined. The decline is higher in some projects than in others (Figure VII). For example, in the Bhoomi project, while about 30 percent of users were paying bribes in the manual system, less than one percent are continuing with the practice in the computerized system.

("/I 35.0

30.0

25.0

20.0

15.0

10.0

5.0

0.0 Bhoom- Bhoomi- KAVERI CARD eSeva AMC

RTC Mutation

a Manual H Cunputerized

Figure VII. Proportion Paying Bribes (Percentage) The proportion of transactions in which a bribe was paid in KAVERI continues to be high at 21.61 percent in the computerized system. A detailed study of one of the centers indicated that any type of system breakdown leads to corruption. The breakdown can be on account of an overload of demand in comparison to the capacity of the system to process registrations. Agents play a key role in

promoting corruption. Private operators also exhibit rent-seeking behavior if given an opportunity. Systematizing queues by appointments helps prevent breakdown.

An interesting phenomenon is that the amount of bribe users pay is generally higher in computerized systems in all the projects (Table VI). General inflation in the price levels is perhaps part of the reason for this. It may also suggest that the complexity of the types of favor requested through bribes have increased. The estimates of bribe amounts are less accurate than other results, as the sample of respondents paying bribes was small (50-75), resulting in high standard error of the estimates.

Table VI. Amount Paid to Functionaries and IntermediariesIAgents (IRs)

I

Other amount paid to intermediaries/agents (Rs) Manual 39.44 46.13 91.92 256.41 77.27 N 11

Computerized 1 78.75 1 50.00 1 200.75 1 398.61 1 - 1 60.00

N=sample size; S.E.=standard error

The overall governance rating is higher for computerized systems than ~nanual systems (Figure VIII).23 In the case of Bhoomi, there was a marked improvement in transparency and fairness, whereas KAVERI showed hardly any improvement in the quality of governance.

Figure VIII. Difference in Overall Governance Score (five-point scale)

Bhooni KAVEU CARD eSeva

23 A paired t-test was applied to assess whether the differences between the overall governance scores for the computerized and manual systems were significantly different. These were found to be significantly different at 99 percent confidence level. For details of the test statistic, refer to Annex V and project fact sheets in Annex VI.

The results for the projects in Chile appear to complement these findings in the projects in India. There was a general consensus among respondents that ChileCompra has resulted in the public sector being held more accountable for its spending. Focus groups and interviews revealed that it is possible for a supplier to verify and challenge why the government had rejected a bid in ChileCompra; however, ChileCompra was rated quite poorly on all the governance indicators by the vendors, whereas the public sector agencies rated it as moderately positive. Public sector users responded that ChileCompra had somewhat reduced corruption while there was no such endorsement from private sector users. The survey recorded strong agreement from public sector players that the rules and procedures of ChileCompra were clearly stated, while only 26 percent of the respondents agreed with this from the private sector. Public sector players strongly agreed that ChileCompra tends to add transparency to public sector operations, while weaker endorsement was shown from the private sector. The Tax Online system appears to have improved the perception of fairness among businesses, as revealed through focus group interviews. All businesses that have a permit to trade issued from the municipality are now automatically included in the system if their annual earning is above a certain threshold. Earlier, many such enterprises were outside the system. This measure is perceived as fairer by the business community. There was no strong endorsement, however, that the Tax Online had either increased transparency, eliminated corrupt practices, promoted confidential and secure data, or that the SII takes into consideration the opinion of citizens while developing norms and rules.

4. Comparison of five Indian projects on overall client impact

Table VII compares the composite scores on a five-point scale based on 18 common attributes encompassing the three dimensions discussed earlier. The score also factors in the importance of the attribute for each project. Respondents were asked to select three attributes considered important by them from a set of 18 attributes of a service delivery system. A weightage was assigned to each attribute on the basis of the proportion of respondents who selected the attribute. Weighted scores were computed for each attribute as the product of the weight and the difference between the average scores for the attribute on the manual and computerized systems. The sum of these differences denotes the improvement in composite scores after computerization.

The results indicate that Bhoorni shows the maximum improvement, as its manual version had the lowest score. eSeva also shows a very significant improvement, as the computerized counters are rated close to very good in the composite score. KAVERI and CARD indicate only marginal improvement over the manual system. Ranking of projects using this data, particularly the composite rating, can represent the degree of success of the project from the point of view of clients. Bhoomi and eSeva can be rated as very successful projects, AMC as moderately successful, and KAVERI and CARD as projects where there is considerable scope for improvement.

Table VII. Descending Order of Improvement in Composite Scoresz4 --

Nlanual Computerized Difference

AMC I 3.367 4.118 0.75 1

Bhoomi

eSeva

KAVERI I 3.345 3.897 0.552

2.857 4.461 1.604

3.388 4.658 1.270

Three kinds of attributes have been selected more often than others, as shown in Table VIII. They include strong preference for attributes on i) transactional efficiency, ii) improved governance, and iii) quality as measured by error rate and convenience. An important conclusion is that these attributes are different for different projects. If such an exercise was to be done before designing an application it would provide useful insights into benefits that need to be targeted and the kind of process reform required. It is rarely done in practice either before or after the project implementation.

CARD

Table VIII. Top Four Desired Attributions of Services

3.775 3.929 0.153

Project BHOOMI

Dwired Attribdtiun b~Seruiccs Error-free I No delay in I Less waiting time I Fewer visits

KAVERI

CARD

Figure IX presents radial charts for manual and computerized delivery for each project on the key dimensions included in the assessment framework. The overall score for manual and computerized delivery is presented as a circle in the same chart. Since different types of indicators have been measured differently (some using a five-point perception rating and others as actual measurements), a methodology of predetermined "goalposts" was used to normalize them. It reflects the feasible upper and lower limits to the rnleasures, adopted by the United Nations Development Programme (UNDP) in computing the Human Development ~ n d e x . ~ ~ Performance on each dimension is expressed as a value between 0 and 1 by applying the formula:

Dimension index = (actual value - minimum value) /(maximum value - minimum value)

transaction Less corruption

Less time and effort reauired

24 Differences between the composite scores for the computerized and manual systems were found to be significantly different at 99 percent confidence level. For details of the test statistic (paired sample t-test), refer to project fact sheets in Annex VI. 25 Human Development Report 2006, United Nations.

Less waiting time

Less corruption

Convenient time schedule Greater transparency

eSeva

AMC

transaction Greater transparency

Less waiting time

Fair treatment

Good complaint handling system

Less time and effort required Less time and effort required

Error-free transaction Less corruption

Less waiting time

Fair treatment

AMC

rbsence of

Quality of Governanc uality of Service

Score on Indicators for Manual System - - - . Overall Score for Manual System Score on Indicators for Computerized System __ Overall Score for Computerized System

These radial charts allow for a quick visual assessment of the rating of each project based on five key dimensions of client impact in its manual and computerized versions. The larger the area of the pentagon is, the higher the rating of the project is. It is easy to surmise that eSeva has been highly rated. Bhoomi appears to have shown the maximum improvement over the manual system. KAVERI has the smallest pentagon; that is, it is rated the lowest and has registered marginal improvement through computerization. Similarly, the composite ratings represented by the concentric circles show very marginal improvement in KAVERI and significant improvement in Bhoomi and eSeva. The charts also establish that the overall rating derived from individual components and composite scores are consistent.

The six key parameters that have been used to generate the radial charts are presented in Table IX. Scores obtained by different projects on these indicators after elimination of scale bias are represented in Annex VII

Table M. Key Indicators for Assessing Impact on Client

5. Variability of client impact across different delivery centers of a project

, , ' I d a * ' Cost

~ f f i c i e n c ~ 26

Quality of Service

Quality of Governance

Absence of Corruption

Overall Score

There are visible variations in the survey results in different locations of service facilities. Earlier analysis of client impact was based on the total number of respondents, typically surveyed in seven to eight different locations. Data was analyzed to understand if there were differences by location of service facility on a few key dimensions across all projects. The mean for each location is tested against the total sample mean for statistically significant differences (Annex VIII). On improvements in composite scores, which is an overall evaluation, there is significant variation in KAVERI, AMC, and Bhoomi. The variation in CARD is small, as the improvement itself is marginal. In eSeva the overall improvement is significant but the variation between locations is small. It is interesting to note that eSeva is the only application where the capacity of the service center is designed to match the activity levels.

Variation is largest among Bhoomi service centers. In particular, one service center (Location 4) facility scored the worst in terms of composite score. Respondents evaluated

Farmefer Number of Trips

Waiting Time (minutes)

Quality of Service (five-point scale)

Quality of Governance (five-point scale)

Proportion Paying Bribes (%)

Composite Score (five-point scale)

26 Since it is difficult to predict the worst scenario in the case of number of trips that would be required to avail a service, the highest average number of trips (in the manual system of registration in Karnataka) has been considered for the purpose of scale equalization. Similarly, for the Eficiency indicator, the maximum and minimum of actual averages of the time spent at the service center during each visit have been used as the lower and upper goalposts respectively.

,M*m Ydwb 0

14.466

5

5

0

5

> < . . < > I

M W m b Value 3 . 3 7 0 ~ ~

184.786

1

1

100

1

a 0.575-point deterioration in overall performance with the computerized system. The increase in travel costs because of the change in the location of service facilities from villages to taluka headquarters probably contributed to this result. Users of Location 4 experienced the largest increase in travel costs after computerization. Quite unexpectedly, the number of trips per transaction was also substantially higher in this service center. Users waited longer at the computerized service facility than under the manual system. Moreover, the mutation service provided in this location is the only case among surveyed facilities of the Bhoomi project in which the proportion of users paying bribes increased. There was a breakdown in the service delivery process in this particular facility. While the precise reasons for deterioration and the frequency of similar incidences in the entire Bhoomi project require further studies, the situation in Location 4 highlights the comprehensiveness of efforts required to improve service quality. Computerized system installation by itself does not ensure positive transformation of services.

Direct economic impact on agencies was expected in terms of operating costs, collection of tax revenue, and collection of transaction fee. There are inherent problems, however, in assessing impact on costs. For different agencies, comlputerized delivery of services touches different portions of the overall activity portfolio. For example, AMC is an organization that delivers large-scale physical service!; such as cleaning of roads and public areas, public lighting, and maintenance of urban infrastructure. Computerized delivery of services impacts less than one percent of all employees. Often, the prevalent costing systems do not permit identification of operating costs for only the computerization project. For agencies such as offices of the subregistrar the impact is more widespread, as the core service is computerized.

As mentioned in Section 3 of Chapter 2, it was difficult to collect cost data from the manual operations. By and large, the major cost component was labor. The case studies reveal that human resources were not reduced in any project after computeri:zation. In some projects, workload was reduced and, therefore, additional time could be dLevoted to other tasks. There could be cost implications in the future, as the transactions are growing at a much faster rate as compared with labor, as was seen in the case of KAVERI. The human resource and other costs incurred by the government have remained within a narrow range for the manual and computerized systems. After computerization, the total cost as a percentage of revenue from fees has declined from 6.48 to 4.23 percemt in two years.

Given these limitations, data on impact on agencies collected is summarized in Table X. Data shows that there has been a significant increase in revenues from transaction fees and tax revenue (wherever applicable) in recent years.

Table X. Impact on Agency

Total vroiect investment (Rs million) I 216.35 I 400 1 300 1 600 1 250

Bhoomi RTC Mutation

Annual operating expenses (Rs million) I 159.32 I 111.13 i 64.5 1 168.9 1 19.2 Number of clients served annually (million) I 7.62 I 1.33 1 0.33 1 15.3 1 0.29

KAVEKI CARD eSeva

Tax revenue (Rs million)27

Annual number of transactions (million) I 1 1.97 1 1.03 1 2.47 1 1.03 1 37.02 1 0.7 1

AMC

n.a. 1 19,245.07 1 17,282.68 1 n.a. 1 1,974.26 Tax revenue base (Rs million)

Growth rate in tax revenue

Transaction volume base (million) 1 6.56 1 0.29 1 1.49 1 0.41 1 10.5 1 0.27

n.a.

n.a.

Growth rate in transaction fee (percentage) 1 22.23 1 87.83 11.59 24.74 1 120.15 1 17.51

Growth rate in volume of transactions (percentage)

Transaction fee (Rs million)

Transaction fee base (Rs million)

n.a.: Not Applicable The years for current and base jigures for tax revenue, transaction volume and transaction fee are set differently for each project depending on project starting date and data availability. Please refer to Annex V for more detail. Growth rates are annualized

9,033.16

28.67

All of this increase, however, cannot be attributed entirely to computerization. Based on discussions with agency staff, impact on transactions, fees, and tax revenues was analyzed qualitatively. Some transactions, like bill payment in eSeva, are mandatory. The growth likely represents a shift from traditional manual based payment counters run by agencies and a natural growth in electricity and water connection. Other transactions, such as RTC and Mutation, are not mandatory. In addition to overall growth in land holdings, farmers may have been encouraged to keep their land records updated because of Bhoomi. In KAVER129 and CARD3' the transactions of property registration is mandatory. Part of the growth in other types of transactions3' may be attributed to computerization. For tax revenue, a more transparent system of valuation, rate of growth in property transactions, could explain growth. In AMC the growth in taxes is influenced by a growing base, better compliance, and accurate valuation; however, in all cases, computerization appears to have allowed the agencies to cope with high growth in the number of transactions.

22.23

179.58

98.34

Another key aspect that determines service levels in each project is the quality of maintenance of the infrastructure at various locations. A survey of about 10 front-line

4,765.27

23.95

" KAVER17s figures for tax revenue are revenue from stamp duty Of this amount, Rs 159.7 million was paid to the private operators, whereas the government's share was

Rs 2,467.3 million. 29 In 200041, when the system was manual, 0.63 million properties were registered. In 2005-06, 1.02 million properties were registered, representing an annual growth of 10.27 percent. 'O 1n 1998-99 when the system was manual, about 54,000 properties were registered. The registration jumped eightfold in 1999-2000, one year after computerization.

In the case of KAVERI, requests for non-encumbrance grew at 12.21 percent and copies of registration at 2.25 percent.

87.83

36.12

10.24

n.a.

n.a.

651.36

44.72

10.58 16.78 87.74

203.59

42.01

2,626.9528

1,890.46

38.04

53.32

32.86

1,130.87

300.18

operators in each project indicated that at least some centers had problems with power supply, connectivity, and slow response. A significant proportion (20 to 50 percent) of operators indicated that problems in these three areas are encountered sometimes (as opposed to never, rarely, often, and always). Hardware and software, in comparison, were seen to be more robust.

In terms of agency impact for Chilean cases, the SII (Tax Online system, Chile) indicated a significant increase in the efficiency of internal work processes. Implementation of ChileCompra has resulted in increased competition from suppliers. The total number of opportunities for businesses published on the portal expanded by 75 percent in 2lD05 from the previous year. For example, micro, small, and medium enterprises transacted 47 percent of all transactions in 2005, which was 8 percent higher than in 2004. Government departments found the system was easy to use and reported significant gains in efficiency in the purchase of p