the challenges of obtaining reliable data world bank group investment climate advisory services...

14
The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform success: business surveys, "client satisfaction" and administrative burden

Upload: benedict-holmes

Post on 21-Jan-2016

214 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

The challenges of obtaining reliable data

World Bank Group Investment Climate Advisory Services

Florentin BLANCVilnius, 22 March 2012

Measuring reform success: business surveys, "client satisfaction" and administrative burden

Page 2: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

The need for measurement – knowing where you stand, and where you are going

• Reforms are undertaken – but do the regulated/inspected businesses feel them? Do they see inspectors as a positive factor, or not? Do they complain about burden, lack of transparency etc. – or less?

• Essential to gather positive results in dealing with the regulated “clients” – while also maintaining or improving outcomes for the public – so the two sides should get measured

• Crucial for the Government as a whole – which has to show improvements to the citizens – and for the inspecting agencies, which have to report to the Government, and need success in improving effectiveness and efficiency in a difficult economic / fiscal context

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 3: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Range of instruments – all with specific upsides and downsides

• Business surveys: ask all, or a sample of, inspected businesses how their experience with inspectorate(s) was

• Focus-group discussions: gather several businesses with a certain profile and ask about experiences with range of structured questions

• Both of these methods can be implemented directly by inspectorate or (preferred) through a specialized company or researchers

• Open consultations with stakeholders (e.g. business associations)

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 4: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Focus of our measurement efforts – not just “what” but “how well”…

• Relatively easy to find out from businesses whether something took place or not (“Were you inspected?” – “Did you receive advice?”)

• More difficult to get reliable answers on questions of quality – which are inherently more subjective (and thus where answers can vary…)

• Specific focus in our case is to try and find out “satisfaction” with “quality of consultations”

• If possible, we should try and measure the results rather than subjectively expressed satisfaction

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 5: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Business consultations – what are we trying to achieve

• Immediate aim of consultations is to help business comply: knowing and understanding what they have to do means less violations

• Longer-term goal is to facilitate business creation, growth and diversification by providing more certainty for investment (if one operates within parameters explained in consultation, there is no risk of the business being closed down)

• Using indicators such as “compliance levels” and “business growth” to measure success of consultations is nonetheless problematic because they can be impacted by many other influences…

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 6: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

“Client-satisfaction surveys” – an instrument that can be misleading

• Frequent approach: measure reform success by asking businesses whether they see improvements

• Several risks with such surveys: Insincere answers – whenever the business

fears about confidentiality (e.g. survey given just after inspection…)

Subjective answers – correlated with external events or “latest impressions” and not reflecting actual “average situation” with inspections

• May over-estimate “success” – or hide it…

• Such “subjective” questions are better asked by independent researchers, in focus-groups or in-depth interviews – where the reasons for the answers can be asked and discussed THE WORLD

BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 7: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

An example of qualitative, independent investigation of business perceptions

• From the Business End of the Telescope

• Commissioned by UK LBRO

• Focus-groups, in-depth interviews (with individual businesses, business associations etc.) to understand how businesses perceive inspections, inspectors

• Gives real insights on negatives and positives, what businesses resent and what they appreciate, expect etc.

• Trust in independence of interviewers – professional competence to investigate

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 8: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Designing surveys – asking the “right” questions

• First crucial step: ensure trust of interviewees in independence of survey, confidentiality

• Ask questions that will give more reliable answers:

Objective quantitative indicators: how often, how long, etc. – allowing to track inspections patterns

Qualitative but not “like” or “feel” questions:

Did you get advice that was clear?

Was the inspector proficient in technical issues?

• Develop questions that are clear, precise, yield meaningful answers – questionnaire testing is needed

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 9: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Examples of “quantitative surveys”

• World Bank Group has for 10+ years been supporting surveys of inspections in countries where it assists with reform work

• Provide clear picture on how many businesses inspected, how often, how long etc. – aggregated across country – by sector – by agency etc.

• Of course margin of error – and some answers may be wrong (misunderstandings, bad memory etc.) – but mostly this gets “averaged out” by large numbers of respondents

• Very useful information because (among other issues):

Consolidated data was simply missing – Government, even inspectorates, often did not know situation

Discrepancy between practices and “official numbers”THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 10: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

The cost challenge – and the different technical solutions

• Business surveys have a cost – particularly if you don’t use the “questionnaire filled and collected at the end of the inspection” approach…

• Phone, mail, internet surveys have different costs – but also different levels of reliability

• Probably makes sense to combine “qualitative” instruments conducted by independent researchers, and “quantitative” ones – resources can be pooled by inspectorates, no need to always have a separate study for each

• Key is to properly conceive what we want to know: what are the important questions, and the target groups (all businesses, some specific sectors, sizes etc.?)

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 11: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Trust and credibility

• Businesses (and citizens – as some surveys could target inspected subjects that are not entrepreneurial ones) need to have full confidence that survey results will not harm them – so not only individual confidentiality, but even “group” confidentiality / results should not be traceable

• “Survey fatigue” will soon kick in if there are too many studies, and if they are not seen as having consequences

• Essential for Government, inspecting agencies to show that surveys / studies results are being acted upon and used to design / drive further changes

• Such link to policy actions can also be advertised before a new survey / study starts – to increase response rates / quality

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 12: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Some recommendations (1) – combine approaches

• Business surveys can be used for quantitative data (“did you receive advice?”, “were your questions answered?”) – particularly as long as most inspectorates do not have an information system to consolidate data on their activities

• Using (trusted) independent researchers to conduct qualitative research through focus-groups etc. is likely to provide more insights into satisfaction and remaining problems

• Direct discussions with business representatives are also essential to increase trust between the two sides

• Finally, trying to measure and track outcomes would be a very important step…

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 13: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

Some recommendations (2) – measure outcomes

• Far more difficult because of “attribution” issues and of measurement quality issues (for compliance – since it is measured through inspections => conflict of interest…)

• Areas of focus should be determined depending on baseline – if businesses particularly complain in some areas, these are the ones where to check if changes have taken place

• More business creations or expansions, more jobs, in a sector that had problems, can be indicators that consultations (and other changes) are working

• Also very important: through surveys (possibly internet) or independent research, try and identify why some were discouraged from creating or expanding business activity – this may show the remaining gaps

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency

Page 14: The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Florentin BLANC Vilnius, 22 March 2012 Measuring reform

14

Thanks for your attention

[email protected]

www.wbginvestmentclimate.org

THE WORLD BANK

World Bank Group Multilateral Investment Guarantee Agency