overview of the statistical assessment of the itu ict ... 2016-02 13... · overview of the...
TRANSCRIPT
Sevilla, 19 Feb. 2016 | 1
Overview of the statistical assessment of the ITU ICT Development Index (IDI)
PREDICT Workshop, February 18-19, 2016, IPTS Sevilla
Marcos Domínguez-Torreiro
COIN – Competence Centre on Composite Indicators and Scoreboards JRC Ispra - Italy
Sevilla, 19 Feb. 2016 | 2
Introduction
Statistical assessment of the ITU IDI 2015
COIN future work on ICT related issues
Outline
Sevilla, 19 Feb. 2016 | 3
Support to EC
130 CIs developed or used by the EC – 50% of the EC indices are developed with JRC contribution
Collaboration with international partners
Global Innovation Index 2015, WJP Rule of Law 2014, Environmental Performance Index 2014, Corruption Perceptions Index 2012
Methodology
In-house developed quality control frame (conceptual coherence, multivariate analysis, sensitivity analysis, multi-criteria decision analysis, statistics and policy) (NEW: updated and revised edition currently underway)
Training
Over 50 trainings in the last 12 years
COIN know-how on construction and statistical assessment of composite indicators is requested by over 100 international partners: OECD, WEF, INSEAD, WIPO, UN-IFAD, FAO, Harvard U., Yale U., Columbia U., Cornell U., …
Activity on Composite Indicators – 4 lines
COIN – Competence Centre on Composite Indicators & Scoreboards
Sevilla, 19 Feb. 2016 | 4
Ensuring the conceptual and statistical coherence of an index can be synthesized along five main steps:
1. Consideration of the conceptual framework with respect to existing literature
2. Data quality checks (missing data and outliers)
3. Assessment of the statistical coherence (correlation analysis, dimensionality and
grouping of variables)
4. Assessment of the impact of modeling assumptions (e.g. weighting scheme and
aggregation method) on the rankings
5. Qualitative confrontation with experts in order to get feedback on choices made
during the index development
Statistical assessment - Steps
Statistical assessments carried out by COIN focus mainly on the second, third and fourth steps
Sevilla, 19 Feb. 2016 | 5
1 overall index
3 sub-indices
11 selected indicators
167 countries
The ITU IDI framework
ITU IDI – Measure the level and evolution over time of ICT developments in countries and the experience of those countries relative to other countries
Sevilla, 19 Feb. 2016 | 6
Data checks
Log-transformation - ‘International Internet bandwith’ indicator
Huge dispersion of values - aims at favoring improvements at the low end of this indicator Renders the indicator more correlated with the remaining 4 indicators in the ICT Access dimension
Capping – for all indicators (based on reference values)
In line with the objective of setting ‘ideal’ reference values to be achieved by countries
After log-transforming/capping the indicators are not affected by outliers (i.e. skewness > 2 and kurtosis > 3.5) that could bias the calculation of the aggregate scores or the correlation structure
Sevilla, 19 Feb. 2016 | 7
Principal Components Analysis (PCA) Expectations: Indicators grouped into a pillar and sub-indices grouped into an index
should be described by a single principal component (eigenvalue > 1) capturing most of
the variance in the underlying components
Results: One principal component for each sub-index capturing between 78-86% of the
total variance in the underlying indicators, and the three sub-indices sharing a single
latent dimension that captures 92% of the total variance in the overall index
Statistical coherence
Reliability Analysis (Cronbach-alpha) Expectations: Every aggregate in the IDI framework, be that sub-index or the overall IDI,
should be internally consistent (c-alpha > 0.7)
Results: High c-alpha values evidence that the indicators and sub-indices are measuring the
same underlying construct, and can be considered as statistically reliable aggregates (0.86-
0.91 for indicators, 0.95 when considering the 3 sub-indices)
Sevilla, 19 Feb. 2016 | 8
Cross-correlations (Pearson correlation coefficients) Expectations: Indicators should be more correlated to their own IDI sub-index than
to any other
Results: Indicators have been allocated to the most relevant ICT dimension
Relative importance of the IDI components (Pearson correlation ratio*) Expectations: As defined in the IDI framework, ICT access and ICT use dimensions
are more important than ICT skills (which is based on proxy indicators); the
underlying indicators within dimension are equally important
Results: Access and use dimensions are equally important and more important than
ICT skills; within dimension all indicators are important (> 0.5) in classifying countries
(although slightly unbalanced results in ICT access—dominated by two highly correlated
indicators: households w/computer & households w/Internet)
* The kernel estimates of the Pearson correlation ratio is a non-linear measure of the contribution to the variance of the relevant dimension scores
Statistical coherence
Sevilla, 19 Feb. 2016 | 9
Assessment of potential redundancy of information
Expectations: IDI should highlight aspects of ICT development that do not emerge
directly by looking into the 3 sub-dimensions (access, use and skills) separately
Results: Added value of IDI as a benchmarking tool confirmed by the fact that for
26% up to 52% of the countries sub-index rankings differ from the IDI ranking by 10
positions or more
Statistical coherence
Sevilla, 19 Feb. 2016 | 10
Impact of key modeling assumptions
Uncertainty/robustness analysis (simultaneous and joint impact of modeling choices on the rankings)
Expectations: Country classification based on IDI should depend mainly on the indicators used and not
on the methodological judgments made (e.g. weighting and aggregation choices)
Results: IDI rankings are robust to methodological changes relating to weighting and aggregation rule of
sub-index scores; results are based on a combination of Monte Carlo simulation (1,000 runs for
simulated weights of the three sub-indices) and multi-modeling approach (arithmetic vs. geometric
average)
IDI rank close to simulated median rank (90% countries ±3 positions)
In most cases (75% countries) simulated confidence intervals are narrow enough (less than 6 positions)
Only 3 countries with relatively wide intervals (>15 positions)
Sevilla, 19 Feb. 2016 | 11
As a result of the analyses undertaken, the three-level structure of the index is found to be statistically sound, coherent and balanced, robust to changes in the weights and aggregation rules, and a credible summary measure.
However, on statistical grounds, the index is and should remain open to refinements, such as considering the evolving conditions of ICT related variables (currently % households with a computer is highly correlated with % households with Internet access at home).
Overall assessment of ITU IDI 2015
Sevilla, 19 Feb. 2016 | 12
Cooperation already foreseen with DG CNECT (WP 2016-2017) on: 1. Validation of the EC Digital Economy and Society Index (DESI): (a) Is
the index multi-level structure statistically coherent? (b) What is the impact of modelling assumptions on the index rankings?
2. Benchmarking global and EU Digital-related indices: Compare the indicator frameworks underpinning ITU IDI and DESI—aiming to draw lessons for the improvement of the latter
Invited by NESTA to provide feedback on the European Digital City Index (currently under discussion)
COIN future work on ICT related issues
Sevilla, 19 Feb. 2016 | 13
ITU 2015. Measuring the Information Society Report. Geneva: ITU.
OECD/EC JRC. 2008. Handbook on Constructing Composite Indicators: Methodology
and User Guide. Paris: OECD.
Paruolo, P., Saisana, M., Saltelli, A. 2013. Ratings and rankings: Voodoo or Science?,
Journal of the Royal Statistical Society – A 176(3), 609-634.
Saisana, M., D’Hombres, B., Saltelli, A. 2011. Rickety numbers: Volatility of university
rankings and policy implications, Research Policy 40, 165-177.
Saisana, M., Saltelli, A., Tarantola, S., 2005. Uncertainty and sensitivity analysis
techniques as tools for the analysis and validation of composite indicators. Journal of
the Royal Statistical Society A 168(2), 307-323.
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M.,
Tarantola, S. 2008. Global Sensitivity Analysis: The Primer. Chichester, England: John
Wiley & Sons.
References
