applying process indicators to monitor the editing process
DESCRIPTION
Applying Process Indicators to Monitor the Editing Process. Purpose of Process I ndicators. Monitor and improve the editing process Find measurement errors and their causes and prevent them in future survey rounds Communicate quality to the users . International Trade in Services. - PowerPoint PPT PresentationTRANSCRIPT
Applying Process Indicators to Monitor the Editing Process
Purpose of Process Indicators
• Monitor and improve the editing process
• Find measurement errors and their causes and prevent them in future survey rounds
• Communicate quality to the users
International Trade in ServicesStatistics on international trade in services, wages and transfers are based on a quarterly sample survey involving about 5 000 enterprises and organizations.
Report values of income and expenditure for about 100 service types.
Editing in the survey• Selekt is used• Fatal errors are always followed-up• Suspicious errors are used to create a local score
for each service type based on potential impact and suspicion
• Local scores are aggregated to a global score for each enterprise
• A threshold determines whether the enterprise needs to be followed-up or not
Descriptive information
2013Q1 2013Q2 2013Q3Total number of observations 4041 3876 3890Number of observations failing at least one edit rule 438 297 403Number of observations failing at least one edit rule and adjusted 258 187 168
Expenditure Income 2013Q1 2013Q2 2013Q3 2013Q1 2013Q2 2013Q3Weighted raw value sum 373 433 302 457 694 730
Weighted raw value sum, failing at least one edit rule
319 373 194 368 616 538
Weighted edited value sum 129 130 151 216 214 297
Overall failure rate, 2013Q1-2013Q3
2013Q1 2013Q2 2013Q30
0.02
0.04
0.06
0.08
0.1
0.12
I1 Overall failure rate. Rate of observations failing at least one edit rule.
Overall hit rate, 2013Q1-2013Q3
2013Q1 2013Q2 2013Q30
0.1
0.2
0.3
0.4
0.5
0.6
0.7
I2 Overall hit rate. Rate of observations failing at least one edit rule and adjusted.
Weighted reject ratio, 2013Q1-2013Q3
2013Q1 2013Q2 2013Q30
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Income Expenditure
I3 Weighted reject ratio.
Failure rate and hit rate for each edit, 2013Q1-2013Q3
I4
I5
Failure rate for each edit FR.
Hit rate for each edit HR
Edit rule 2013Q1 2013Q2 2013Q3 FR HR FR HR FR HRQ03 5.05% 54% 0.77% 60% 1.93% 91%Q07 0.12% 20% 0.08% 67% 0.08% 67%Q08 6.14% 67% 6.22% 65% 5.66% 53%Q09 5.32% 68% 5.08% 64% 4.50% 50%Q10 0.40% 63% 0.34% 23% 0.39% 60%Q11 0.89% 53% 0.88% 24% 1.05% 29%Q12 0.74% 77% 0.80% 77% 0.64% 76%Q14 2.13% 72% 2.66% 67% 2.24% 64%Q15 0.20% 100% 0.03% 100% 0.10% 100%Q16 0.64% 85% 1.86% 89% 0.80% 87%Q19 0.47% 95% 0.80% 84% 0.72% 89%Q20 0.17% 57% 0.18% 29% 0.10% 50%
Editing rate for each variable, 2013Q1-2013Q3
I6 Editing rate for each variable.
Expenditure Income 2013Q1 2013Q2 2013Q3 2013Q1 2013Q2 2013Q3
410 Computer services 0.8% 0.4% 0.4%
463 Other services between affiliated enterprises 0.4% 0.4% 0.3%
442 Architectural, engineering and other technical services 0.7% 0.5% 0.4%
143 Goods freight by road 0.3% 0.3% 0.3%
Problems• Tedious to calculate• Difficult to interpret• Often small amounts of data
• Variable specific indicators depend on random fluctuations
Data availability• Dependent on available data • Some thoughts on which indicators we want
should be considered when setting up the system• Structure of data influences difficulty to generate
indicators• A system that generates standard reports could be
beneficial
Conclusion• Standardized indicators could be beneficial• Needs guidance on how to interpret and use them • Will take time to establish indicators in the
organization and within the management teams - What possibilities do we have to spread them to the organization?
Questions• Do you use indicators frequently?• Are they established in your organization?• Have you seen any improvements due to the use
of indicators? • Are standardized indicators a good idea?