Scanner Data – next steps

ahead

Berthold Feldmann – Eurostat

Price statistics; purchasing power parities; housing statistics

Scanner Data Workshop – Stockholm June 20122

Structure of the talk

Scanner data – a new paradigm

Main lessons of the last two days

Challenges

Time table and conclusion

Chapter 1

A new universe

Scanner Data Workshop – Stockholm June 20124

Scanner data as source for consumer price indices

Expected advantages

Better quality of indices Low response burden of enterprises Productivity gains of National Statistical Institutes Data can be used for other purposes

Purchasing Power Parities Average price levels Advanced economic analysis (e.g. substitution

rates) Regional breakdown of CPI/HICP

Scanner Data Workshop – Stockholm June 20125

Scanner data as source for consumer price indices

Expected disadvantages

Potential biases in CPI / HICP

High investment costs (IT)

Some rules of European Implementing Regulations may no longer be applicable Treatment of rebates

Quality adjustment

Direct collection of “price observations” Together we have to check if rules laid

down in legal acts need an adaptation

Chapter 2

Lessons learnt in the Workshop

Scanner Data Workshop – Stockholm June 20127

Some general conclusions

We are still at an early stage in the use of scanner data as a data source

Several problems still need to be solved

Scanner data appears quite promising for the collection of Multipurpose Consumer Price Statistics

Scanner data cannot be used for all COICOP groups

Scanner data offers high quality information of actual transactions

Scanner Data Workshop – Stockholm June 20128

Conclusions on discounts

Discounts are an important stimulus for increased sales important for consumers

Scanner data offers new opportunities for a more precise treatment of discounts compared to traditional price collection

For HICP – the inclusion of all non-discriminatory price reductions should be the rule

Precise recommendations should be drafted a.s.a.p.

We might need to evaluate the existing regulatory framework on price reductions

Scanner Data Workshop – Stockholm June 20129

Practical experience The exchange of views and experiences between

NSIs could save a lot of time

Avoid reinventing the wheel

Full support from the top management of retailers is mandatory

Good collaboration with data providers is a key point

More volatility and some short and medium term differences when changing the price concept (transaction price versus shelf price)

Major challenge: high attrition rate (up to 30%)

Promising results with the use of “internal” classifications as links between EAN and COICOP

Scanner Data Workshop – Stockholm June 201210

Computation methods

From GEKS to RYGEKS

From RYGEKS to imputed Tornquist RYGEKS

Three different methods are tested for the imputation of prices for new and disappearing items

Conclusion

at the lowest level of computation we might need to go beyond Laspeyres ... ...

Mainly used for benchmarking

Chapter 3

Challenges

Scanner Data Workshop – Stockholm June 201212

Link EAN - COICOP

EAN differ from country to country

A high proportion of EAN have a life span of less than a year

This results in a high turnover rate of product types

Automating the link of EAN to COICOP level 5 or 6 is not a trivial task

In the long run a harmonisation of EAN across Europe would be desirable

Scanner Data Workshop – Stockholm June 201213

Use a sample or all data?

Current practise differs between countries

A harmonised approach should be achieved

Advantages and disadvantages of the two approaches should be discussed carefully

Using a sample is closer to the traditional price observations

Using all data might offer new opportunities

Scanner Data Workshop – Stockholm June 201214

Automation of editing EAN data

Problem of many products disappearing and others emerging within a year

Practice of some advanced National Statistical Offices can give guidelines

Scanner Data Workshop – Stockholm June 201215

Weighted or not weighted

average?

(product level) Should quantity information be used at basic calculations?

Quantities may show huge fluctuations

A Europe-wide recommendation should be achieved

Scanner Data Workshop – Stockholm June 201216

Scanner data as a source for

weights

(level of product groups) Quantities from scanner data could be used as weights at the lowest level (COICOP 5 or 6)

In any case, only annual averages should be used as weights

According to Implementing Regulation no changing weights from month to month

Volatility of the data?

Scanner Data Workshop – Stockholm June 201217

Further information collected

Scanner data contains more than prices and quantities

Product description is important for EAN – COICOP link

Information on discounts may be useful for further data analysis for economic purposes

Scanner Data Workshop – Stockholm June 201218

Scanner data versus price collectors Different results are reported from the two

alternative sources for consumer price indices

A thorough analysis is required to explain these differences

Algorithms to be found for possible bias in scanner data

Scanner Data Workshop – Stockholm June 201219

Legal obligations

National law may set different obligations in different countries

Are stores obliged to supply scanner data?

Can European legislation help?

COUNCIL REGULATION (EC) No 2166/1999The statistical units … called upon by the Member States to cooperate in the collection or provision of basic information are obliged to give honest and complete information at the time it is requested and allow the organisations and institutions responsible for compiling official statistics, on request, to obtain information at the level of detail necessary ………

Scanner Data Workshop – Stockholm June 201220

IT environment

High investment costs to start the regular usage of scanner data

Economies of scale possible if similar solutions are used in different countries

Chapter 4

Provisional time table

Scanner Data Workshop – Stockholm June 201222

A proposal for a timetable Continue intensive discussion among experts

Enhance transparency on current practice in all European countries

Eurostat will create an overview of current practice in summer 2012

Discussion at the Working Group meeting in October 2012

Meeting of the Task Force “Multipurpose price statistics” shall draft first recommendations in January 2013

Incorporation of basic rules on the use of scanner data in the new HICP Regulation

Scanner Data Workshop – Stockholm June 201223

Conclusion

The use of scanner data offers several advantages for future price statistics (not only consumer price indices)

Some issue connected with the use of scanner data are not easy to solve and require a harmonised approach

The European legal framework might need to be assessed if some rules need an adaptation for the use of scanner data

Thank you for

your attention!

Any Questions ?