the data deluge: what does it mean for official statistics?

United Nations Economic Commission for EuropeStatistical DivisionUnited Nations Economic Commission for EuropeStatistical Division

The Data Deluge:What Does It Mean for Official

Statistics?

Steven ValeUNECE

steven.vale@unece.org

Contents

What is the Data Deluge? Response of the official statistics

community Industrialisation – a related challenge Changing roles for national statistical

organisations

What is the Data Deluge?

The internet

has 1800

exabytes of

data in 2011

exa = 10^18

50,000 exabytes by 2020

We live inexponential

times!

27 foldgrowth in the next9 years

Are these data interesting?

Probably 99.9% are videos, photos, audio files, text messages and other nonsense

But that still leaves

1,800,000,000,000,000,000

bytes of potentially relevant data

An Observation

More and more people post information about themselves on Facebook, but less and less complete statistical surveys

Should we create

STATVILLE?

Private sector competitors?

Google:• Real-time price indices• Public Data Explorer• First point of reference for the “data generation”

Facebook, store cards, credit agencies, ...• What if they link their data?

The private sector now understands the value of data:

Can they beat us at our own game?

High-Level Group for Strategic Directions in Business Architecture in Statistics

UNECE group, created by the Conference of European Statisticians in 2010

8 heads of national and international statistical organisations

Develop and promote new:

Sources Processes Products

Official Statistics Response

Endorsed by the Conference of European Statisticians on 14 June

We have to re-invent our products and processes and adapt to a changed

world

HLG-BAS Strategic Vision

The Challenges are too big for statistical organisations

to tackle on their own.

We need to work together

What does this mean in practice?

Collaboration

Coordination

Communication

Many international groups and projects are talking about streamlining and industrialising statistics

Industrialisation is:

Common processes Common tools Common methodologies Recognising that all statistics are

produced in a similar way:

No domain is “special” Increased flexibility to adapt to new

sources and produce new outputs

Changing roles in NSOs?

One source = one output Data integration – multiple sources Process quality assurance More focus on analysis and interpretation Partnerships for dissemination

Changing staff and cost profiles Changing organisational culture

Wider Definition of Admin Data?

New sources are “non-statistical” But - similar issues to “traditional”

administrative data sources

Whatever we call the new sources,we can’t ignore them!

Which new sources are you going to check

out?

Questions?

steven.vale@unece.org

www1.unece.org/stat/platform/display/hlgbas