the status of metadata standards and modernstats models in ... · big data admin. data address list...
TRANSCRIPT
![Page 1: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/1.jpg)
The status of metadata standards and ModernStats
models in SURS
ModernStats World Workshop 2018
SURS – Statistical Office of the Republic of Slovenia Julija Kutin
April, 2018
![Page 2: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/2.jpg)
Statistical Standards and ModernStats models in SURS
1. Statistical Standards: a) SDMX b) ISO/IEC 11179 c) DDI
2. ModernStats models: a) GSBPM b) CSPA
3. Challenges and next steps
![Page 3: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/3.jpg)
1.a. SDMX
1. SDMX and statistical data 2. SDMX and metadata 3. General problems with SDMX:
a) Different internal
rules for dissemination
b) Only for Eurostat c) Not standard tables
for indicators d) Different MSDs
![Page 4: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/4.jpg)
1.b. ISO/IEC 11179 – SDM module
![Page 5: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/5.jpg)
Variables and questions in Excel
![Page 6: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/6.jpg)
Usefulness of SDM module 1. E-questionnaires
a) Questions b) Sub-questions c) Tables – challenge
2. Response validation a) Mandatory questions b) Questions with specific values or rank of values c) Responses in a specific format
3. Build the database 4. Data editing
Local variables
![Page 7: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/7.jpg)
1.c. DDI
1. Mappings ISO/IEC 11179 to DDI 2. For generating questionnaires
![Page 8: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/8.jpg)
2.a. GSBPM
![Page 9: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/9.jpg)
REGISTRY
Analysis of needs and requesrt
Dat
a co
llect
ion
The
sele
ctio
n of
ob
serv
atio
n un
its
Dat
a pr
oces
sing
Dat
a an
alys
is
Use
rs
Survey design and preparation
Dat
a m
anag
emen
t
Big Data
Admin. data
SPRS Address list
Row data Maro
data
Final micro data ARCHIVE
D
ata
diss
emin
atio
n
Met
adat
a di
ss.
Admin. data. Admin. data
Metadata Classifications
Data files Data files Data files Data files
SRKG
SRDAP
PAPI
CATI
eSTAT
Adm.
Editing
DISSEM. DB
ESS MH
eDamis (Eurostat)
SDMX HUB
Si_Stat
Resear-ches
Other IO
Publi-cations
SDMX HUB
Si_Stat
CAPI
SOP
Integration
Imputation
Weighting
Agregation
Doc
umen
tatio
n
Qua
lity
asse
ssm
ent
Eval
uatio
n
Tabulation
Disclosure
SCRP
![Page 10: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/10.jpg)
Manage and document statistical production a) Documentary system for statistical surveys b) Descriptions of processes of the statistical survey c) Guidelines for Quality Assurance (QA)
Usefulness of GSBPM
![Page 11: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/11.jpg)
Documentation of statistical survey 1. Before:
a) Missed documentation b) Not structured documents c) In different places
2. A new system:
a) One place for all surveys b) Documents are structured (GSBPM) c) The information can not be lost d) Unauthorised access is not possible e) Comparability between surveys f) Preparation of documentation is planned part of the survey implementation
![Page 12: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/12.jpg)
System STATDOK is built at 3 levels
I. level
Basic information of the survey
Standardized Excel template
II. level
Description of the phase, sub-processes
7 standardized Word templates
III. level
Implementing documents
Standardized templates and not standardized documents
![Page 13: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/13.jpg)
I. level One sheet for each phase
STATDOK example
![Page 14: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/14.jpg)
II. level 7 templates - one for each sub-process
STATDOK example
![Page 15: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/15.jpg)
1. Methodologist
2. Head of the organizational unit
3. Process / sub-process administrator and leadership
How did we do it? How should it be done? How is it done in other surveys? Do we have data for 19XX? Are they comparable? Where can we find them?
Analysis of documentation Education of co-workers Optimisations
Standardization Similarity Personal data Critical points
…
STATDOK is usefull
![Page 16: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/16.jpg)
2.b. CSPA ESTAT questionnaire generation
1. ESTAT web data collection portal 2. Metadata driven questionnaire generation 3. ISO/IEC 11179 metadata model
a) Concepts, codes/representations, categories, etc., are similar to
DDI concepts.
![Page 17: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/17.jpg)
Generate questionnaire
Manage survey data collection
SDM Meta-data
ISO/IEC 11179
SDM Questionna-ire generator
SDM Questionna-ire creator
Blaise
XSD
ESTAT ENO reuse context
ENO Questionna-ire generator
Pogues Questionna-ire designer
DDI DDI Xforms
Blaise MD
Orbeon MD
Blaise manual editing Design
questionnaire Web Questio-nnaire Blaise
Web Questio-nnaire
Orbeon
Excel form
Colle-cted data
Oracle
WS
WS
WS
DDI
![Page 18: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/18.jpg)
4. Challenges and next steps 1. Go on with the SDMX standard 2. Meet with DDI deeper 3. Review of internal questionnaire design standard
(complex questionnaires generation) 4. Rebuild SDM module 5. Use ENO DDI compliant SDM as common metadata
repository for questionnaire design & generation (Blaise, Xform, Pdf, etc.)
6. GSBPM will be the basis of our further development 7. The use of CSPA standard should be considered for the specification and development of services
![Page 19: The status of metadata standards and ModernStats models in ... · Big Data Admin. data Address list Row data Maro data Final micro data ARCHIVE. Data dissemination Metadata ESS diss](https://reader033.vdocuments.site/reader033/viewer/2022043011/5fa700bf531bda55915b739c/html5/thumbnails/19.jpg)
Thank you for your attention!