metadata quality assurance framework at qqml2016 - short
TRANSCRIPT
Metadata Quality Assurance Framework
Péter Király <[email protected]>Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen, Germany
QQML2016 8th International Conference on Qualitative and Quantitative Methods in Libraries2016-05-24, London
2
Metadata Quality Assurance Framework
the problemthere are „good” and „bad” metadata
records
3
Metadata Quality Assurance Framework
Typical issues – non-informative field
Title is not informative
non informative:„photograph, framed”,„group photograph”„photograph”
vs
informative:„Photograph of Sir Dugald Clerk”,„Photograph of "Puffing Billy"
4
Metadata Quality Assurance Framework
Typical issues – Field overuse
What is the meaning of the field? (overuse)
TextGrid OAI-PMH response
5
Metadata Quality Assurance Framework
Why data quality is important?
„Fitness for purpose” (QA principle)
no metadata no access to data no data usage
more explanation:Data on the Web Best PracticesW3C Working Draft 19 May 2016https://www.w3.org/TR/dwbp/
6
Metadata Quality Assurance Framework
Europeana Data Quality Committee
Online collaboration Use case documents Problem catalog Tickets Discussion forum #EuropeanaDataQuality
Bi-weekly teleconf Bi-yearly face-to-face
meeting
Topics Usage scenarios Metadata profiles Schema modification Measuring Event model Proposals for data
providers
7
Metadata Quality Assurance Framework
What it is good for?
improve the metadata improve services: good data → functions improve metadata schema &
documentation propagate „good practice”
Domains: cultural heritage sector research data management and
archiving
8
Metadata Quality Assurance Framework
Research hypothesis
hypothesiswith measuring structural elements we
can predict metadata record quality
9
Metadata Quality Assurance Framework
Research hypothesis
proposed solutionan open source measuring and reporting
toolMetadata Quality Assurance Framework
10
Metadata Quality Assurance Framework
What to measure?
11
Metadata Quality Assurance Framework
Measurements
Schema-independent structural featuresexistence, cardinality, uniqueness,
length,dictionary entry, data type conformance
Use case scenarios („fit for purpose”)Requirements of the most important
functions
Problem catalogKnown metadata problems
12
Metadata Quality Assurance Framework
Discovery scenarios and their metadata requirements
Europeana’s most important functions
1. Basic retrieval with high precision and recall2. Cross-language recall3. Entity-based facets4. Date-based facets5. Improved language facets6. Browse by subjects and resource types7. Browse by agents8. Browse/Search by Event9. Entity-based knowledge cards and pages10. Categorised similar items11. Spatial search, browse, and map display12. Entity-based autocompletion13. Diversification of results14. Hierarchical search and facets
Credit: the document was initialized by Timothy Hill, Europeana’s search engineer
13
Metadata Quality Assurance Framework
Discovery scenarios and their metadata requirements – Entity-based facets
ScenarioAs a user I want to be able to filter by whether a person is the subject of a book, or its author, engraver, printer etc.
Metadata analysisIn each case the underlying requirement is that the relevant EDM fields for objects be populated by identifying URIs rather than free text. These URIs need to be related, at a minimum, to a label for each of the supported languages.
Measurement rules The relevant field values should be resolvable URI each URI should have labels in multiple languages
14
Metadata Quality Assurance Framework
Problem catalog
Catalog of known metadata problems in Europeana
Title contents same as description contents Systematic use of the same title Bad string: "empty" (and variants) Shelfmarks and other identifiers in fields Creator not an agent name Absurd geographical location Subject field used as description field Unicode U+FFFD ( )� Very short description field ...
Credit: the document was initialized by Timoty Hill, Europeana’s search engineer
15
Metadata Quality Assurance Framework
How to define measurements?
16
Metadata Quality Assurance Framework
Problem catalog – proposed basis of implementation
Shapes Constraint Language (SHACL)https://www.w3.org/TR/shacl/
A language for describing and constraining the contents of RDF graphs. It provides a high-level vocabulary to identify predicates and their associated cardinalities, datatypes and other constraints.
sh:equals, sh:notEquals sh:hasValue sh:in sh:lessThan, sh:lessThanOrEquals sh:minCount, sh:maxCount sh:minLength, sh:maxLength sh:pattern
17
Metadata Quality Assurance Framework
early measurement resultsand their visualization
18
Metadata Quality Assurance Framework
overall view collection view record view
Completeness – 40 measurementsField cardinality – 27 measurementsUniqueness – 6 measurementsLanguage specification – 20 measurementsProblem catalog – 3 measurementsetc.
links
measurementsaggregated numbers
19
Metadata Quality Assurance Framework
completenessWhat is the ratio of populated fields in
records?
20
Metadata Quality Assurance Framework
Field frequency / main
Alternative title is a rare field
21
Metadata Quality Assurance Framework
Field frequency per collections / all
no record has alternative title
every record has alternative title
22
Metadata Quality Assurance Framework
multilingualityDo we know the language of a field
value?
23
Metadata Quality Assurance Framework
Multilinguality
@resource is a URI
@ = language notation in RDF
no language specification
24
Metadata Quality Assurance Framework
Language frequency / barchart
25
Metadata Quality Assurance Framework
Language frequency / barchart
same language, different encodings
26
Metadata Quality Assurance Framework
Language frequency / Treemap with resources
has no language specification
has language specificationIs a URI
27
Metadata Quality Assurance Framework
uniqueness (entropy)How unique the terms are in a field?
28
Metadata Quality Assurance Framework
Entropy – term uniqueness / main
1 means a unique term0.0000x means a very frequent term
These are cumulative numbersentropycumolative = term1 + ... + termn
29
Metadata Quality Assurance Framework
Entropy – term uniqueness / collection
max is exceptional (=1425 * mean)
unique records
not or less unique records
30
Metadata Quality Assurance Framework
Entropy – term uniqueness / refining the picture
bulk of records are close to zero
although 25% are between 0.05 and 1.25
31
Metadata Quality Assurance Framework
Entropy – term uniqueness / terms
explanation of uniqueness score
TF-IDF values come from Apache Solr
term frequency: 1document freq.: 2uniqueness score: 0.5
32
Metadata Quality Assurance Framework
problem catalogDoes the record have any specific issues?
33
Metadata Quality Assurance Framework
Problem catalog – same title and description
there is one title and description which is the same
... and we have 9 such records
34
Metadata Quality Assurance Framework
Problem catalog – same title and description – example
35
Metadata Quality Assurance Framework
completeness sub-dimensionsAre the sub-dimensions (field groups supporting specific functionalities)
complete?
36
Metadata Quality Assurance Framework
Record view – functionality matrix
existing
missing
functionalities
37
Metadata Quality Assurance Framework
miscellaneous
38
Metadata Quality Assurance Framework
Further steps
Incorporating into Europeana’s ingestion tool Process usage statistics (logs, Google Analitics) Human evaluation of metadata quality Measuring timeliness (changes of scores over time) Machine learning based classification & clustering Incorporating into research data management tool Cooperation with other projects
39
Metadata Quality Assurance Framework
Architectural overview
Apache Spark (Java)
OAI-PMH client (PHP)
Analysis with Spark (Scala) Analysis with R
Web interface(PHP, d3.js)
Hadoop File System
JSON files
Apache Solr
Apache Cassandra
JSON filesJSON files image files
CSV files CSV files
recent workflowplanned workflow
40
Metadata Quality Assurance Framework
Follow me
Europeana Data Quality Committee http://pro.europeana.eu/europeana-tech/data-quality-committee
research plan and blog http://pkiraly.github.io
site http://144.76.218.178/europeana-qa/
source codes https://github.com/pkiraly/europeana-qa-spark https://github.com/pkiraly/europeana-qa-r
@kiru, https://www.linkedin.com/in/peterkiraly