Combining the strengths of UMIST andThe Victoria University of Manchester

Quality views: capturing and exploiting the user perspective on information quality

Paolo Missier, Suzanne Embury, Mark GreenwoodSchool of Computer Science, University of Manchester

Alun Preece, Binling JinDepartment of Computing Science, University of Aberdeen

www.qurator.orgDescribing the Quality of Curated e-Science Information

Resources

Combining the strengths of UMIST andThe Victoria University of Manchester

Outline• Information and information quality (IQ) in e-science

• Quality views: a quality lens on data

• Semantic model for IQ

• Architectural framework for quality views

• State of the project and current research

Combining the strengths of UMIST andThe Victoria University of Manchester

Information and quality in e-science

• Scientists are increasingly required to place more of their data in the public domain

• Scientists use other scientists' experimental results as part of their own work

In silico experiments(eg Workflow-based)

Lab experiment

In silico experiments(eg Workflow-based)

Public BioDBsPublic

BioDBsE-science experiment

Can I trust this data?What evidence do I

have that it is suitable for my experiment?

• Variations in the quality of the data being shared

• Scientists have no control over the quality of public data

• Lack of awareness on quality: difficult to measure and assess– No standards!

Combining the strengths of UMIST andThe Victoria University of Manchester

A concrete scenarioQualitative proteomics: identification of proteins in a cell sample

Step 1 Step nCandidate Data

for matching(peptides peak lists)

Match algorithm

Reference DBs- MSDB- NCBI- SwissProt/Uniprot

Wet lab

Information service (“Dry lab”)

Hit list:{ID, score, p-value,…}

False negatives: incompleteness of reference DBs, pessimistic matching

False positives: optimistic matching

Combining the strengths of UMIST andThe Victoria University of Manchester

The complete in silico workflow 1: identify proteins; 2: analyze their functions

What is the quality of this processor’s output?

Is the processor introducing noise in the flow?GO = Gene Ontology

Reference controlled vocabulary for describing protein function (and more)

How can a user rapidly test this and other hypotheses on quality?

Combining the strengths of UMIST andThe Victoria University of Manchester

The users’ perception of quality

Scientists often have only a blurry notionof their quality requirements for the data

“One size fits-all” approach to quality does not work– Scientists tend to apply personal acceptability criteria to data

– Driven mostly by prior personal and peers’ experience

– Based on the expected use of the data• What levels of false positives / negatives are acceptable?

It is difficult for users to implement quality criteria and test them on the data

Combining the strengths of UMIST andThe Victoria University of Manchester

Quality views: making quality explicitOur goals:

• To support groups of users within a (scientific) community in understanding information quality on specific data domains

• To foster reuse of quality definitions within the community

Approach:

• Provide a conceptual model and architectural framework to capture user preferences on data quality

• Let users populate the framework with custom definitions for indicators and personal decision criteria

– The framework allows uses to rapidly test quality preferences and observe their effect on the data

– Semi-automated integration in the data processing environment

Quality views:A specification of quality preferences and how they apply to the data

Combining the strengths of UMIST andThe Victoria University of Manchester

Basic elements of information quality

1 - Quality dimensions:

A basic set of generic definitions for well-known non-functional properties of the data• Ex. Accuracy: describes “how close the observed value is to the actual value”

2- Quality evidence:

• Any measurable quantities that can be used to express formal quality criteria

• Evidence is not by itself a measure of quality

Ex. “Hit ratio in protein identification”

3- Quality assertions:Decision procedures for data acceptability, based on available evidence

Combining the strengths of UMIST andThe Victoria University of Manchester

The nature of quality evidenceDirect evidence: indicators that represent some quality

property– Algorithms may exist to determine the biological plausibility of an

experiment’s outcome– may be costly, not always available, and possibly inconclusive

Indirect evidence: inexpensive indicators that correlate with other more expensive indicators– Eg some function of “hit ratio” and “sequence coverage”– Need experimental evidence of the correlation

Goals:design suitable functions to collect / compute evidenceassociate evidence to data (data quality annotation)

Combining the strengths of UMIST andThe Victoria University of Manchester

Generic (e-science) evidence• recency: how recently the experiment was performed, or

its results published– Evidence: submission, publication dates

• submitter reputation: is the lab well-known for its accuracy in carrying out this type of experiments– Metric: lab ranking (subjective)

• publications prestige: are the experiment results presented in high-profile journal publications– Metric: Impact Factor and more (official)

Collecting data provenance is the key to providing most of these types of evidence

Combining the strengths of UMIST andThe Victoria University of Manchester

Semantic model for Information Quality

The key IQ concepts are captured using an ontology:

• Provides shareable, formal definitions for– QualityProperties (“dimensions”)

– Quality Evidence– Quality Assertions– DataAnalysisTools: Describe how indicators are computed

• The ontology is implemented in OWL DL– Expressive operators for defining concepts and their relationships

– Support for subsumption reasoning

Combining the strengths of UMIST andThe Victoria University of Manchester

Domain-specificUser-orientedConcrete qualities

Wang and Strong, Beyond Accuracy: What Data Quality Means to Data Consumers, Journal of Management Information Systems, 1996

Top-level taxonomy of quality dimensions

Genericdimensions

Combining the strengths of UMIST andThe Victoria University of Manchester

Main taxonomies and properties

Class restriction:MassCoverage is-evidence-for . ImprintHitEntry

Class restriction:PIScoreClassifier assertion-based-on-evidence . MassPIScoreClassifier assertion-based-on-evidence . Coverage

assertion-based-on-evidence: QualityAssertion QualityEvidence

is-evidence-for: QualityEvidence DataEntity

Combining the strengths of UMIST andThe Victoria University of Manchester

Associating evidence to data• Annotation functions compute quality evidence values for

datasets and associate them to the data– Defined in the DataAnalysisTool taxonomy as part of the ontology

Combining the strengths of UMIST andThe Victoria University of Manchester

Quality assertions

Defined as ranking or classification functions f(D,I):Input: • dataset D

• vector I = [I1,I2,…In] of indicator valuesPossible outputs:

• A classification {(d,ci)} for each d D

• A ranking {(d,ri)} for each d DThe classification scheme C = {c1,..ck} and the ranking interval [r,R] are

themselves defined in the ontology

Assertions formalize the user’s bias on evidence as computable decision models on that evidence

Example:

PIScoreClassifier partitions the input dataset into three classes {low, avg, high} based on a function of [HitScore, MassCoverage]

Combining the strengths of UMIST andThe Victoria University of Manchester

Quality views in practiceQuality views are declarative specifications for:

• desired data classification models and evidence– I = [I1,I2,…In]

– classi(d), ranki(d) for all d D

• condition-action pairs, eg:• If <condition on class(d), rank(d), I> then <action>

• Where <action> depends on the data processing environment

– Filter out d

– Highlight d in a viewer

– Send d to a designated process or repository

– …

• Quality views are based on a small set of formal operators• They are expressed using an XML syntax

Combining the strengths of UMIST andThe Victoria University of Manchester

Execution model for Quality views• QVs can be embedded within specific data management

host environments for runtime execution– For static data: a query processor

– For dynamic data: a workflow engine

Host environment

Declarative(XML) QV

EmbeddedExecutable

QV

QV compiler

Dataset D

D’

Qurator quality framework- Quality assertion services

Quality view on D’

Combining the strengths of UMIST andThe Victoria University of Manchester

User model

Compose quality view

IQ ontologyCompile

and deploy

Execute on test data

AssessView results

(Update assertion models)

Quality assertion services

Re-deploy

bindings

(XML)

Implementing rapid testing of quality hypotheses:

Combining the strengths of UMIST andThe Victoria University of Manchester

The Qurator quality framework

Combining the strengths of UMIST andThe Victoria University of Manchester

Compiled quality workflow

Combining the strengths of UMIST andThe Victoria University of Manchester

Embedded quality workflow

Combining the strengths of UMIST andThe Victoria University of Manchester

Example effect of QV: noise reduction

Combining the strengths of UMIST andThe Victoria University of Manchester

Summary

• A conceptual model and architecture for capturing the user’s perception on information quality

– Formal, semantic model makes concepts• Shareable• Reusable• Machine-processable

• Quality views are user-defined and compiled to data processing environments (possibly multiple)

• The Qurator framework supports a runtime model for QVs

• Current work:– Formal semantics for QVs– Exploiting semantic technology to support the QV specification task– Addressing more real use cases

Main paradigm: let scientists experiment with quality concepts in an easy and intuitive way by observing the effect of their personal bias

Combining the strengths of UMIST andThe Victoria University of Manchester

Combining the strengths of UMIST andThe Victoria University of Manchester

Combining the strengths of UMIST andThe Victoria University of Manchester