analysis of inter-annotator agreement (text mining & reg. annotation) regcreative jamboree ,...

ANALYSIS OF INTER-ANNOTATORAGREEMENT

(TEXT MINING & REG. ANNOTATION)RegCreative Jamboree ,

Friday, December, 1st, (2006)

MARTIN KRALLINGER, 2006MARTIN KRALLINGER, 2006

TEXT MINING & REG. ANNOTATION

MAIN ASPECTSExplore annotation overlap

Discuss variability in annotation

Text mining and regulatory element annotation: needs, limits, tasks



SOCIOLOGY OF GENOME ANNOTATION(Lincoln Stein 2001)


Models of annotation

Museum model: small group of specialized curators

Jamboree model: a group of biologists and bioinformaticians come together for a short intensive annotation workshop

Cottage industry: decentralized effort of annotators among therecruited community

Factory model: highly automated methods

(Elsik et al, 2006)



WHY PRE-JAMBOREE QUEUE? Get familiar with annotation system (before jamboree)!

Understand content and annotation strategy of Oreganno

Detect aspects which require improvements such as incompleteness, ambiguity or wrong structures in annotation strategy, guidelines or documentation -> active Feedback (Questionnaire and wiki)

Assess consistency of the current annotation procedures

Explore which aspects affect annotation agreement

Estimate difficulty of task (alternative interpretation, uncertainty, etc,..)


SIMILARITY MEASURES


Similarity calculation popular subject in computer science Different entities considered: Feature vectors: Alignment, Cosine, Dice, Euclidean, … Strings or sequences of strings (text): averaged String Matching, TFIDF Sets: Jaccard, Loss of Information, Resembalance Sequences: Levensthein Edit Distance Trees:Bottom-up/Top-down Maximum Common Subtree, Tree Edit Distance Graphs: Conceptual Similarity, Graph Isomorphism, Subgraph Isomorphism, Maximum Common Subgraph Isomorphism, Graph Isomorphism Covering, Shortest Path Information theory: Jiang & Conrath, Lin, Resnik

Bioinformatics: sequence similarity, structural similarity, similarity of gene expression Here similarity between human annotations

( refer to SimPack project examples)


MEASUREMENT OF OBSERVER AGREEMENT


Assumption when independent annotators agree they are correct?! Statistical agreement measures for categorical data Overall proportion of agreement Pairwise comparison; Cohen’s kappa; Pearson Chi-square Weighted kappa for multiple categories High accuracy implies high agreement Kappa sometimes is inconsistent with accuracy measured as AROC

Measurement of Observer agreement Kundel andPolansky, Statistical concepts Series (2003)

Kappa coefficient


ANNOTATOR AGREEMENT FOR WSD

MARTIN KRALLINGER, 2006MARTIN KRALLINGER, 2006 A case Study on Inter-Annotator Agreement for WordSense Disambiguation, Ng et al

Word Sense Disambiguation (WSD) a central problem in NLP WSD: discerning the meaning of a word in context Two human annotators may disagree in their sense assignment Agreement of human annotators often the baseline for evaluation of automated approaches Case study using more than 30,000 instances of the most frequently occurring nouns and verbs in English Sense tagged word in sentences manually by two groups of annotator to WordNet Used the Kappa score to measure inter-annotator agreement considering effect of chance agreement Difficult to achieve high agreement when they have to assign refined sense tags Importance of example sentences for the usage of each word sense


AGREEMENT OF SPEECH CORPORA


Phonetically annotated speech corpora Quality of manual annotations affected by:

Implicit incoherence: labeling incoherent due to human variability in perceptual capacities and other factors

Lack of consensus on coding schema: manual annotationsreflect the variability of the interpretation and application ofthe coding schema by the annotators

Annotator characteristics: individual characteristics of coderssuch as familiarity with the material, amount of former training,motivation, interest and fatigue induced errors

Measuring the reliability of Manual annotations ofSpeech corpora, Gut and Bayerl


CHALLENGES FOR OREGANNO ANNOTATION


Complexity of gene regulation Need of ontologies and lexical resources Deep inference of domain expert curators Spatial, temporal, experimental conditions Range of entity types: genes, regulatory sequences, proteins Gene family and individual gene member distinction TF binding site sequence extraction and mapping to genome TF mapping to normalized database entries (NCBI, Ensembl) Archeology-like annotation: annotation of old papers BUT GENE REGULATION IS ONE OF THE MAIN BIOLOGICAL INFORMATION (ANNOTATION) ASPECTS!


SOURCES FOR ANNOTATION VARIABILITY (1)


Curator background (biologist, bioinformatician,...)

Familiarity with the annotation system

Number of previously annotated papers or proteins

Prior knowledge on the regulated gene or TF

Prior knowledge (experience) on the experimental types

Sub-domain knowledge (e.g. developmental biology or OS)

Publication date (reflect the state of knowledge)


SOURCES FOR ANNOTATION VARIABILITY (2)


Nr. of papers annotated the same day (fatigue effect)

Unclear or partial documentation of certain annotation aspects

Annotation type (ontology of annotation types?, CV?)

Nr. of pages, figures, tables, references,…

Consultation of additional resources (material, databases, web)

Different degrees of granularity in annotation

Differences in the recall of manually extracted annotations (all ?)

Sequence (paper/database, strand, typos, length)


REGCREATIVE CASE STUDY: PREJAMBOREE (1)


Relatively few articles -> only exploratory examination

Annotation type: 9/11 (2071609, 10674400: RR vs. TFBS)

Considerable difference in average nr. of annotations/paper

Some only extracted a single annotation others basically every annotation mentioned in the paper

Almost perfect agreement in organism source (1 case of human and mouse disagreement), but genes correct!

Very high agreement on the gene names, only few user defined cases (which are difficult to evaluate)


REGCREATIVE CASE STUDY: PREJAMBOREE (2)


Certain disagreement in TF names, many are user defined!

Evidence class: high agreement many Transcription regulator site, and unknown

Evidence type: high agreement, some more complete than others, (again, some annotate all the types others only some of them)

Evidence sub-type: similar to evidence types, but in generala little lower agreement than for the evidence type.


Transcription names factor: PREJAMBOREE


User defined

NCBI

Ensembl

Unkown



Example case 1: TF annotation variance


7534794

Curator B UNKNOWN USER DEFINED Curator B AP-1 USER DEFINED Curator B AP-1 USER DEFINED Curator A c-Rel/p65 heterodimer USER DEFINED Curator A UNKNOWN USER DEFINED Curator A UNKNOWN USER DEFINED

A

B


Example case 2: TF annotation variance


1718972

Curator A Tcf1 NCBI Curator B Tcf1 NCBI Curator B C\EBP family USER DEFINED Curator B C\EBP and NF-1 USER DEFINED Curator B Tcf1 NCBI Curator B UNKNOWN USER DEFINED Curator B UNKNOWN USER DEFINED Curator B UNKNOWN USER DEFINED


Example case: difference in evidence types

A

B

AB

10674400

Curator A REGULATORY REGION Curator B TRANSCRIPTION FACTOR BINDING SITE Curator B TRANSCRIPTION FACTOR BINDING SITE

2071609

Curator A REGULATORY REGION Curator B TRANSCRIPTION FACTOR BINDING SITE Curator B TRANSCRIPTION FACTOR BINDING SITE



3038906

Curator A TRANSCRIPTION FACTOR BINDING SITE Col1a2 UNKNOWN

TCCAAACTTGGCAAGGGCGAGA

CLASS:OREGEC00001 TYPE: OREGET00003 SUBTYPE:OREGES00015CLASS:OREGEC00001 TYPE: OREGET00001 SUBTYPE:OREGES00003

Curator B 1->TRANSCRIPTION FACTOR BINDING SITE Col1a2 Nfia

TTCCAAACTTGGCAAGGGCGAGAGAGGGCGA

CLASS:OREGEC00001 TYPE: OREGET00003 SUBTYPE:OREGES00033CLASS:OREGEC00001 TYPE: OREGET00003 SUBTYPE:OREGES00015CLASS:OREGEC00002 TYPE: OREGET00001 SUBTYPE:OREGES00003CLASS:OREGEC00002 TYPE: OREGET00001 SUBTYPE:OREGES00003

Different amount of annotation extracted

A

B

A

B


REGCREATIVE CASE STUDY: JAMBOREE


Intensive annotation strategy: face to face with other curators and expert annotators Get direct feedback and provide suggestions Promote integration of additional aspects in the annotation structure as well as annotated information types Populate the database with new annotation records Explore efficient curation training strategies Create Gold Standard collection of annotation records, maybe useful to allow example-based annotation training/evaluation Explore demands of biologists / curators to text mining community - > where it would be useful


REGCREATIVE CASE STUDY: POST-JAMBOREE


Monitor improvements in the annotation consistency

Allow consistent community-based annotation

Promote integration of additional aspects in the annotation structure as well as annotated information types

Increase efficiency in populating the database

Construction for text mining training collection


ANNOTATION CONSISTENCY



For selection as relevant paper for curation For the evidence class For the evidence types For the evidence subtypes For the regulated genes For the transcription factors For cell types How to structure comments Other aspects: ...

TEXT MINING TASKS FOR GENE REGULATION EXTRACTION



Detection of relevant articles: abstracts or full text Extraction of ranked list of regulated genes: mention

or normalized gene (database entries) Extraction of ranked list if TF Extraction of ranked list of evidence type IDs

together with name and text passage (sentence) Extraction of ranked associations between these

genes and TF Extraction of associations to other controlled

vocabularies or ontologies

analysis of inter-annotator agreement (text mining & reg. annotation) regcreative jamboree ,...

Documents

annotation agreement

annotationtext mining

annotation system

wsd martin krallinger

kappa coefficienttext

tasksmartin krallinger

annotationmartin krallinger

annotationannotator