IR & Metadata

Metadata

Didn’t we already talk about this?We discussed what metadata is and its

types– Data about data– Descriptive metadata is external to meaning

of content– Semantic metadata is related to content

How is it created?– Catalogers, authors, data entry, etc.– Requires lots of human effort

Automating Metadata

Can some metadata be assigned automatically?– Yes, depending on how willing you are to live

with mistakes– But humans also make mistakes …

How to determine metadata values?– Natural language processing– Pattern matching– Term/phrase recognition– Information retrieval

Natural Language Processing

Use rules of sentence construction (grammar) to “understand” the meaning of the text.

Difficulties– Grammar is not from grammar school– Human communication requires non-literal

interpretation

What types of metadata fields could NLP provide?

Example: weather forecasts

Pattern Matching

Use patterns (e.g. regular expressions) to locate and interpret specific forms of meaning

Difficulties– Patterns must be expressible in pattern

language– Lots of variations require lots of patterns– Polysemy

What types of metadata fields could pattern matching provide?

Term/Phrase Matching

Look for specific terms or phrases in order to determine document characteristics

Difficulties– No understanding of context– Polysemy

What types of metadata fields could term/phrase matching provide?

Information Retrieval

Use statistical analysis of vocabulary use and document structure to determine document characteristics

Difficulties– No understanding of terms– No understanding of semantic context

What types of metadata fields could information retrieval provide?

Practical Metadata

No metadata extraction algorithm works 100% of the time– Could send results to human to okay

• Still requires lots of human resources

– Need to decide how good algorithm has to be or how sure the algorithm is if it provides confidence values before accepting results

INFOMINE– Project crawling and generating metadata for

scholarly resources on the Web– Has 100,000 automatically created records

Types of Metadata Extraction

Assignment– Assigns values drawn from text of the

document– NLP, pattern matching, term/phrase matching

Classification– Assigns values from a controlled vocabulary– Use machine learning during training stage to

match document attributes (e.g. term vector) to element in controlled vocabulary

Evaluating Metadata Extraction

Automatic evaluation– Based on document set with human-expert

previously assigned metadata– Compare similarity between system-assigned and

human-assigned metadata– Limited to document/metadata sets where the values

are known

Human evaluation– Subject experts rate the appropriateness of the

assigned metadata– Allows for near misses and alternate values– Expensive to do

Metadata Extraction Metrics

Single-value metadata fields– Accuracy is a good performance measure– Partial match fields

• Parent or child in ontological hierarchies

Multi-value metadata fields– Precision = # right / # assigned– Recall = # right / # of expert-assigned values

Semantic summaries and keyphrases– Content-word precision = # same words / # words– Content-word recall = # same words / # expert words– Requires stopword and stemming

INFOMINE Assignments

Title– Single value open text field– Title tag worked well

Creator– Multiple value field– Used “creator” meta tag if there (good

precision, no smarts)

Keyphrase– Used “keyword” meta tag with PhraseRate

(IR approach)

INFOMINE Assignments

Description– 1-2 paragraphs long– Meta tags and AutoAnnotator (NLP + IR approach)

LCSH– Selected from over 200,000 values– Determines nearest neighbor in human-assigned

data set (IR and ML)

INFOMINE Category– Put document in set of nine categories– Uses nine binary classifiers created using ML

Summary

Metadata is useful but expensive– Lots of human effort to generate– Need to automate when possible

Metadata generation– NLP, pattern matching, term/phrase matching, IR– Approaches appropriate for generating different types

of metadata

Evaluating generated metadata– Automatic vs. human evaluation– Accuracy, precision/recall, etc.