1 cs 430 / info 430 information retrieval lecture 19 metadata 1

36
1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

Post on 19-Dec-2015

227 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

1

CS 430 / INFO 430 Information Retrieval

Lecture 19

Metadata 1

Page 2: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

2

Course Administration

Page 3: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

3

Descriptive Metadata

Some methods of information retrieval search and browse descriptive metadata about the objects.

Descriptive metadata typically consists of a catalog or indexing record, or an abstract, one record for each object. The record acts as a surrogate for the object.

• Usually the metadata is stored separately from the object that it describes, but sometimes is embedded in the object.

• Usually the metadata is a set of text fields.Textual metadata can be used to describe non-textual objects, e.g., software, images, music.

Page 4: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

4

Documents and Surrogates

The sea is calm to-night.

The tide is full, the moon lies fair

Upon the straits; -- on the French coast the light

Gleams and is gone; the cliffs of England stand,

Glimmering and vast, out in the tranquil bay.

Come to the window, sweet is the night-air!

Only, from the long line of spray

Where the sea meets the moon-blanch'd land,

Listen! you hear the grating roar

Of pebbles which the waves draw back, and fling,

At their return, up the high strand,

Begin, and cease, and then again begin,

With tremulous cadence slow, and bring

The eternal note of sadness in.

Author: Matthew Arnold

Title: Dover Beach

Genre: Poem

Date: 1851

Document

Notes:

1. The surrogate is also a document

2. Every word is different!

Surrogate (catalog record)

Page 5: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

5

Surrogates for Non-textual materials

See next page for a textual catalog record about a non-textual item (photograph).

Text based methods of information retrieval can search a surrogate for a photograph

Document Surrogate (catalog record)

Page 6: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

6

Library of Congress catalog record (part)

CREATED/PUBLISHED: [between 1925 and 1930?]

SUMMARY: U. S. President Calvin Coolidge sits at a desk and signs a photograph, probably in Denver, Colorado. A group of unidentified men look on.

NOTES: Title supplied by cataloger. Source: Morey Engle.

SUBJECTS: Coolidge, Calvin,--1872-1933. Presidents--United States--1920-1930. Autographing--Colorado--Denver--1920-1930. Denver (Colo.)--1920-1930. Photographic prints.

MEDIUM: 1 photoprint ; 21 x 26 cm. (8 x 10 in.)

Page 7: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

7

Categories of Descriptive Metadata

Catalog: metadata records that have a consistent structure, organized according to systematic rules. (Example: Library of Congress Catalog)

Abstract: a free text record that summarizes a longer document.

Indexing record: less formal than a catalog record, but more structured than a simple abstract. (Example: PubMed)

Page 8: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

8

Metadata Format

A metadata format is a set of rules that describe the content and format of a set of metadata records, e.g.:

• AACR (Anglo American Cataloging Rules) / MARC

• Dublin Core

• FGDC (Federal Geographic Data Committee's Content Standard for Digital Geospatial Metadata)

• IEEE Standard for Learning Object Metadata

Page 9: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

9

Uses of Metadata in Information Retrieval

Metadata is used in Information Retrieval systems in conjunction with or instead of full text indexing:

• For physical objects, e.g., books

• For non-textual materials, e.g., pictures, maps, datasets

• For specialized areas where high recall is important (e.g., medicine), or where features such as intended audience are hard to extract from the text (e.g., education)

• When people are ignorant of the power of full text indexing (which is surprisingly common)

Page 10: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

10

Uses of Metadata in Information Retrieval

Descriptive metadata provides capabilities that are not possible with full text indexing:

• Allows fielded searching

author = "Goethe"

• Suitable for non-textual material

type = "picture" and subject = "Ithaca"

• Can be used with controlled vocabulary

language = "en" (English)

Page 11: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

11

Information Retrieval with High Recall

Full-text Indexing (automated)

• Text only. Most effective on medium-length documents on related topics. High recall requires tuning system to the specific collection and skilled users.

Catalogs and Indexes (created manually)

• Can be used for all formats of material

• Requires close quality control of metadata creation

• High recall requires tuning system to the specific collection and skilled users.

Page 12: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

12

Using Metadata for Information Retrieval

The basic operation of information retrieval is to match the way that a user describes an information requirement (a query), against the way that items are described (an index).

The success of conventional catalogs (e.g., MARC + Anglo-American Cataloguing Rules) or indexing services (e.g., Medline) comes from the combination of:

• precise language to describe items

• trained and experienced users to formulate queries.

Page 13: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

13

Library Catalogs

Examples:

Cornell University Library catalog:

http://catalog.library.cornell.edu/

Library of Congress, Prints and Photographs:

http://www.loc.gov/rr/print/catalog.html

Page 14: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

14

Origins of Library Catalogs

Bibliographic Objective:

• To bring together like items

• To differentiate among similar ones

Sir Anthony Panizzi, Keeper of Books at the British Museum (1856-67).

His Ninety-One Rules (1841) were the basis of modern catalog rules.

Page 15: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

15

Origins of Library Catalogs

Information Discovery:

• to enable a person to find a book of which either the author, title or subject is known

• to show what the library has by a given author, on a given subject, or in a given kind of literature

• to assist in the choice of a book as to its edition (bibliographically) or to its character (literary or topical).

Charles Ammi CutterLibrarian of the Boston Athenaeum

Rules for a Dictionary Catalog, 1874

Page 16: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

16

Origins of Library Catalogs

Classification:

• Division of subject matter into a hierarchy.

• Typically used in libraries to provided a subject-based order for shelving books.

Melvil DeweyActing Librarian of Amherst College (1874)

Dewey Decimal system of book classification, uses the numbers 000 to 999

to cover the general fields of knowledge and decimals to fit special subjects.

Page 17: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

17

Library Catalogs: Technology Changes over the Years

Materials to be catalogued:

• Originally books

• Extended to serials, maps, music, etc., but concepts still rely heavily on experience with books

Form of catalog:

• Entries in books (Panizzi)

• Index cards (Cutter)

• Online databases (Kilgour)

Page 18: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

18

Shared Cataloguing: OCLC

OCLC -- Large centralized transaction processing database system (http://www.oclc.org/)

When a library catalogs a book it deposits MARC record in OCLC

Other libraries can copy the record

• saves duplication of cataloguing

• OCLC has a database of holdings from all libraries

OCLC database has 69 million records, serves 42,000 libraries

When developed by Fred Kilgour in 1967, OCLC was a pioneering computer system (had to develop own network, computer terminal, etc.)

Page 19: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

19

Catalogs as Investments

Costs:

• Conventional Catalog Records are created by skilled librarians. (cost estimate $100 per record).

• OCLC's catalog has 69 million records. Total investment is several billion dollars.

Cataloguing Standards:

• Enable libraries to share records

• Combine records of the past with records created today

• Allow readers and librarians to move between libraries

Page 20: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

20

Layers of a Library Catalog

Encoding

• Rules that define how catalog records are encoded in a computer system, e.g., XML mark-up.

Syntax

• Rules that define the fields and subfields, whether repeated, optional, etc.

Semantics

• Rules that define the values of the field and subfield, with instructions for cataloguers of what data to include and how to decide when choices have to be made.

Page 21: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

21

Library Cataloging using the Anglo American Cataloguing Rules

Anglo American Cataloguing Rules (AACR2)

• Rules for each category of material, e.g., monographs (books). Specify what fields should be used and what data to include in each field. Text strings were originally intended for printed catalog cards.

MARC format

• An exchange format for catalog records. Includes encoding rules and syntax specification.

"MARC Catalog"

• Catalog in MARC format, where content of each field follows AACR2.

Page 22: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

22

Anglo American Cataloguing Rules

The Anglo American Cataloguing (AACR) rules provide detailed rules for

• the choice of fields

• the content of the data that goes into each field

• the syntax of the data that goes into each field

The rules are an excellent example of technical writing: precise but clear. For an example, see:

http://www.cs.cornell.edu/Courses/cs430/2006fa/slides/AACR.pdf

Page 23: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

23

Name authority files

An Authority File "brings together like items and differentiates among similar ones."

• Caroline R. Arms or Caroline Ruth Arms?

• Which William Phillips of Cardiff?

• Mark Twain or Samuel Clemens?

• Epithets:

of Cardiffdoctor

• Dates:

1832 - 1876flourished 1860 circa 1832 - 1876

Page 24: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

24

LC Control Number: n 87870182 HEADING : Arms, Caroline R. (Caroline Ruth) 000 00907cz 2200205n 450 001 4383796 005 19890706143144.8 008 70909n|acannaab |a aaa c 010 __ |a n 87870182 035 __ |a (DLC)n 87870182 040 __ |a InU |c DLC |d DLC 100 10 |a Arms, Caroline R. |q (Caroline Ruth) 400 10 |w nna |a Arms, Caroline Ruth 400 10 |a Arms, C. R. |q (Caroline Ruth) 670 __ |a Arms, W.Y. Report on the performance problems of the

RLIN computer system, 1982: |b t.p. (Caroline R. Arms) 670 __ |a LC data base, 8/24/87 |b (hdg.: Arms, Caroline Ruth;

usage: Caroline R. Arms, C. R. Arms) 670 __ |a Campus networking strategies, 1988: |b CIP t.p.

(Caroline Arms) 670 __ |a Phone call to pub., 2/10/88 |b (Caroline Ruth Arms;

studied at Oxford) 670 __ |a Campus strategies for libraries and electronic

information, c1990: |b CIP t.p. (Caroline Arms) data sheet (b. 10-24-45)

953 __ |a bz46 |b bd24

Name authority: example

Page 25: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

25

Subject information

Library of Congress Subject Headings

Academic libraries--United States--Automation

Hierarchical classification

Library of Congress call number: Z675.U5C16

Dewey Decimal Classification: 027.7

Creation and maintenance of lists of subject headings and classifications is a never ending task.

Page 26: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

26

MARC Format

The MARC format was developed in the late 1960s as a tagging scheme for exchanging catalog records on magnetic tape. It remains the standard way to represent such data.

At present, MARC is steadily being converted (slowly) to modern computing formats, e.g., Unicode, XML.

Page 27: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

27

MARC: Monograph catalog record

Citation

Caroline R. Arms, editor, Campus strategies for libraries and electronic information. Bedford, MA: Digital Press, 1990.

Page 28: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

28

MARC fields

tag value

001 89-16879 r93

050 Z675.U5C16 1990

082 027.7/0973 20

245 Campus strategies for libraries and electronic title statement information/Caroline Arms, editor.

260 {Bedford, Mass.} : Digital Press, c1990. publisher

300 xi, 404 p. : ill. ; 24 cm. collation440 EDUCOM strategies series on information technology series title

504 Includes bibliographical references (p. {373}-381).

020 ISBN 1-55558-036-X : $34.95

Page 29: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

29

MARC fields (continued)

650 Academic libraries--United States--Automation. subject heading

650 Libraries and electronic publishing--United States.

650 Library information networks--United States.

650 Information technology--United States.

700 Arms, Caroline R. (Caroline Ruth)

040 DLC DLC DLC

043 n-us---

955 CIP ver. br02 to SL 02-26-90

985 APIF/MIG

Page 30: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

30

MARC Encoding

tag: 260

subfield a: {Bedford, Mass.} :

subfield b: Digital Press,

subfield c: c1990.

MARC encoding:

&2600#abc#{Bedford, Mass.} :#Digital Press,#c1990.%

[Definitely not a modern encoding!]

Note that the content is designed to be part of a printed catalog record and is not in a convenient format for computer manipulation.

Page 31: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

31

Modernizing MARC

1. Keep the content of the catalog record

2. Convert to Unicode for representing scripts

3. Convert to XML for tagging cataloguing metadata.

MARCXML (MARC 21 XML)http://www.loc.gov/standards/marcxml/

[Direct conversion to XML tagging]

Metadata Object Description Schema (MODS)http://www.loc.gov/standards/mods/

[Subset of MARC with data clean-up]

Page 32: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

32

MARC XML

• Simple and Flexible MARC XML Schema

The schema retains the semantics of MARC. Fields are treated as elements with the tag as an attribute and indicators treated as attributes. Subfields are treated as subelements with the subfield code as an attribute.

• Lossless Conversion of MARC to XML

• Roundtripability from XML back to MARC

• Data Presentation by writing a XML stylesheet

• Validation of MARC data

• Extensibility

Page 33: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

33

MODS Example (extracts)

<mods>

<titleInfo> <title>Sound and fury :</title> <subTitle>the making of the punditocracy /</subTitle></titleInfo>

<name type="personal"> <namePart>Alterman, Eric</namePart> <role> <roleTerm type="text">creator</roleTerm> </role></name>

Page 34: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

34

MODS Example (extracts)

<typeOfResource>text</typeOfResource>

<originInfo> <place> <placeTerm type="text">Ithaca, N.Y</placeTerm> </place> <publisher>Cornell University Press</publisher> <dateIssued>c1999</dateIssued></originInfo>

<language> <languageTerm authority="iso639-2b" type="code">eng</languageTerm></language>

</mods>

Page 35: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

35

Notes on MARC

A great achievement:

• Developed in 1960s

• Magnetic tape exchange format for printing catalog records

• The dawn of computing:

mixed upper and lower casevariable length fields, repeated fieldsnon-Roman scripts

• 100(?) million records with standard content and format

• Thousands of trained librarians (millions?)

Page 36: 1 CS 430 / INFO 430 Information Retrieval Lecture 19 Metadata 1

36

Notes on MARC

A great problem:

• Not designed for computer algorithms

• One record per item (poor links between records)

• Tied to traditional materials and traditional practices

• Not Unicode

• 100 of million records at $100 -- $10 billion

A classic legacy system!