open archives initiative
DESCRIPTION
Open Archives Initiative. Where we are, Where we are going. Carl Lagoze 4 th OAF Workshop September, 2003. Where we are now. De facto standard for Internet information exchange Deployed extensively and internationally (digital) libraries Museums Eprint repositories Research projects. - PowerPoint PPT PresentationTRANSCRIPT
Open Archives Initiative
Where we are,
Where we are going
Carl Lagoze4th OAF WorkshopSeptember, 2003
Where we are now
• De facto standard for Internet information exchange
• Deployed extensively and internationally– (digital) libraries– Museums– Eprint repositories– Research projects
Protocol Stability
• OAI-PMH has been stable since release– No functional changes, just typographic edits– Validation of leadership/participation model
• No plans for a 3.0 release– Core protocol will not be extended– Minor 2.x release could occur (more later)– Additional implementation guidelines (more
later)
NSDL and OAI-PMH
The NSDL Context
• National STEM (Science, Technology, Engineering, Mathematics, Medicine) Digital Library
• Major National Science Foundation project targeted at the application of web and Internet to (STEM) education
• $25M over six years to over 100 projects– Collections– Services– Targeted Research– Core Integration
• Aggregation rather than collection– Core integration team will not manage any collections
• Spectrum of interoperability– Accommodate diversity of participation models– Open interfaces and standards permitting plug in of
array of value-added services• One library many portals
– Accommodate multiple quality and selection metrics– Tailor presentation of content and nature of services
to audience needs
NSDL technical guidelines
Level Agreements Example
Federation Strict use of standards AACR, MARC(syntax, semantic, Z 39.50and business)
Harvesting Digital libraries expose Open Archivesmetadata; simple metadata harvesting
protocol and registry
Gathering Digital libraries do not Web crawlerscooperate; services must and search enginesseek out information
Spectrum of interoperability
• This is a big task that no one has done before!• Work on the priorities
– Focus on one point on spectrum of interoperability• Metadata harvesting• Incorporate NSF funded collections and selected other collections
– Leverage existing (or at least emerging) technologies and protocols• OAI, uPortal, Shibboleth, SDLIP, InQuery
– Provide reliable base level services• Search and Discovery, Access Management, User Profiles, Exemplary
Portals, Persistence• Plant some seeds for the future
– Machine-assisted metadata generation– Automated collection aggregation– Web gathering strategies
Translating to initial goals
• Central storage of all metadata about all resources in the NSDL– Defines the extent of NSDL collection– Metadata includes collections, items, annotations, etc.
• MR main functions– Aggregation– Normalization– redistribution
• Ingest of metadata by various means– Harvesting, manual, automatic, cross-walking
• Open access to MR contents for service builders via OAI-PMH
Metadata Repository
Metadata Strategy
• Collect and redistribute any native (XML) metadata format
• Provide crosswalks to Dublin Core from standard formats – DC-GEM, LTSC (IMS), ADL (SCORM),
MARC, FGCD, EAD
• Concentrate on collection-level metadata• Use automatic generation to augment
item-level metadata
Importing metadata into the MR
Collections
Harvest
Staging area
Cleanup and
crosswalks
Database load
Metadata Repository
Exporting metadata from the MR
NSDL services
Create OAI server tables
Metadata Repository
SQL queries OAI server Harvest NSDL services
Create OAI server tables
Metadata Repository
SQL queries OAI server Harvest
NSDL and OAI-PMH Two years later
• Concepts are good, practice is hard
• Issues– Metadata is hard
• http://www.well.com/~doctorow/metacrap.htm
– XML is hard– Protocols are hard
• Static repositories (more later)
– IP is relevant (more later)
Some Essential Metadata Questions
• Review original (DC) metadata assumptions– Metadata is essential for good resource
discovery– “Joe Sixpack” could create metadata
• Account for current realities– 2003 is not 1994– Google, etc. keeps getting better
A u to m a tic Ind e xing
Con
text
ual I
nfor
mat
ion
(e.g
., w
eb li
nks)
M e tad a ta R ichne ss /Q ua lity/T ru s t
Metadata Space
Metadata Triage
Reconsidering the Dublin Core Requirement
• Questions about utility of unqualified DC– The conundrum….
• Specification too loose to serve intended interoperability goal
• But more complex metadata may be too hard
• Limited energy for interoperability– Data providers implement required DC at
expense of better metadata
• Use of protocol for purposes other than resource discovery
Rethinking record-oriented model
B ase Web Graph
N SDL Selec tions
Desc riptive Metadata
A nnotations
B randing
Collec tion (Semantic )
P eople and Organizations
Equivalenc e
Implications for record-oriented harvesting????
Topology Evolution
Repository
OAI-PMHServer
Repository
OAI-PMHServer
Repository
OAI-PMHServer
Repository
OAI-PMHServer
LinkingService
OAI-PMHHarvester
BrowseService
OAI-PMHHarvester
SearchService
OAI-PMHHarvester
Simple Data Provider, Service Provider Topology
Topology Evolution (cont.)
O A I-P M HH arv es ter
OA
I-PMH
Server
Met
adat
aR
epos
itor
y
S ear c hS er v ic e
O A I-P M HH arv es ter
Br o w s eS er v ic e
O A I-P M HH arv es ter
C o llec tio n
O A I-P M HS erv er
C o llec tio n
O A I-P M HS erv er
C o llec tio n
O A I-P M HS erv er
C o llec tio n
O A I-P M HS erv er
Metadata Aggregator
Topology Evolution (cont.)
OA
I-P
MH
Har
vest
erOA
I-PM
HS
erver
OA
I-P
MH
Har
vest
er
OA
I-PM
HS
erver
OA
I-PM
HS
erver
OA
I-PM
HS
erver
OA
I-P
MH
Har
vest
er
OA
I-PM
HS
erver
OAI-PMH p2p network
OAI-P2pMH Issues
• Document (metadata) location– Exploit unique identifiers, use efficient key-based
location mechanisms (distributed hash tables)• Provenance-based queries
– Metadata records may go through refinement and/or translation phases as they move through value-added aggregators.
– Exploit provenance guidelines• Network harvesting
– Broadcast query (Gnutella) inefficient– Exploit techniques for efficient routing of queries (P-
trees)
OAI-PMH and Intellectual Property
• Protocol exists in a context where information providers have concerns about use of intellectual property
• OAI-PMH is nominally about metadata, but…– Rich metadata is an intellectual product– The protocol can be used to transmit anything
(e.g. content) that can be encoded in XML– Generally metadata leads to content so….
OAI-rights effort
• Goal is to investigate and develop means of expressing rights about metadata and resources in the OAI framework.
• The result will be an addition to the OAI implementation guidelines that specifies mechanisms for rights expressions within OAI-PMH. – No changes to core protocol
OAI-rights Effort (cont.)
• Extensible, providing a general framework for expressing rights statements within OAI-PMH. – Not an effort to develop a new rights expression
language
• Use Creative Commons licenses as a motivating and deployable example.
• Release of specification by 2nd quarter ’04• Invited OAI-rights group
– Standard OAI development model
Dimensions of OAI-PMH and rightsEntity Association
• Metadata: concern in NSDL for (re)use of rich metadata
• Content: predominant application of the protocol to resource discovery and ultimate access makes this important
Dimensions of OAI-PMH and rights Aggregation Association
• OAI-PMH aggregations– Repository– Set– Item
• Rights association with an aggregation may provide shortcut (e.g., the rights for all resources in a repository/set…)
• Cost of shortcut is pseudo-statefulness, possibly complex overriding rules
Dimensions of OAI-PMH and rightsBinding
• Choices– exploit mechanisms in metadata formats e.g., DC-
rights– restrict the rights statements to some more specific
protocol mechanism– allow some mixture of these methods.
• DC-rights problems– Semantics is restricted to rights about resource– Can’t embed XML in dc value– What if DC is not required
• Burden on harvesters if rights embedding is not explicit but scattered across several locations
OAI-PMH Static Repositories
• Provide a lightweight mechanism for data provider participation
• Intended for relatively small and static collections
• Two components– Static Repository XML format
• Semantically equivalent to Identify and ListRecords• Invisible to harvester
– Static Repository Gateway• Virtual data provider for static repository data• Unique baseURL for each “contained” static repository
Static Repositories andStatic Repository Gateway
S R 1
S R 2
S R 3
S ta ticR e p o s ito ry
G a te w a y
HT T P G E T
H a rv e s te r
h ttp :/ /s rg .o r g /s r1 /v er b = . . . .
h ttp :/ /s rg .o r g /s r2 /v er b = . . . .
h ttp :/ /s rg .o r g /s r3 /v er b = . . . .
Static Repositories Open Issue
Relationship to RSS?????
Conclusions
• Interoperability and lowest common denominator• Rapid advances automated methods
– Moore’s law– Smart algorithms– Benefits of issues of scale
• Combining human effort and automated methods– Extracting order from chaos– Learning from order
• Move beyond resource discovery