barriers to interoperability technical and not so technical william e. moen school of library and...
TRANSCRIPT
Barriers to InteroperabilityTechnical and Not So Technical
William E. Moen<[email protected]>
School of Library and Information SciencesTexas Center for Digital Knowledge
University of North TexasDenton, TX 72603
5th Annual GILS Conference, April 7-10, 2003 Lisle, IL
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 2
Interoperability: Volume or value
“…information production is growing at about 50% a year…
Yet the amount of time people spend consuming is growing only 1.7% a year…
As digital production and storage grow ever cheaper…a critical task ahead will be to stop volume from simply overwhelming value.”
John Seely Brown and Paul Duguid, The Social Life of Information, 2002
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 3
Defining Interoperability
System-oriented definition The ability of two or more systems or components to
exchange information and use the exchanged information without special effort on either system
User-oriented definition User’s ability to successfully search and retrieve, or
otherwise interact with information in a meaningful way and have confidence in the results
The condition achieved when two or more technical systems can exchange information directly in a way that is satisfactory to users of the systems (AAP)
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 4
Assessing Interoperability
Binary Interoperable Not interoperable
Continuum More or less interoperable Acceptable levels of interoperability
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 5
Fundamental assumptions Your users will interact with resources other than
those available locally Your system will interact with other systems’
content Other systems will interact with your content Systems and content were often meant for local or
proprietary use
In the networked environmentdiverse systems and varied content
must interact to serve users
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 6
The foundation: Communication protocols Interoperability is about communication – albeit
between machines/software Two systems interchange bits/bytes Protocols (computer-to-computer communications)
govern the exchange of bits/bytes Rules governing the structure, order, and content of the
bits/bytes/messages between machines
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 7
Protocols and record formats/markup Protocols enable communication between systems Record formats/syntax/markup address resources HTML = HyperText Markup Language
Markup language for simple web documents Display oriented
MARC = Machine Readable Cataloging Communications format for exchange of bibliographic
information XML = Extensible Markup Language
Markup language for web environment Content oriented
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 8
Factors affecting interoperability Multiple protocols
Z39.50, HTTP, SOAP, etc.
Multiple data formats, syntax, metadata schemes MARC 21, UNIMARC, XML, / ISBD/AACR2-based, Dublin Core
Multiple vocabularies, ontologies, disciplines LCSH, MESH, AAT
Multiple languages, Multiple character sets Multiple and disparate systems
Information retrieval systems, search functionality, etc.
Indexing, word normalization, and word extraction policies
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 9
Assumptions
Increasing interoperability factors
Increased complexity/costs to achieve interoperability
How to identify barriers to interoperability: In general? In specific applications?
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 10
Interoperability TestingRealizing the Vision of Networked Access to Library Resources: An Applied Research and Demonstration Project to Establish and Operate a Z39.50 Interoperability Testbed
A U.S. Federal Institute of Museum and Library Services National Leadership Grant
Research and demonstration project to: Establish a rigorous and trusted Z39.50 testbed Develop methodology and metrics for assessment From demonstration operational testbed
Goal: Improve Z39.50 semantic interoperability among libraries for information access and resource sharing
FOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…
http://www.unt.edu/zinterop/
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 11
Components of the Testbed Test dataset
400,000 MARC 21 records from OCLC Z39.50 reference implementations
Z-client, Z-server, information retrieval system Test scenarios & searches
Searches with known result records from dataset Benchmarks
Results of test searches against reference implementations
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 12
Interoperability findings Successes:
Z39.50 profiles improve interoperability Z-clients and Z-servers can be configured to improve Common indexing policies support better interoperability
New problems: Word extraction (what constitutes a word) Data normalization (special characters, diacritics) Indexing (leading articles, “first in field” searches)
What is good enough interoperability?
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 13
ZLOT project The Library of Texas
A statewide virtual library to extend access to information
Z Implementation Component of the Library of Texas Developing a standards-based approach for the virtual library Identifying requirements for resource discovery across library
catalogs and online databases Demonstrating feasibility and utility of resource discovery tools
• Collection Boundaries
• Organizational Boundaries
• Format Boundaries
FOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…
http://www.unt.edu/zlot/
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 14
One approach for framing interoperability Information communities / communities of practice Networked information retrieval occurs within and
across information communities Information communities
Focal community Extended community Extra community
Costs to achieve interoperability vary depending… Within focal community Within extended community Between disparate communities
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 15
Focal Community Community agreements exist (e.g., standards and rules) Interoperability factors reduced…more easily achieved
Libraries as Focal Community
Relatively homogeneous Standards-based MARC records Content and structure prescribed by AACR Commonly available access points Use of controlled vocabularies Z39.50 widely implemented But…
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 16
Information exchange within/between…
Focal Community(e.g., Libraries)
Focal Community(e.g., Archives)
Focal Community(e.g., Museum)
Extended Community(e.g., Cultural Heritage)
Focal Community(e.g., Geospatial )
Extra Community
Focal Community(e.g., Natural History
Museums)
Extended Community
Focal Community(e.g., Geospatial )
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 17
GILS as information community Conceptualizing GILS as information community Way to discuss barriers to interoperability:
Types of data and resources? Common resource description scheme Commonly used database semantics? Common record syntax (e.g., XML) Commonly understood access points/search concepts? Commonly used controlled vocabulary? Common protocols for information retrieval and
exchange?
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 18
Services context for interoperability Information communities provide setting Information communities offer services
Value-added services to end users Infrastructure services for providers
Mapping a service environment … Needs Services Service interfaces Content
… to identify interoperability barriers/challenges
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 19
Characterizing the networked library Virtual library components An environment for provision of
Services Resources
Not a digital library Not all resources available are in digital/electronic form
An evolving product that responds to Users’ needs Available resources Current and emerging technologies
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 20
Service-centric networked library
UserGroups
Resources
Technology
Management
SERVICES
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 21
Networked library components User groups
Local, onsite Distributed, remote
Services Local Collaborative Remote
Resources Information
• Local• Distributed
Human
Technology Local applications Distributed applications Standards
Management Local Collaborative
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 22
User-driven services
LIBRARY SERVICES
UserGroup
UserGroup
UserGroup
UserGroup
LibraryResources
LibraryTechnology
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 23
Building services for the networked library What are the requirements for the services? What are alternatives for deploying the services? What technologies are available to support the
services? What standards are available? What are the tradeoffs of standards and non-
standards approaches?
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 24
Services for the networked library Infrastructure/internal
Collection identification, selection, and acquisition/access
Collection building and digitization
Collection organization and preparation
Inter-organization access Trust and authentication Billing and payment Preservation and archiving
Patron Resource discovery service Resource access service Reference service Instruction service Patron account service
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 25
Virtual library services environment
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 26
Architecture
PROVIDER LAYER
SERVICES LAYER
USER LAYER
Communication Protocols
Communication Protocols
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 27
Provider layer
DigitalContent
Librarian Library Supplier
Provider Type Dependent on VL Service
ResourceDiscovery
Service
Virtual Reference
Service
ResourceSharingService
DocumentDeliveryService
USER LAYER1
3
2
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 28
Services layer
Standard Information Interchange Among Services
Financial
Authorization
ResourceSharing
PatronAccount
LibraryInformation
EducationResourceDiscovery
Reference Personalization
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 29
Resource discovery: Services view
Browse
SelectCollection
Evaluate
Search
Common User Interface
Personalize
Save
Acquire
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 30
Resource discovery: Architectural view
CONTENT TexShare TRAIL VC Others
USER LAYER Library Home Office Mobile
SERVICES LAYER Resource Discovery
Z39.50
HTTP
OTHERDirectory
Authentication
SOAP
SOAP
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 31
Finding / exposing Finding
I can find only what’s been exposed Exposing
How content is exposed determines the how and if I can find it Common agreements on exposing and finding
Protocols Syntaxes Semantics
• Database• Language
Goal is to connect users to information Strategy is interoperability on multiple levels (protocol,
syntax, semantics, organization, etc.)
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 32
Service interfaces framework
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 33
Resource discovery: Operational view
Content
Content
Content
Content
Content
OAIRepository
Standards-basedSearch & Retrieval
Protocols
Proprietary or Non-standard
Search & Retrieval Protocols
Content
Content
Content
Gateway
CommonInterface
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 34
Infrastructure standards -- NISO ANSI/NISO Z39.50: Information Retrieval (Z39.50): Application
Service Definition and Protocol Specification ISO 10160 & ISO 10161: Interlibrary Loan Application Service
Definition & Protocol Specification ANSI/NISO Z39.83: Circulation Interchange Protocol (NCIP) ANSI/NISO Z39.84: Syntax for the Digital Object Identifier - 2000 ANSI/NISO Z39.85: Dublin Core Metadata Element Set - 2001 ANSI/NISO Z39.88: The OpenURL Framework for Context-Sensitive
Services (2002) ANSI/NISO Z39.89:The U.S. National Z39.50 Profile for Library
Applications (2002) Networked Reference Services (standards development underway) MetaSearch Initiative (may lead to new standards activities)
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 35
Standards supporting services Each standard enables different services
Networked information retrieval (Z39.50) Automated interlibrary loan (ILL) Circulation applications (NCIP) Reference linking (OpenURL) Resource description (Dublin Core)
But they can work together to support robust and interoperable information access and resource sharing services
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 36
Web services
Building Blocks for Constructing Applications
Service A
Resource Discovery
Service B
Authorization
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 37
Web services: Finding a service
Service A
Resource Discovery
Service B
Authorization
ServiceRegistry
ServiceRequestor
ServiceProvider
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 38
Web services: Enabling technologies
ServiceRegistry
ServiceRequestor
ServiceProvider
XML-Based Information Exchange
SOAP
UDDI - WSDLUDDI - WSDL
WSDL
UDDI: Universal Description, Discovery and Integration WSDL: Web Service Description FrameworkSOAP: Simple Object Access ProtocolXML: Extensible Markup Language
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 39
Web services
ResourceDiscoveryService
Service B
Authorization
ServiceRegistry
UDDI - WSDLPublishUDDI - WSDL
Find
WSDL - SOAP
Invoke Service
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 40
ZING – Z39.50 International Next Generation Make intellectual/semantic content of Z39.50 more
broadly available Make Z39.50 more attractive by lowering barriers to
implementation Use of XML – to represent and encode data Use of HTTP – for transport Use of SOAP – for interaction between client and server
based on Remote Procedure Call (RPC) Several initiatives: ZOOM, ez39.50, ZeeRex,
SRW/UFOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…
http://www.loc.gov/z3950/agency/zing/zing.html
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 41
Search/Retrieve Web Service – Overview Builds on Z39.50 and web technologies Retains Z39.50 concepts:
Results sets Abstract access points Abstract record schemas Explain & Diagnostics
Web technologies: XML, SOAP/RPC, HTTP Combines several Z39.50 features into two “operation types”
Search/Retrieve Explain
Provides a standards-based method for representing and issuing queries, and retrieving records
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 42
Interoperability and the user Technical interoperability as foundation Value-adding services respond to user tasks Usability of the service
Reliable Predictable Ease of use User interface …
Moen 5th Annual GILS Conference -- April 8, 2003 -- Lisle, IL 43
Final thoughts Interoperability is essential Interoperability is multi-faceted Interoperability is ultimately based on agreements
Formal standards Community agreements
Agreements result from Social processes Willingness to collaborate Balance of local needs in the networked environment
It’s the “not so technical” we need to work on