opening the black box of “relevance work”: a domain analysis

Opening the Black Box of “Relevance Work”:A Domain Analysis

Betsy Van der Veer MartensSchool of Library and Information Studies, University of Oklahoma, 4502 East 41st Street, Room 1J30, Tulsa,Oklahoma 74135. E-mail: [email protected]

Connie Van FleetSchool of Library and Information Studies, University of Oklahoma, 120 Bizzell Library, Norman, Oklahoma73019. E-mail: [email protected]

In response to Hjørland’s recent call for a reconceptual-ization of the foundations of relevance, we suggest thatthe sociocognitive aspects of intermediation by informa-tion agencies, such as archives and libraries, are a nec-essary and unexplored part of the infrastructure of thesubject knowledge domains central to his recommended“view of relevance informed by a social paradigm” (2010,p. 217). From a comparative analysis of documents from39 graduate-level introductory courses in archives, refer-ence, and strategic/competitive intelligence taught in13 American Library Association-accredited library andinformation science (LIS) programs, we identify fourdefining sociocognitive dimensions of “relevance work”in information agencies within Hjørland’s proposedframework for relevance: tasks, time, systems, andassessors. This study is intended to supply sociocogni-tive content from within the relevance work domain tosupport further domain analytic research, and to empha-size the importance of intermediary relevance work for allsubject knowledge domains.

Introduction

In Hjørland’s recent (2010) restatement of the impor-tance of the sociocognitive foundations of relevance, heproposes that “relevance . . . is understood in relation tohuman activities: Something is relevant in relation to a taskif it supports the fulfillment of this task. Relevance in LIS isabout what kinds of knowledge or information are fruitfulfor different purposes: What kinds of knowledge serve as atool for some human actions?” (p. 229). In reconsideringSaracevic’s magisterial analyses of relevance research(1975, 2006), Hjørland argues against the current view of

relevance as an unresolved dichotomy between a “system’sview” that equates relevance with “objective” outputs (p.218) versus a unrealistically universal “user’s view” (p. 218)that conflates relevance with individual relevance judgments(p. 224), and urges a return to the epistemological or“subject knowledge” view (p. 217). He acknowledges thecomplex interrelationship between the contingent nature ofsubject knowledge and the ongoing social construction ofreality within the broader knowledge ecology, and theimportance of empirical explorations of human goals,values, and interests within these. This both extends anddeepens Hjørland’s seminal work on domain analysis(2002a, 2002b; Hjørland & Albrechtsen, 1995). His domainanalytic project, while sometimes criticized for its inatten-tiveness to methodological specifics (Tennis, 2003), its over-attentiveness to domain details (Szostak, 2010), and itslimitations in comparison with more comprehensive modelsof information seeking and retrieval (e.g., Ingwersen &Järvelin, 2005), remains an innovative and important contri-bution for other researchers in the LIS field.

Hjørland (2010, p. 232) concludes that a truly pragmaticview of relevance allows for the notion that a given docu-ment or piece of information may or may not be relevant toa particular task at a certain point in time, but that it maybecome more or less relevant at other times, for other tasks,by other assessors, or within other systems. In his previousmapping of domain actors, institutions, and documents, hepositions “the library” as a black box occupying an interme-diate position between information producers and users(Hjørland 2002a, p. 447). The library is also central to bothhis rejection of relevance as being either purely objective(system’s view) or purely subjective (user’s view). Hesays that “the ‘system’ in question could be, for exam-ple, a public library. Documents are selected, cataloged,classified, and indexed, perhaps influenced by ideas taught

Received July 5, 2011; revised November 23, 2011; accepted November 25,

2011

© 2012 ASIS&T • Published online 14 February 2012 in Wiley OnlineLibrary (wileyonlinelibrary.com). DOI: 10.1002/asi.21699

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 63(5):936–947, 2012

at schools of library and information science. Often suchideas are based on the view that the library should be basedon application of objective and neutral principles of man-agement and often ideologies about ‘user-centered’ servicesare dominant. In the end, however, some priorities have beenmade, and some information needs are better served thanothers. Libraries are thus also ‘subjective’ in some ways”(2010, p. 219). Although he notes that libraries may usehighly qualified subject specialists (2010, p. 221), his dis-cussion does not include how or why their services contrib-ute to the operations of the “black box” of the library.

The library black box can indeed be both “objective” and“subjective,” as it is far from being a single, static system.Rather, it comprises the important intermediating servicesprovided by those “relevance workers” who staff thearchives, the libraries, and other information agencies thatprovide a variety of interlocking systems for intellectualwork of all types, that is the overall relevance work domain.This domain provides an essential but largely unexaminedinfrastructure that underlies any study of “relevance,” andrequires the same research attention to its sociocognitiveaspects as does the analysis of any other domain. Hjørland’sproposed “pragmatic model” of relevance involving task,time, assessor, and system appears well worth such furtherexploration. Examining the educational processes involvedin preparing “relevance workers” can do much to provide asociocognitive context, because the libraries and other infor-mation agencies depicted in his model are staffed by thosewho have undergone these educational processes or aresupervised by those who have.

As Hjørland notes, “relevance research” has never beenfully integrated into the broader field of LIS research (2010,p. 219). We suggest that the construction of the knowledgeorganizing systems that he correctly names as one of thecore competencies within LIS is embedded in the ongoingsocial construction of what we term here “relevance work.”This work is the practice of information intermediation asexercised in the evolving information environment, support-ing a wide variety of information sources, services, andsystems as well as information creators, intermediaries, andusers, all operating under an equally wide ranging set ofgoals and constraints. In fact, relevance work is central toLIS, which is one of the few fields that engages both prac-tically and theoretically with issues of meta-relevance, andhas done so since long before the introduction of comput-erized information systems prompted Cuadra and Katter’scelebrated article (1967) on “opening the black box of‘relevance.’ ”

Opening the Black Box of Relevance Work

The practice of reference has always been based on rel-evance intermediation: finding the “best” matches for anindividual patron’s expressed need within the confines of aparticular collection (Tyckoson, 2011). Reference educationwas built on this foundational paradigm of the archivist orlibrarian as the guardian and gatekeeper for an individual

collection, whose knowledge of the surrogates for specificitems within that location always informs interactions overtime with a succession of users of various items of thatcollection. Green’s instructions for reference librarians(1876), Dana’s library primer (1920), Ranganathan’s fivelaws of library science (1931), Schellenberg’s distinctionsbetween the “informational” and “evidential” aspects ofdocuments within an archive (1956), Aguilar’s (1967) andAllen’s (1977) work on internal and external informationchannels in organizations, Atkinson’s concept of thelibrary’s “control zone” (1996), and other seminal work inthe field all assume such a structure or system.

The establishment and existence of such structuresclearly assume some kind of information intermediation(Taylor, 1986, pp. 121–124). This combination of explicitand implicit intermediation is precisely what distinguishesLIS education from other forms of knowledge work(Donovan, 2008). Beginning in the 1970s, the “disinterme-diation” of the library has been viewed with alarm as thebeginning of the end of the role of the librarian as interme-diary (Miller, 2007). In fact, Markey’s 2004 curriculum-based study of trends in LIS education proposed thatschools de-emphasize “information intermediation.” Shesuggested that “staking claim to the ‘users’ niche may bedangerous for librarians and information professionalsbecause it places them in an intermediary role. When infor-mation technology assumes the role of the intermediary as itdoes in computer-based information retrieval, users mayhave no need to interact with intermediaries and bypassthem in their search for information. . . . If librarians focusexclusively on users, they run the risk of becoming increas-ingly irrelevant” (Markey, 2004, p. 334).

However, as Buckland (1991) suggested, access to infor-mation is never unmediated, whether such intermediation isvisible (White, 1992) or invisible (Fox, 2010). Hjørland(2010, p. 218) notes too that the illusion that a computer-mediated search somehow ensures an “objective” algorith-mic relevance is unwarranted. “Search neutrality,” unme-diated by the values and interests of content providers,search engine companies, and the many other stakeholdersin the search process, has been shown to be illusory (e.g.,Goldman, 2010; Grimmelmann, 2010). In fact, the studyof the intersecting biases of search engines and their usershas become a subject of intense investigation (e.g., Segev,2010). Mann (2007) eloquently points out in his ThePeloponnesian War and the Future of Reference, Catalog-ing, and Scholarship in Research Libraries that the with-drawal of librarian expertise from information integrationand intermediation efforts would result in an incalculableloss to serious scholarship.

We would argue that the value of information interme-diation is increasing, not decreasing, in financial and legalfields, for example, but that its use in relevance work hasnot been fully appreciated, given that the reference field’sfocus to date has been on the decrease in “ready reference”queries rather than on the increase in more sophisticateddata-driven ones. Meanwhile, LIS schools today continue

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—May 2012 937DOI: 10.1002/asi

to educate a wide variety of information intermediaries,including reference librarians, instructional librarians, stra-tegic intelligence specialists, digital archival access con-sultants, medical informationists, and legal informationresources managers. Over time, the objectives and out-comes of these LIS educational processes are foundationalto the practice of relevance work in supporting other formsof knowledge work. Examining relevance work offers a wayto gain an understanding of LIS’s contribution to the eco-logy of knowledge work in general.

Accordingly, this study offers a preliminary look at hownovice information intermediaries are professionally social-ized into the “established norms about relevance” (Hjørland,2010, p. 229) that underpin work in libraries and otherinformation agencies. It examines explicit and implicitnorms for relevance work within a selected group of LIScourses to explore their applicability to Hjørland’s relevancemodel, especially in regard to the “library” as representedthere. And, finally, it employs a document-based approachconsistent with Hjørland’s own emphasis on the “academyprinciple,” in which subject knowledge expertise is pre-sumed to be paramount (2010, p. 221).

Information Intermediation

Information intermediation is practiced in a variety ofsettings. Reference sources and services courses have beena mainstay of LIS education for many years, while coursesfocusing on intermediary work in archival (Cox, Yakel,Wallace, Bastian, & Marshall, 2001) and intelligence(Shelfer & Goodrum, 2000) environments are fairly recentadditions to such curricula. All three may be consideredpart of the relevance work domain, as they all provide impor-tant intermediary functions in their different environments.Though the histories of work in archives (Berner, 1983),reference (McArthur, 1986) and intelligence (Laqueur,1985) are long, they have seldom been considered together asvariants of relevance work within LIS and, accordingly,neither have their research activities. Therefore, this sectionwill briefly review recent research on information inter-mediation in these key settings.

Research on reference work in libraries is enjoying aresurgence (see Radford & Lankes, 2010 and Zabel, 2011for recent compilations) since, as a broader spectrumof information sources becomes available, long-standinglibrary standards and practices for their appraisal andacquisition no longer seem adequate. This has led to a newfocus on determining whether and how information can bejudged to be authoritative and accurate (e.g., Burkholder,2010; Rettig, 2006), including suggestions of a movetowards reliability and community-based rather thanauthority and collection-based standards for referenceactivities (e.g., Lankes, 2008). Accordingly, referenceresearch has moved beyond early quantitatively orientedreference theories such as Trueswell’s (1969) 80/20 rule orHernon and McClure’s (1986) 55% rule. Current referenceresearch offers new models that focus on the analysis of

various determinants of overall client satisfaction (e.g.,Radford, 1999), the optimization of reference transactionsystems (e.g., Saxton & Richardson, 2002), the develop-ment of wide-area social reference networks (e.g., Shachaf,2009), and the creation of more interactive and collabora-tive instruction/reference services (e.g., Pawley, 2003;Agosto, Rozaklis, MacDonald, & Abels, 2011). Interest-ingly, Kwon (2010) offers a cognitive sociological over-view of these reference “thought communities” that makesa distinction similar to that of Hjørland’s among the “user,”“system,” and “knowledge” perspectives.

Within archives research, the “archival mind-set”(Diamond, 1994) has tended to focus more on theappraisal, arrangement, and custodial functions involved inthe preservation of documents and records than on accessby their prospective users. However, there has beengrowing recognition that to survive both culturally and eco-nomically, archives of all kinds must become more access-oriented (Huvila, 2008). Accordingly, research on thearchival user has made considerable advances in recentyears (Yakel, 2010). The challenge of helping users todevelop what Yakel and Torres (2003) call “archival intel-ligence” reflects the reality that specific contents ofarchives continue to be correctly arranged according toprinciples of provenance and original order, rather thanaccording to subject headings or other library-oriented clas-sification systems, as are most finding aids (Cox, 2007).Duff and Fox (2006) comment that even highly skilledusers are still often dependent on the mediation of thearchivist, and Duff (2010) notes that such mediation is inneed of much further research.

Organizational intelligence research is based on a verydifferent foundation: an organization-centric one that focuseson timely decision making informed by the rigorous analysisof “actionable intelligence” found by intelligence specialiststhrough a variety of means (Luhn, 1958). Although Aguilar’sseminal work on managerial intelligence (1967) focused onthe individual manager as both environmental scanner andstrategic decision maker, larger organizations, both publicand private, normally separate the functions of managerialstrategic cognition and strategic intelligence work (DeAlwis,Majid, & Chaudry, 2006). This is because of the recognitionthat such activities have become too complex to be handledby a single person (Choo, 2006), that most informationanalysis can benefit from shared perceptions (Bernhardt,1999), and that most organizational decisions today requireconsensus by a variety of stakeholders who require a convinc-ing array of evidence before taking any action (Fahey, 2007).With few exceptions (e.g., Johnson, 2003a, 2003b), themajority of research on intelligence mediation has been per-formed by practitioners and is oriented towards practicalissues such as the most effective methods to identify topicsfor investigation (Herring, 1999), to assemble group intelli-gence efforts that also make the best use of individualresearch competencies (Fahey & Herring, 2007), or toexamine variations on the so-called “intelligence cycle”(McGonagle, 2007).

938 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—May 2012DOI: 10.1002/asi

Information Intermediation Education

In North America, professionalization of library andinformation science workers is done through the certifica-tion of particular educational programs, based largely oncoursework involving what are considered to be the philo-sophical and practical foundations of the field (Swigger,2010). American Library Association-accredited programsare required to provide program objectives stated in terms ofstudent learning outcomes (American Library AssociationAccreditation Standards, 2008). Each course in a program’scurriculum, therefore, is supposed to provide informationrelevant to those statements. These normally comprise acatalog course description, accompanied in many instancesby a set of course goals or objectives, and student learningobjectives or outcomes. These materials often form part ofthe syllabus developed by each course instructor for theinformation of students enrolled in that class (Habanek,2005). While individual instructors for a particular coursemay design their syllabi differently, these normally representdifferent ways in which to achieve the agreed-upon learningobjectives (Parkes & Harris, 2002). Although formallystated learning objectives and outcomes are not universallyaccepted in academe, the influence of instructional designmethods, the theoretical turn away from content-centeredteaching towards student-centered learning, and demandsfor evidence-based evaluation make them both popular andprevalent (Brosman, 1998; Diamond, 2008). Such previousstudies of trends in reference curricula as those of Broadwayand Smith (1989), Powell and Raber (1994), and Adkins andErdelez (2006) have included the study of course syllabi,while O’Connor (2011) makes special mention of data fromcourse objectives and outcomes.

Our project, therefore, studies these documents, as docu-ment study is recommended as one of Hjørland’s 11 recom-mended domain analytic approaches (2002a). Because thecorpus includes actual syllabi and associated reading listsfrom courses taught by faculty instructors in AmericanLibrary Association (ALA)-accredited institutions, we alsoemploy his “academy principle” of expert verification of keyconcepts (Hjørland, 2010, p. 221). Kwasnik (1999, p. 39)points out in her discussion of knowledge discovery throughclassification that classification research on “dimensionsguided by theory or a model usually do a better job ofreflecting knowledge in the domain because they rest on aconsensual framework of description.” To guide our searchfor a “consensual framework” in this analysis, we exploredthe coursework corpus materials especially for commonthemes and topics that did and did not match those modalcategories that Hjørland suggests may be of special impor-tance within the overarching relevance work domain: task,systems, assessors, and time. Our study is also informed byTennis’s “two axes” method of domain analysis, which pro-vides a methodological lens (2003). We use, first, Tennis’s“areas of modulation” (what specific areas of the domain arecalled, and what they cover) to explore the parameters ofrelevance work as taught within ALA-accredited programs.

Second, we use Tennis’s “areas of specialization” (the focaland intersecting points of the domain) to distinguish amongspecific manifestations of those areas.

Study Methodology

All North American schools with ALA-accreditedprograms offer at least one reference-related course, butnot all offer courses in archival or intelligence work aswell. In order to insure within-program comparability, wechose to assemble our coursework corpus only from those13 different schools that offered separate, introductorycourses in all three information intermediation areas(archives, intelligence-seeking organizations, and libraries)to obtain roughly comparable data on each course fromthese schools. Table 1 shows those courses identified asfocusing on beginning archival, reference or strategic intel-ligence skills.

During the month of October 2010, course catalogdescriptions and associated materials, including learningobjectives, student learning outcomes, and, for the 11 of the13 programs for which these were publicly available, recentsyllabi, were downloaded from each school’s website. In adeliberate effort to use only those course materials that wereclearly representative of each school’s public contribution toprofessional discourse about these areas, only course mate-rials that were readily accessible through each school’swebsite were downloaded. The downloaded materials werethen transferred into an Excel spreadsheet database foranalysis. Although more generally used for quantitative dataanalysis, Excel is also well suited for qualitative data analy-sis because of its ease of organization and ability to facilitateindependent multidimensional coding (Meyer & Avery,2008). The analysis comprised the two authors

• sorting the courses, based on both course title and description,into “areas of modulation,” that is, the type of informationagency on which the intermediation activities are stated asbeing focused: archives, libraries, or “other” organizations;

• assigning each learning objective or learning outcome to oneor more of Hjørland’s four categories (system, task, assess-ment, time);

• developing subcategories for each of the four main categoriesto which the particulars of the various learning objectives,outcomes, and the related activities and assignments from thevarious syllabi could be most appropriately assigned to deter-mine “areas of specialization”;

• determining the specialization “foci” and “intersections”(how the descriptions of the particulars compare and contrastamong courses focusing on the different intermediary infor-mation agencies;

• agreeing on the categorical structures that depict these com-parisons in sufficient abstraction to provide a diagram of theoverall “intermediation” within these sectors of the relevancework domain that can be considered to be the “focal” ordefining ones; and

• developing a generic model of the defining dimensions ofrelevance work that could be applied to all information inter-mediating agencies.


Areas of Modulation in the RelevanceWork Domain

This section describes the general “areas of modulation”as presented within the educational materials of the 39 LIScourses listed above, as well as comparing and contrastingthe “areas of specialization” within each. It provides a moreabstract level of discussion that brings together the specificsfrom the various tables shown below.

System

Hjørland’s first main category is that of system, which, asa general schematic, contains all the intermediary systemsand their components. Within intermediation education,however, this denotes the particular system’s interactiveenvironment, such as the library or archive, the culture or“ethos” surrounding it, and the level of instruction needed tomake effective use of it.

Ethos

Archival, reference, and intelligence professionals gen-erally subscribe to the belief that information-related pro-fessions are under a special obligation to conform to thehighest ethical standards in regard to the collection, use,and dissemination of information. References to codes ofethics from the ALA, the Society of American Archivists,and the Society of Competitive Intelligence Professionalsappeared frequently within the syllabi and associated

reading lists. Within each setting, these ethics also consti-tute a focus for organizational culture, or ethos. Althoughusers may not be fully aware of all of these professionalconstraints, they are often directly impacted by the result-ing behaviors and information provision. In the archivalenvironment, the overriding ethos can be summarized asthe protection and representation of archival evidence in itsoriginal context. In the reference environment, the overrid-ing ethos involves a service orientation that focuses onequitable treatment of users’ requests in terms of satisfyingtheir information needs, accompanied by the protection ofthe individual user’s privacy in regard to the informationrequest itself. In intelligence work, these ethical standardsinclude refraining from misrepresenting the identity of theinquirer and the purpose of the inquiry, and a high levelof investigative integrity in the treatment of all sources,especially those involving human beings who may beharmed either personally or professionally by providing therequested information.

Interaction

The overall environment in which the information systemoperates sets the terms of intermediation and interaction.Within archives, such mediation traditionally involves con-sultation with reference archivists and finding aids. The useris able to peruse and make notes concerning specific originalrecords of interest under supervision on the archive’s pre-mises, but not to remove or (in many instances) to copythem. Such phrases from archival course learning objectives

TABLE 1. Graduate level course numbers and titles.

School Archives course Reference course Intelligence course

British Columbia ARST 540 Archival Public Services LIBR 503 Foundations of InfoSources & Services

LIBR 548B Issues in Info Services:Competitive intelligence

Dalhousie University INFO 6800 Archives INFO 5530 Info Sources &Retrieval

INFO 6310 Resources for BusinessIntelligence

University of Denver LIS 4800 Intro to Archives &Records Management

LIS 4060 Reference LIS 4203 Competitive Intelligence

Dominican University LIS 775 Archival Administration &Services

LIS 704 Reference & OnlineServices

LIS 884 Competitive Intelligence

Drexel University INFO 560 Intro to Archives I INFO 521 Info Users & Services INFO 678 Competitive IntelligenceIndiana University S581 Archives & Records

ManagementS501 Reference S555 Strategic Intelligence

University of Maryland LBSC 605 Archival Principles,Practices, & Programs

LBSC 650 Info Access Services INFM 714 Principles of CompetitiveIntelligence

McGill University GLIS 645 Archival Principles &Practice

GLIS 619 Info Services & Users GLIS 665 Competitive Intelligence

University of Oklahoma LIS 5423 Archives LIS 5513 Info Sources & Services KM/LIS 5553 CompetitiveIntelligence

Simmons College LIS 438 Intro to Archival Methods& Services

LIS 407 Reference/Info Services LIS 530M Competitive Intelligence

University of Texas-Austin INF 389R Intro to ArchivalEnterprise I

INF 382D Intro to Info Resources &Services

INF 382P Competitive IntelligenceResources & Strategies

Wayne State University LIS 7710 Archival Administration LIS 6120 Access to Info LIS 7490 Competitive Intelligence& Data Mining

University ofWisconsin-Milwaukee

752 Archival Outreach: Programs &Services

521 Introduction to ReferenceServices & Resources

833 Competitive Intelligence &Business Information


as “examining the nature and need for public programming”and “the legal and ethical issues related to access to archivalrecords” express some of the restrictions that bound archivalreference work. Within libraries, the common use of theterms “reference transaction” and “reference encounter”throughout many of the syllabi indicate that the majority ofreference relationships tend to be temporary rather than onesthat endure for a longer period of time. In intelligence work,however, the relationship tends to be longer lasting, if onlybecause the sensitivity of the information sought is such thatthe fewer people involved, the more secure the environment.

The other difference is in the relative power of the twoparties: in the library environment, the librarian exerts morepower and the user has only the option of refusing theinformation and/or terminating the interaction, while inthe competitive intelligence environment, the user is usuallyof higher status than the intelligence professional, and canwield more authority in setting the boundaries of the inquiryand the nature of an “acceptable” answer. This is somewhatobliquely expressed in the learning objective phrased as“develop an actionable CI product/service to meet the needsof a particular audience.”

TABLE 2. Relevance work domain areas of modulation and specialization.

Domain: Areas of modulation

Agency Archives Libraries Other organizations

Dom

ain:

Are

asof

spec

ializ

atio

n

System-specific environmentEthos Mentions of SAA’s Code of Ethics

Protect sanctity of records as evidenceMaintain the intellectual integrity of

aggregations of recordsPreserve the contexualization of

aggregation of records

Mentions of ALA’s Code of EthicsProtect of users’ privacy paramountRespect for copyrightService orientation

Mentions of SCIP’s Code of EthicsRefrain from identity misrepresentationInvestigative integrityRespect for intellectual property rights

Interactions Sporadic: Usually new customersunfamiliar with organization ofspecific archive

Varies: transactions with both regularand new customers of library

Routine: regular contact with knowncustomers (often managerial) oforganization

Instruction Sometimes applicable (emphasis onadhering to access policies)

Often applicable (emphasis is ondemonstrating search process)

Seldom applicable (emphasis is onproviding results)

Task-specificQuery modes Electronic: Encoded Archival

DescriptionElectronic: Metadata

(Keywords/Tags/Subject Headings)Dependent on context: oral/aural/visual

Print: Archival Finding Aids Print: Abstracts, Indices, TOCsExpertise Search skills Search skills Search skills

Authentication skills Expository skills Analytic skillsContextual knowledge (historical or

environmental)Collection knowledge (service- or

education-oriented)Mission knowledge (industry- or

government-oriented)Tools Document based: Collection based: Situation based:

1st level: collection finding aids 1st level: general reference sources 1st level: industry or political newssources

2nd level: collection-level research 2nd level: specialized databases &resources

2nd level: business-specific orgovernment-specific sources

3rd level: individual document research 3rd level: referral to experts specificpublication sources

3rd level: human intelligence sourcessignal sources

Time-specificNeeds assessment Orientation Interview (initial and follow

up as necessary)Reference Interview (initial/follow-up as

necessary)Information Requirements (at different

stages of “intelligence cycle”)Effort Effort is either standardized or

customized (determined by archivalpolicies) but may include value touser

Effort is standardized (determined bylibrary policies) regardless of value touser

Effort is customized (determined byorganizational policies) based onvalue to user

Time Occasionally time-sensitive: determinedby archival policies

Often quite time-sensitive: determinedby nature of user need

Extremely time sensitive: determined bynature of information and user need

Assessment-specificProduct Authenticated evidence (usually

unpublished information)Relevant information(usually published

information) or referralActionable intelligence (usually

unpublished information)Distribution Location of potentially relevant results

for customerDelivery of results to customer Preparation of results for customer

Evaluation Of security: avoiding damage/loss Of number/type of customer requests Of credibility of sources/informationOf number/type of customer requests Of tools Of usefulness of results to clientOf results: focus is on broad recall

rather than narrow precisionOf results: both precision and recall as

surrogate for relevance

Note.


Instruction

The role of instruction varies substantially within thethree environments. Reference archivists usually providelittle instruction beyond an initial “orientation interview”because users tend to have special expertise of theirown that will assist them in best utilizing the archive’scontents. Reference librarians often engage in instructionalactivities, either to demonstrate the use of specific toolssuch as a database or to improve general informationliteracy skills. “To develop basic library instruction skills”is a common instructional objective in reference courses.And, finally, instruction of the end user is seldom partof intelligence work. Although the provenance of theparticular information and the process used to obtain it canbe critically important, it is not customary to teach theusers of the intelligence product how to obtain it them-selves, as the cost of managerial attention is such that itdemands only highly relevant information for decisionmaking. This is shown by the frequent mention of strategicmanagerial decision making in the intelligence coursereadings.

Task

Hjørland’s second major category is that of task. Withinintermediation education, this refers to the specific skillsneeded to assist users in their search tasks. These skills arecomprised of query modes, expertise, and tool use.

Query Modes

Query modes refer to the ability to formulate a variety ofqueries in order to make the most efficient and effective useof discovery tools such as catalogs, indices, and other searchengines. This appears in reference course learning objectivesas “the ability to formulate search strategies that will effi-ciently identify and locate relevant information.” Althoughthe reference librarian is expected to be conversant with agrowing number of discovery tools, the initial formulationof most queries can be addressed through classic referencetraining in search definition, scope, and specifics as des-cribed in these course syllabi. A common learning activityfor students in reference classes is to construct and completea search for a particular user.

Reference archivists tend to be more intimatelyinvolved with the arrangement of their individual archivethan reference librarians are with the cataloging oftheir reference collections, as the former are far lessstandardized than are the latter. Accordingly, the prepara-tion of finding aids is a common learning activity inarchival syllabi, targeted toward becoming familiar enoughwith a particular collection to enhance its query modedevelopment.

Within strategic intelligence, while published informa-tion (so-called “open source intelligence”) is often useful,interviewing human sources of potentially relevant infor-

mation can be an essential part of the search. This type ofquery mode can be considered more akin to investigativejournalism or forensic examination, but it can also draw onenhanced reference interview skills, shifting the focus frominformation sought for the subject to information soughtfrom the subject. The ability to phrase questions in such away as to extract critical information ethically from inter-viewees is one of the essential query mode skills of theintelligence specialist.

Expertise

Although the development of expertise is important in thearchival, reference, and intelligence environments, these canentail quite different types of skills. One way in which thesedistinctions are expressed within our corpus is that, whilereference archivists and reference librarians normally“direct” their users to sources, an intelligence specialist triesto “develop” sources for their users.

Expertise in the archival environment normally requiresexperience with the actual items of the collection in additionto knowledge of the appropriate finding tools. Expertise inthe library environment is generally text retrieval-based,requiring well-developed competencies in bibliographic andother retrieval tools, subject vocabularies, and query strate-gies, as well as a comprehensive understanding of both thelocal and global collection resources. The development ofexpertise in the intelligence environment can involve the useof ordinary business research sources, but also often requiresother more specialized elicitation techniques.

Although the courses we analyzed were intended to beat the elementary level, the reading lists and textbooksclearly indicate that there are higher levels of mastery thatan experienced professional would be expected to attain.For example, advanced expertise for reference archivistslies in their knowledge of the depth of their collection,while the expertise of reference librarians lies more oftenin their knowledge of collection breadth. For intelligencespecialists, advanced expertise lies in knowledge of theorganization’s specific mission, goals, and overall environ-ment, including competitors. In business, this can includethe industry and regulatory environment, while in govern-ment, this can include the sociopolitical environment, bothdomestic and foreign.

A third, more abstract level of expertise lies in the abilityfor “relevance workers” not only to retrieve relevant infor-mation but also to assess its attributes as accurate, current,credible information in terms of user requirements. For ref-erence archivists, this lies in their understanding of therecords appraisal process, which helps them to contextualizethe evidence within their archives for users. For librarians, itlies in their expository ability, which allows them to describeand explain the value of information sources and results to avariety of users and potential users. For intelligence special-ists, it involves their ability to analyze sources and resultsrather in their ability to explain or expound upon them tonovice users.


Tools

The teaching of appropriate tools for relevance work isoften considered the heart of such courses. Many referencecourses allow ample time for students to familiarize them-selves with a wide variety of commonly used referencesources. In the archival classes, time is similarly devoted tostressing the relationship between finding aids and the criti-cal collection factors of original order and provenance. Inthe intelligence environment, however, the task may requirethe use of very different, largely incompatible sources totriangulate the likelihood of a particular piece of data beingaccurate, so the intelligence syllabi tend to address the use oftools in a much more critical fashion, as their accuracy andveracity cannot be taken for granted in the same way as inthe archival and library environment.

Time

Hjørland’s third main category is that of time, which hementions as a potential variable in the context of a user’sparticular relevance needs. From the standpoint of informa-tion mediation training, however, time is multidimensional.Although these time dimensions include needs assessmentof the user’s time constraints and considerations, they alsoinclude system effort time constraints and considerations,which can vary dramatically among the three environments.And, finally, information can be viewed as having a timehorizon of its own, and its intermediation may also involveusage by multiple users over a short or long term timehorizon.

Needs Assessment

Identifying user needs continues to be central to mediatedsearch of all kinds. An initial orientation interview is cus-tomary within the archival setting, in which the referencearchivist explains the archive’s arrangement and policies,while the user explains the goals of the visit. In libraries, thecomplexity of classic reference activities is now well recog-nized in both face-to-face and online settings. Basic trainingin needs assessment (the reference interview) comes onlysecond to training in search and sources themselves in ref-erence course syllabi. Within strategic intelligence work,there is a corresponding difficulty in trying to not only fulfillbut also predict user needs for information that may or maynot be available. These courses, therefore, often involve bothgeneralized and specific environmental scanning at a varietyof levels (business, industry, international, etc.)

Effort

In most archives and reference library settings, there is astandardized level of service for each system, though therecan be some variation, based on specific user needs. Somelibraries do offer additional services for additional fees,and some archives do give priority and special service to

particularly qualified users. In general, however, archivesand libraries attempt to value every user equally, and tobudget overall projected costs accordingly, rather than toallocate them to individual users. As one example, trainingin “ready reference” service skills in reference classesunderscores this view of routinized customer service. In theintelligence environment, however, the importance of theinformation sought can dramatically impact the amount oftime and money necessary for the effort, so it would beappropriate to note that costs are more likely to be custom-ized to reflect actual value in this environment, and thatsome intelligence class readings emphasize the importanceof cost-benefit analysis for individual intelligence searchactivities.

Time

Time as a determinant of relevance is an important factorin all three environments. The present value of informationis probably best understood in the library environment, asthe same piece of information can be of utility to manydifferent users, with many different objectives. Referencecourse readings often mention “just-in-time information”and “saving the time of the user.”

The future value of information is best appreciated in theintelligence environment, as the shelf life of competitiveintelligence is much shorter than that of information frombooks. So-called “competitive advantage,” which oftenappears in the intelligence readings, is contingent on actionbeing taken by those who are privy to novel, valued infor-mation. It quickly becomes obsolete (expressed in intelli-gence readings by the acronym OBE, “overtaken by events”)and can perhaps even become a liability for the organizationas it becomes more widely available.

Conversely, the past value of information is shown in thearchival setting. Archival information about the past mayincrease significantly in value over time, as it becomes aprimary source of evidence for the future. The archival envi-ronment is also traditionally known for retrieval results thatemphasize abundance over relevance, given that exact speci-fications for an individual record are less likely to succeedthan a more general query that may include the desiredrecord. The value of information within the archive is that itprovides evidence of the past itself, though mediated by bothmemory and method, so “preservation” and “stewardship”are key concepts for the archival syllabi.

Assessment

Hjørland’s final main category is that of assessors, whichwe express more broadly here as assessment, in that theassessment by each individual end user is only a partialdeterminant for the system as a whole. In intermediationeducation, assessment involves the consideration of theinformation product, its distribution, and its evaluation byboth the intermediary and the end user.


Product

Although the ultimate product of any query is obviouslyinformation of some kind, distinctions among the types ofinformation sought can be marked. Within the archival envi-ronment, the curricular emphasis is on “the record”: particu-lar evidence rather than on general information, as theproduct of an archival investigation is often unique andoccasionally irreplaceable. Physical documents and originalrecords can also bear additional evidentiary traces of theirorigins, uses, and history that provide special value to theuser beyond the informational content itself.

Within the library reference environment, the “relevant”information is likely to be authoritative, published, and avail-able from a variety of sources, either electronic or printed.This “relevance,” however, is “system generated” in terms ofrecall and precision from a particular collection beingsearched: it is not necessarily most relevant to the specificquery, which may have boundary conditions (prior searches,access issues, level of specificity) that are important to theuser. But the products of reference work, especially thoserelated to factual questions, have the acknowledged value ofproviding a very wide array of publicly available knowledgewhich can be re-used for future queries.

Within the strategic intelligence environment, however,the most-prized information is so-called “actionable intelli-gence,” that is, a novel piece of information that can, more-over, provide an opportunity for action on the part of theorganization. It is, therefore, less likely to be part of a pub-lished collection, except in the case of previously over-looked items of information that may provide a particularinsight into an organizational opportunity or dilemma.

Distribution

With the exception of certain specialized archives andlibraries, the initial results or a referral elsewhere is also theend point for archival and library distribution processes. Inthe intelligence environment, however, there is often aprocess of preparing the information in particular ways tobest fit the requirements of the end user. This may involveproviding assessments as to the accuracy of the informationprovided, connecting the new information to previouslydelivered intelligence, and analyzing the intelligence ina variety of ways. This “value-adding” can become evenmore important as the skills and knowledge of the intelli-gence analyst increase. The preparation of “intelligencebriefings” is a common final assignment in the intelligenceclasses.

Evaluation

Another key distinction between the reference specialistin archives and libraries and the intelligence specialist is theevaluation of products and processes. While reference evalu-ation focuses on the success of defining and fulfilling aquery with respect to the available collection resources

(hence the emphasis on “recall” and “precision” informationretrieval metrics), the intelligence evaluation focuses on thepragmatic utility of the ultimate results as “actionable intel-ligence.” There is no emphasis on evaluating a “collection”because items may be drawn from a variety of sources,which may or may not be systematically “collected,” andsome sources may be known to be of much less credibilitythan others. The key similarity in evaluation among all these,however, is that evaluation tends to focus on aggregateactivities rather than on a single product or transaction.

Defining the Relevance Work Domain

The final part of our analysis involved using Tennis’s“axes” to define which of the previously defined areas ofspecialization could be seen to “intersect” across the differ-ent agency types, and which were “focal” ones that could beseen as definitive ones that were determined by (and, thus,also determined) particular agency type. Although theseaspects of relevance may be opaque to the individual “user,”they are critical to the overall knowledge domain and, pre-sumably, the knowledge ecology as a whole. Table 3 belowshows the intersecting and focal areas of relevance workwithin these three domains.

Our findings agree with Hjørland’s proposition thatsystems, task, time, and assessor do define the nature ofrelevance for the individual. However, we also find thatrelevance in the aggregate (that is, for multiple users) ismediated by certain characteristics of the agency and its“relevance workers.” These include:

• For system: the “ethos” that pervades the individual agency.

• For task: the “query modes” that are made available throughthe individual agency.

• For time: the “time values” that are characteristic of theindividual agency.

• For assessment: the “product” that can result through use ofthe individual agency.

The “focal point” or defining dimensions are shownseparately in Table 4.

Conclusion

Retaining Hjørland’s original “pragmatic” conceptionof relevance as being sociocognitively accomplished by“system, time, task, and assessor,” our study adds to hismodel by identifying more substantive dimensions of “rel-evance work.” These would include the system’s ethos asdetermining the information being made accessible to users,the varying time values of the information required andprovided, the different assessments being made of the infor-mation and its uses, and, finally, the strengths and limitationsof the various ways in which information can and cannot bequeried on behalf of user tasks.

We propose that Hjørland’s map of the knowledgedomain ecology (2002a, p. 447) would benefit by replacing


his static black box of the library with the more detailedstructure shown in Figure 1. We show with these permeablelayers that although only the outermost layer of the blackbox (the query mode provided for a particular user’s task) isseen directly by the user, this as well as product assessmentand time value are mediated by the system’s underlyingethos, which is an integral set of sociocogntive processesembedded inside its “relevance workings.”

As the query mode interfaces to the library black boxevolve into primarily virtual rather than purely physicalforms (e.g., Kennedy, 2011), understanding its philosophiesand functions in new environments becomes even more criti-cal. We suggest that adding these dimensions of relevancework within intermediating information agencies providesnew insights for Hjørland’s reconceptualization of thesociocognitive/subject domain approach to relevance. Addi-tional study of the relevance work domain beyond the edu-cational settings and information agencies described herewould also be beneficial to a fuller understanding of anysubject domain. And, finally, as it is the aggregated valueover time of all relevance work that underlies the extraordi-nary value of the variety of subject domains that it supportsand serves, opening the black box of relevance work evenwider will help in better understanding its contributions tothe pragmatic functioning of “relevance” throughout theknowledge ecology.

TABLE 3. Relevance work domain foci and intersections.

Domain: Areas of modulationD

omai

n:A

reas

ofsp

ecia

lizat

ion

System-specificEthos Focal point/nonintersecting:

Protection of sanctity of records as evidence (archives)Protection of users’ privacy paramount (libraries)Protection of integrity of investigation (intelligence organizations)

Interaction Intersecting: interactions with clients can be sporadic, variable, or regularInstruction Intersecting: Can range from minimal through exhaustiveTask-specificQuery Modes Focal point/nonintersecting:

Queries are limited to particular collection (archives)Queries are usually limited to particular collection (libraries)Queries are not limited to particular collection (intelligence organizations)

Expertise Intersecting: search skills obligatory, required levels of analytic, expository, and collection/contextual knowledge varyTools Intersecting: Can be document based, collection based, and/or source basedTime-specificTime Focal point/nonintersecting:

Primary value placed on past information (archives)Primary value placed on current information (libraries)Primary value placed on future information (intelligence organizations)

Effort Intersecting: System effort exertion varies according to organizational policiesNeeds assessment Intersecting: Can range from self-service only through repeated interviews over timeAssessment-specificProduct Focal point/nonintersecting:

Authenticated evidence (archives)Relevant information or referral (libraries)Actionable intelligence (intelligence organizations)

Distribution Intersecting: Access provided to document, information, or intelligence item, depending on contextEvaluation Intersecting: Aggregated analysis of numbers/types of requests and fulfillment success

TABLE 4. Defining dimensions of relevance work for informationagencies.

The defining dimensions of three relevance work domains

System-specificEthos Focal point:

Protection of sanctity of records as evidence(archives)

Protection of users’ privacy paramount (libraries)Protection of integrity of investigation (intelligence

organizations)Task-specificQuery modes Focal point:

Queries are limited to particular collection (archives)Queries are usually limited to particular collection

(libraries)Queries are not limited to particular collection

(intelligence organizations)Time-specificTime Focal point:

Primary value placed on past information (archives)Primary value placed on current information

(libraries)Primary value placed on future information

(intelligence organizations)Assessment-specificProduct Focal point:

Authenticated evidence (archives)Relevant information or referral (libraries)Actionable intelligence (intelligence organizations)


Acknowledgments

The authors wish to thank two anonymous reviewers fortheir insightful comments on an earlier draft of this article.

References

Adkins, D., & Erdelez, S. (2006). An exploratory survey of referencesource instruction in LIS courses. Reference & User Services Quarterly,46(2), 50–60.

Agosto, D.E., Rozaklis, L., MacDonald, C., & Abels, E. (2011). A model ofthe reference and information service process: An educators’ perspective.Reference & User Services Quarterly, 50(3), 235–244.

Aguilar, F.J. (1967). Scanning the business environment. New York:Macmillan.

Allen, T.J. (1977). Managing the flow of technology: Technology transferand the dissemination of technological information within the R&Dorganization. Cambridge, MA: MIT Press.

American Library Association. (2008). Standards for accreditation of mas-ter’s programs in library & information studies. Chicago, IL: AmericanLibrary Association.

Atkinson, R. (1996). Library functions, scholarly communication, and thefoundations of the digital library: Laying claim to the control zone.Library Quarterly, 66(3), 239–265.

Berner, R.C. (1983). Archival theory and practice in the United States:A historical analysis. Seattle, WA: University of Washington Press.

Bernhardt, D.C. (1999). Consumer versus producer: Overcoming the dis-connect between management and competitive intelligence. CompetitiveIntelligence Review, 10(3), 19–26.

Broadway, M.D., & Smith, N.M. (1989). Basic reference courses in ALA-accredited library schools. Reference Librarian, 25–26, 431–448.

Brosman, C.S. (1998). The case for (and against) departmental syllabi.Academic Questions, 11(4), 58–65.

Buckland, M.J. (1991). Information as thing. Journal of the AmericanSociety of Information Science, 42(5), 351–360.

Burkholder, J.M. (2010). Redefining sources as social acts: Genretheory in information literacy instruction. Library Philosophy andPractice. Retrieved from http://www.webpages.uidaho.edu/~mbolin/burkholder.htm

Choo, C.W. (2006). The knowing organization: How organizations useinformation to construct meaning, create knowledge, and make decisions(2nd ed.). New York: Oxford University Press.

Cox, R.J. (2007). Revisiting the archival finding aid. Journal of ArchivalOrganization, 5(40), 5–32.

Cox, R.J., Yakel, E., Wallace, D., Bastian, J.A., & Marshall, J. (2001).Archival education in North American library and information scienceschools. Library Quarterly, 71(2), 141–194.

Cuadra, C., & Katter, R. (1967). Opening the black box of “relevance.”Journal of Documentation, 23(4), 291–303.

Dana, J.C. (1920). A library primer. Boston, MA: The Library Bureau.De Alwis, G., Majid, S., & Chaudry, A.S. (2006). Transformation in man-

agers’ information seeking behavior: A review of the literature. Journalof Information Science, 32(4), 362–377.

Diamond, E. (1994). The archivist as forensic scientist: Seeing ourselves ina different way. Archivaria, 38, 139–154.

Diamond, R.M. (2008). Designing and assessing courses and curricula: Apractical guide (3rd ed.). San Francisco, CA: Jossey-Bass.

Donovan, J.M. (2008). Skating on thin intermediation: Can librariessurvive? Legal Reference Services Quarterly, 27(2–3), 95–116.

Duff, W. (2010). Archival mediation. In T. Eastwood & H. MacNeil (Eds.),Currents of archival thinking (pp. 115–136). Santa Barbara, CA: Librar-ies Unlimited.

Duff, W., & Fox, A. (2006). “You’re a guide rather than an expert”:Archival reference from an archivist’s point of view. Journal of theSociety of Archivists, 27(2), 129–153.

Fahey, L. (2007). Connecting strategy and competitive intelligence:Refocusing intelligence to produce critical strategic inputs. Strategy &Leadership, 35(1), 4–12.

Fahey, L., & Herring, J. (2007). Intelligence teams. Strategy & Leadership,34(1), 13–20.

Fox, R. (2010). Mining the digital library. OCLC Systems & Services,26(4), 232–238.

Goldman, E. (2010). The end of the search engine utopianism. In B. Szoka& A. Marcus (Eds.), The next digital decade: Essays on the future of theInternet (pp. 461–473). Retrieved from http://www.techfreedom.org/publications

Green, S.S. (1876). Personal relations between librarians and readers.Library Journal, 1, 74–81.

FIG. 1. Opening the black box of relevance work.


Grimmelmann, J. (2010). Some skepticism about search neutrality. InB. Szoka & A. Marcus, (Eds.), The next digital decade: Essays onthe future of the Internet (pp. 435–459). Retrieved from http://www.techfreedom.org/publications

Habanek, D.V. (2005). An examination of the integrity of the syllabus.College Teaching, 53, 62–64.

Hernon, P., & McClure, C. (1986). Unobtrusive reference testing: The 55%rule. Library Journal, 111(7), 37–41.

Herring, J.P. (1999). Key intelligence topics: A process to identify anddefine intelligence needs. Competitive Intelligence Review, 10(2), 4–14.

Hjørland, B. (2002a). Domain analysis in information science: Elevenapproaches — traditional as well as innovative. Journal of Documenta-tion, 58(4), 422–462.

Hjørland, B. (2002b). Epistemology and the socio-cognitive perspective ininformation science. Journal of the American Society for InformationScience & Technology, 53(4), 257–270.

Hjørland, B. (2010). The foundation of the concept of relevance. Journal ofthe American Society for Information Science & Technology, 61(2),217–237.

Hjørland, B., & Albrechtsen, A. (1995). Toward a new horizon in informa-tion science: Domain-analysis. Journal of the American Society forInformation Science, 46(6), 400–425.

Huvila, I. (2008). Participatory archive: Towards decentralized curation,radical user orientation, and broader contextualization of records man-agement. Archival Science, 8, 15–36.

Ingwersen, P., & Järvelin, K. (2005). The turn: Integration of informationseeking and retrieval in context. Dordrecht, The Netherlands: Springer.

Johnson, L.K. (2003a). Bricks and mortar for a theory of intelligence.Comparative Strategy, 22(1), 1–28.

Johnson, L.K. (2003b). Preface to a theory of strategic intelligence. Inter-national Journal of Intelligence and Counterintelligence, 16(4), 638–663.

Kennedy, S. (2011). Farewell to the reference librarian. Journal of LibraryAdministration, 51(4), 319–325.

Kwasnik, B. (1999). The role of classification in knowledge representationand discovery. Library Trends, 48(1), 22–47.

Kwon, H. (2010). Systems-centered versus user-centered librarianship: Acognitive sociological view. In M.L. Radford & R.D. Lankes (Eds.),Reference renaissance: Current and future trends (pp. 99–113). NewYork: Neal-Schuman Publishers.

Lankes, R.D. (2008). Credibility on the Internet: Shifting from authority toreliability. Journal of Documentation, 64(5), 667–686.

Laqueur, W. (1985). World of secrets: The uses and limits of intelligence.New York, NY: Basic Books.

Luhn, H.P. (1958). A business intelligence system. IBM Journal, 2(4),314–319.

Mann, T. (2007). The Peloponnesian war and the future of reference,cataloging, and scholarship in research libraries. Report preparedfor AFSCME 2910. Retrieved from http://www.guild2910.org/Pelopponesian%20War%20June%2013%202007.pdf

Markey, K. (2004). Current educational trends in the information andlibrary science curriculum. Journal of Education for Library and Infor-mation Science, 45(4), 317–339.

McArthur, T. (1986). Worlds of reference: Lexicography, learning andlanguage from the clay tablet to the computer. Cambridge UK: Cam-bridge University Press.

McGonagle, J.J. (2007). An examination of the “classic” CI model. Journalof Competitive Intelligence & Management, 43(20), 71–86.

Meyer, D.Z., & Avery, L. (2008). Excel as a qualitative data analysis tool.Field Methods, 21(1), 91–112.

Miller, W. (2007). Reference services over the past century: Moving fromthe center to the fringes. The Reference Librarian, 48(2), 3–7.

O’Connor, L.G. (2011). The education of reference librarians: A detailedsurvey and analysis. In D. Zabel (Ed.), Reference reborn: Breathing newlife into public services librarianship (pp. 317–337). Santa Barbara, CA:Libraries Unlimited.

Parkes, J., & Harris, M.B. (2002). The purposes of a syllabus. CollegeTeaching, 50(2), 55–61.

Pawley, C. (2003). Information literacy: A contradictory coupling. LibraryQuarterly, 73(4), 422–452.

Powell, R.R., & Raber, D. (1994). Education for reference/informationservice: A quantitative and qualitative analysis of basic reference courses.Reference Librarian, 43, 145–172.

Radford, M.L. (1999). The reference encounter: Interpersonal communica-tion in the academic library. Chicago: American Library Association.

Radford, M.L., & Lankes, R.D. (Eds.) (2010). Reference renaissance:Current and future trends. New York: Neal-Schuman.

Ranganathan, S.R. (1931). The five laws of library science. Madras, India:Madras Library Association.

Rettig, J. (2006). Reference service: From certainty to uncertainty.Advances in Librarianship, 30, 100–143.

Saracevic, T. (1975). Relevance: A review of and a framework for thethinking on the notion in information science. Journal of the AmericanSociety for Information Science, 26(1), 321–343.

Saracevic, T. (2006). Relevance: A review of the literature and a frameworkfor thinking on the notion in information science. Part II. Advances inLibrarianship, 30, 3–71.

Saxton, M.L., & Richardson, J.W. (2002). Understanding reference trans-actions: Transforming an art into a science. San Diego, CA: AcademicPress.

Schellenberg, T.R. (1956). Modern archives: Principles and techniques.Chicago: University of Chicago Press.

Segev, E. (2010). Google and the digital divide: The bias of online knowl-edge. Oxford, UK: Chandos Publishing.

Shachaf, P. (2009). Social reference: Toward a unifying theory. Library &Information Science Research, 32, 66–76.

Shelfer, K.M., & Goodrum, A. (2000). Competitive intelligence as anextension of library education. Journal of Education for Library andInformation Science, 41(4), 353–361.

Swigger, B.K. (2010). The MLS project: An assessment after 60 years.Lanham MD: Scarecrow Press.

Szostak, R. (2010). Comment on Hjørland’s concept theory. Journal of theAmerican Society for Information Society & Technology, 61(5), 1076–1077.

Taylor, R.S. (1986). Value-added processes in information systems.Norwood, NJ: Ablex Publishing.

Tennis, J.T. (2003). Two axes of domains for domain analysis. KnowledgeOrganization, 30(3/4), 191–195.

Trueswell, R. (1969). User circulation satisfaction vs. size of holdingsat three academic libraries. College and Research Libraries, 30, 204–213.

Tyckoson, D.A. (2011). Issues and trends in the management of referenceservices: A historical perspective. Journal of Library Administration,51(3), 259–278.

White, H.S. (1992). The reference librarian as information intermediary.The Reference Librarian, 17(37), 23 - 35.

Yakel, E. (2010). Thinking inside and outside the boxes: Archival referenceservices at the turn of the century. Archivaria, 49, 140–160.

Yakel, E., & Torres, D.A. (2003). AI: Archival intelligence and user exper-tise. American Archivist, 66, 51–78.

Zabel, D. (2011). Introduction. In D. Zabel (Ed.) Reference reborn: Breath-ing new life into public services librarianship (pp. xiii–xx). SantaBarbara, CA: Libraries Unlimited.
