H. Lundbeck A/S 21 Apr 2023 1

Assessing the effectiveness of your current search and retrieval function

Anna G. Eslau, Information Specialist, H. Lundbeck A/SMarianne Lykke Nielsen, Associate Professor, Royal School of Library and

Information Science

Case story evaluating human metadata indexing versus automatic query expansion using a

corporate thesaurus

H. Lundbeck A/S 21 Apr 2023 2

Agenda

• Motivation• Case study

– Research partners– Purpose– Test design– Findings– Conclusions

• Summing up

H. Lundbeck A/S 21 Apr 2023 3

Motivation

• A lot of money has been invested – but does our current search and retrieval function perform as expected?

• An advanced and time consuming indexing task has been laid upon our end users – but is our current indexing strategy effective?

• Do we have - as high quality - alternatives to manual indexing?

H. Lundbeck A/S 21 Apr 2023 4

Agenda

• Motivation• Case study

– Research partners– Purpose– Test design– Findings– Conclusions

• Summing up

H. Lundbeck A/S 21 Apr 2023 5

Case study - Research partners

• H. Lundbeck A/S– Pharmaceutical company– 5000 employees, in > 40 countries– Information systems with electronic documents– Corporate thesaurus– Users and search requests

• Royal School of Librarianship– Thesaurus research expertise– Domain knowledge from former research project

• Ensight A/S– Verity K2 search engine and Intelligent Classifier– Technical expertise

H. Lundbeck A/S 21 Apr 2023 6

Purpose of case study

To evaluate1. Information retrieval based on

controlled, human indexing (controlled metadata)

2. Information retrieval based on full-text indexing, with thesaurus-based automatic query expansion

H. Lundbeck A/S 21 Apr 2023 7

Case study – Retrieval system and indexing policy

• Electronic document management system (EDMS) and bibliographic information system containing research documentation

• Indexing policy– Written indexing policy– Mandatory training of indexers – Corporate Thesaurus– Human, controlled indexing– Topical checklist/Facetted indexing

• Searching by controlled metadata and full-text• Domain specific thesaurus containing 5,500

concepts and 16,000 terms

EDMS 1/2 - Indexing

EDMS 2/2 – Searching

H. Lundbeck A/S 21 Apr 2023 10

Lundbeck Thesaurus 1/3

H. Lundbeck A/S 21 Apr 2023 11

Lundbeck Thesaurus 2/3

H. Lundbeck A/S 21 Apr 2023 12

Lundbeck Thesaurus 3/3

H. Lundbeck A/S 21 Apr 2023 13

Agenda

• Motivation• Case study

– Research partners– Purpose– Test design– Findings– Conclusions

• Summing up

H. Lundbeck A/S 21 Apr 2023 14

Test design - Retrieval performance of different search strategies

• Three different search strategies were evaluated:1. Searches based on natural language

(words from original request) in full text 2. Searches based on natural language in full

text expanded with words from thesaurus (query expansion with synonyms and narrower terms)

3. Searches based on (manually assigned) controlled keywords in selected metadata fields

H. Lundbeck A/S 21 Apr 2023 15

Test design - Query expansion

• Search for information about intravenous administration of a drug AND Alzheimer’s disease:

’Intravenous OR IV OR Intravenously OR…’AND’Alzheimer’s disease OR Alzheimer’s disorders

OR Alzheimer type dementia OR…..’

H. Lundbeck A/S 21 Apr 2023 16

Lundbeck Thesaurus

H. Lundbeck A/S 21 Apr 2023 17

Test design - Test persons and retrieval system

• Persons– Query expansion tests were carried out by the

thesaurus manager and did not involve end-users– Evaluation of search results were carried out by end

users – 4 subject experts (Medical advisers) who had formerly answered the search requests

• System– Verity K2 search system was used as test retrieval

system for the query expansion test work– Original document management systems were used

as retrieval system for the metadata searches

H. Lundbeck A/S 21 Apr 2023 18

Test design - Test thesaurus

• The Lundbeck Thesaurus was the test thesaurus. The thesaurus formed basis for query formulations: - Synonyms and narrower terms were picked

from the thesaurus for the test searches based on expansion of natural language in full text searches

- Preferred keywords were picked from the thesaurus for the test searches based on controlled keywords in selected metadata fields.

H. Lundbeck A/S 21 Apr 2023 19

Test design - Test collection

• 25,384 document objects from two different sources– 24,369 document objects from a

bibliographical (BRS) information system (internal research reports and published research articles)

– 1015 documents from the full-text EDMS system (internal research reports)

H. Lundbeck A/S 21 Apr 2023 20

Test design - Search requests

• 10 search requests were selected from a set of searches which in real life had been carried out in the corporate information systems

Work task 7: You are a medical reviewer. A physician has contacted you. He would like to have data on the use of Citalopram and Reboxetine together to treat resistant depression. He wants any reporting of possible interactions.

Indicative request: Find reports, papers or case stories that investigate the possible interaction of Citalopram and Reboxetine

on resistant depression

H. Lundbeck A/S 21 Apr 2023 21

Agenda

• Motivation• Case study

– Research partners– Purpose– Test design– Findings– Conclusions

• Summing up

H. Lundbeck A/S 21 Apr 2023 22

Findings – Performance

Recall (% relevant docs retrieved out of total no. of relevant docs)

Searchstrategy

SJ1 SJ2 SJ3 SJ4 SJ5 SJ6 SJ7 SJ8 SJ9 SJ10

Full-text 42 52 88 38 79 54 39 3 12 7

Full-text with

QE (syn)

64 68 100 76 89 100 39 100 100 68

Full-text with

QE (syn, nt)

100 90 100 87 89 100 39 100 100 73

Metadata 0 0 0 33 29 61 100 1 0 45

Precision (% relevant docs out of all retrieved docs) went down from 33% to 24% with query expansion

SJ = Search Job, QE = Query Expansion

Findings – Human indexing problemsIndexing problems Frequency

(%)N = 156

Explanations

1. Conceptual analysis

A1 Omission of topic 69 • Indexers fail to remember facets and topics that are not explicitly mentioned in indexing policy or checklist• Indexing policy recommend to check specific document sections such as title, table of content, etc. why indexers, especially in long documents, tend to omit topics from other document sections

÷

A2 Misinterpretation and wrong perspective of topic

14 • Indexers misunderstand topic due to lack of topical and domain knowledge ÷

A3 Omission of implicit topic

2 • Difficult for indexers to determine degree of topical interpretation and domain-orientation ÷

2. Translation

B1 Topic indexed at BT level

7

B2 Topic indexed with incorrect keyword

8 • Indexers misunderstand meaning and use of keywords ÷

H. Lundbeck A/S 21 Apr 2023 24

Findings – Other metadata

• Topical retrieval and situational relevance ranking - the importance of contextual parameters– Document type– Publication year– Source– Language– Author

H. Lundbeck A/S 21 Apr 2023 25

Findings – Thesaurus

• Thesaurus– Relevant synonyms (acronyms with multiple

meanings should be omitted)– Logical hierarchies– High topical relevance

H. Lundbeck A/S 21 Apr 2023 26

Findings – Documents and search requests

• Document collection– OCR scanned documents may contain errors =>

false positive hits– Large (>100 pages) full text documents lower

precision (irrelevant hits)

• Search requests– If people are searching using very general terms, QE

will be extremely complicated/extensive, the more levels of QE we choose to add

– Different types of facets result in• Different relevance assessment according to

document types• Different recall in metadata search

H. Lundbeck A/S 21 Apr 2023 27

Findings – Search software

• Search software settings are important– Stemming– Case sensitivity– Character sensitivity (())– Number of search terms allowed– Zoning

H. Lundbeck A/S 21 Apr 2023 28

Agenda

• Motivation• Case study

– Research partners– Purpose– Test design– Findings– Conclusions

• Summing up

H. Lundbeck A/S 21 Apr 2023 29

Conclusion – Thesaurus and QE

• A domain specific thesaurus are well suited for QE

• QE improves recall but decreases precision

• QE with synonyms only are in most cases sufficient

H. Lundbeck A/S 21 Apr 2023 30

Conclusion - Search result display

• Users want to see all hits (recall is important)

• Manual sorting of search results by (other than topical) metadata is requested by the users

• Ranking based on e.g. zoning is not always useful

H. Lundbeck A/S 21 Apr 2023 31

Conclusion – Indexing policy

• Difficult to obtain complete, accurate and exhaustive human indexing

• Findings suggest that searching for specific topics should be based on full-text indexing, supported by thesaurus based query expansion

• Human indexing should focus on few, important, well-defined topics, e.g. used to develop taxonomies for broad browsing

• Non-Topical context metadata are important in assessment of document relevance– Document type– Publication year– Source– Language– Author

H. Lundbeck A/S 21 Apr 2023 32

Conclusion – Implications for Lundbeck

• Lundbeck Thesaurus has been integrated with bibliographic information system to perform automated QE

• EDMS upgrade planned where QE should be possible

• OCR scanning of existing documents are considered

• Metadata on document types in EDMS are evaluated and under revision (simplified)

• New models on how to add metadata are considered (dictionaries)

• New indexing tools for the users are developed (indexing keys)

H. Lundbeck A/S 21 Apr 2023 33

Agenda

• Motivation• Case study

– Research partners– Purpose– Test design– Findings– Conclusions

• Summing up

H. Lundbeck A/S 21 Apr 2023 34

Summing up

• If your current search and retrieval function does NOT perform as expected, your organisation may loose important information

• You may have an indexing strategy (which is good…) but evaluation may reveal that the resource investments could be used even better

• Evaluation is important, it may save your organisation money over time