Improving Efficiencies Through Cost-Benefit Analysis of Metadata

Creation

Joyce Celeste ChapmanNCSU Libraries Fellow

Metadata and Digital Object roundtable: lightning talks

SAA 2010 Annual (August 11, 2010)

Cost and value of metadata

• We assume there to be inherent value in the work we do with metadata

• Libraries are lacking metrics for measuring cost and value of metadata

• Problem: unlike for-profits, we cannot model on cost versus sales

Operational definitions of “value”

• We must identify our own operational definitions of value against which we can evaluate cost

• Examples: – Value as use/circulations– Value as discovery success– Value as the ability to operate successfully on

the open Web

Operational definitions of “value”

• We must identify our own operational definitions of value against which we can evaluate cost

• Examples: – Value as use/circulations

– Value as discovery success– Value as the ability to operate successfully on

the open Web

Users, archival metadata, and value through discovery success

• User study: how frequently do users use certain elements in information discovery (specifically: when determining the relevancy of resources returned in results list)?

• Time study: how long do processors spend creating these same metadata elements?

Methodology

Part I: User study• 10 advanced archival researchers• 5 subjective information discovery tasks

Part II: Time study• 14 collections, 9 processors (5 archivists and 4

catalogers), 2 partner institutions*• Time data collected to the minute for metadata

creation

* NCSU and Avery Research Institute for African American History and Culture, SC

Which elements were studied?

1. Abstract2. Biographical / Historical Note3. Scope and Content Note (collection-level)4. Subject Headings5. Collection Inventory6. Other*

* Catch-all category for all other elements. Required to analyze ratios.

Disclaimer

• We have very few data sets for timing data (14 split among 3 groups). This is not enough to be sure of anything! More data needs to be tracked before we can be sure that patterns we see for are accurate.

Analyzing usability findings

Participant behavior ranked elements in the following order from most used to least used:

1. Collection Inventory2. Abstract3. Subject Headings4. Scope Content5. Biographical Note

Behavioral scores by order visited

Behavioral scores by order visited

Problematic metadata overlap

• Some participants were confused about the overlap between content in Abstract and Scope and Content Note

• Out of all the instances in which participants navigated to the Abstract, 64% of the time, they never subsequently looked at the Scope and Content Note

Timing analysisAverage ratios

Timing analysisAverage ratios

Timing analysisAverage ratios

Timing analysisReal time

Cost to Value? Issues and Questions

1. Compared to use, a disproportionately high % of time is spent creating Biographical Notes.– Data acquires meaning in context: do we care if

we spend a high % of metadata creation time on the Biographical Note if that translates to real numbers that are not “significant” by our institutional standards?

– We might want to regulate time spent on metadata, but only for metadata creation that exceeds a certain real time baseline.

Data acquires meaning in context

• Examples in real numbers from NCSU:– Collection A

Biographical Note = 51% of total metadata time = 24 minutes

– Collection BBiographical Note = 43% of total metadata time

= 9.8 hours

Cost to Value? Issues and Questions

2. Abstract is high value, users often go there first, and some use only the Abstract and never Scope and Content Note.

3. Users emphasize importance of Collection Inventory. Are we spending enough time there to equal the value rating?

Next steps: study other facets of value

• This study examined only one facet of value. In order to form a complete picture of value for metadata, further studies must be conducted.

Thanks!

For more information, a longer presentation on this study will be given at the SAA Description Section meeting on Friday 1:00-3:00pm.

Contact:

Joyce_Chapman@ncsu.edu