tried and tested? evaluating data entry consistency in the component model prototype brit hopmann,...
TRANSCRIPT
Tried and Tested?Evaluating Data Entry Consistencyin the Component Model Prototype
Brit Hopmann, July 1, 2008
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Presentation
• Background & aim consistency check
• Images & data
• Analysis & results (in 3 steps)
• Problems & possible solutions
• Conclusion & discussion
data entry
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Background & aim consistency check
Background
• Problems with data entry ambiguity
• Consequences: decreased recall and precision
Aim
• Check and improve data entry consistency
• Solutions for existing problems?
• Changes?
data entry
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Images
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Data
• 5 images
• 25 features (+ 2 discarded)
• 6 respondents
Analysis in 3 steps
• Categorizing the answers (per image / per feature)
• Analysis per image and per feature
• Problems and possible solutions
Analysis
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 1
example: answers for image4
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 1
Categorizing the answers (per image / per feature)
3 categories
category consistency
times percentage
no problem 57 45,6%
indecisive 16 12,8%
problem 52 41,6%
There were many problematic descriptions.
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 2
Summary per watermark (5 images)
None of the images was described unanimously.
image no prob ≈ problem
per image
image 1 15 2 8 fine
image 2 13 3 9 fine
image 3 10 3 12 problematic
image 4 11 3 11 indecisive (≈)
image 5 8 5 12 problematic
total 57 16 52
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 2
Summary per feature (25 features + 2 discarded)
Described consistently 5-7 features*
3 of 5 images not consistently described 10-8 features*
* Depending on quality of definitions
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 3
How problematic are specific features?
1. Is or might the feature be problematic with other watermarks?
2. What caused the problem and can it be solved easily?
-> image quality, definition, subjective description
3. What is the distinctive value of a particular feature?
4. Is there a possible solution?
5. Which problems will remain problematic after having applied a specific solution?
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Analysis – step 3
What could contribute to a solution?
• Grouping
• Allow assigning multiple features
• Drawback: increased decision effort
• Extra tool
• For instance graphical indicator of angle
• Drawback: exceptions such as rotated scans
• Clarify definitions
• For instance description of additional elements
• Relatively straightforward to apply
• Otherwise
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Crucial Question
Which features will still pose a problem with a particular solution applied?
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Conclusion
Problems that will eventually remain
• Eyes: alignment, distance, position
• Horns: endings, orientation, inside
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Conclusion
Remaining problems
• Ears: endings, roundness, width
• Eyes: shape
• Head: shape
• In-between horns: shape
• Nose: concomitance with other elements, shape
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Discussion
Features that evoke subjective descriptions
(shape!) cause problems
Thinking about possible changes in the model
• Clarify certain issues: clear definitions, instruction
• Expand model: extra tool, grouping
• Simplify model: adjust or take out features
data
Reconsider role of component model within Bernstein
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?
Discussion
Questions and Comments?
Koninklijke Bibliotheek – National Library of the Netherlands
Tried and Tested?