Online Virtual Chat Library Online Virtual Chat Library Reference Service:Reference Service:

A Quantitative and Qualitative A Quantitative and Qualitative AnalysisAnalysis

Dr. Dave HarmeyerDr. Dave HarmeyerAssociate ProfessorAssociate Professor

Chair, Marshburn Memorial LibraryChair, Marshburn Memorial LibraryAzusa Pacific UniversityAzusa Pacific University

OutlineOutline

1.1. Purpose of the StudyPurpose of the Study

2.2. Research QuestionsResearch Questions

3.3. MethodologyMethodology

4.4. Variables (I.V., D.V.)Variables (I.V., D.V.)

5.5. Significant FindingsSignificant Findings

6.6. ConclusionsConclusions

7.7. Questions & AnswersQuestions & Answers

Purpose of the StudyPurpose of the Study

Virtual chat reference augments face-to-face Virtual chat reference augments face-to-face reference interviewreference interview

Library reference literature lacks research-Library reference literature lacks research-based findings to back up recommended based findings to back up recommended practicespractices

This study fills the void with a theoretical This study fills the void with a theoretical conceptual model based on an empirical conceptual model based on an empirical study of chat reference transactions study of chat reference transactions

Research QuestionsResearch Questions

1. What measurable indicators are found for virtual 1. What measurable indicators are found for virtual chat reference transactions, looking exclusively chat reference transactions, looking exclusively at data created from the chat reference at data created from the chat reference transcripts? transcripts?

2. Do published reference interview guidelines from 2. Do published reference interview guidelines from RUSA, a set of other strategies and the nature of RUSA, a set of other strategies and the nature of the query contribute to an accurate answer? the query contribute to an accurate answer?

3. What conceptual model of best practices can be 3. What conceptual model of best practices can be suggested by an analysis of the data? suggested by an analysis of the data?

MethodologyMethodology

Two-and-a-half years of archived academic library chat Two-and-a-half years of archived academic library chat transcripts using Krippendorff’s (2004) content analysis transcripts using Krippendorff’s (2004) content analysis

333 random transcripts from 2,500 333 random transcripts from 2,500 Analyzing 16 independent variables and their relationship Analyzing 16 independent variables and their relationship

with one dependent variable of an accurate reference with one dependent variable of an accurate reference answeranswer

Pearson correlations and ANOVA variance testsPearson correlations and ANOVA variance tests 120 virtual librarians at 43 American institutions120 virtual librarians at 43 American institutions 320 remote patrons accessing the service through one 320 remote patrons accessing the service through one

Southern California undergraduate/masters universitySouthern California undergraduate/masters university

Methodology (cont.)Methodology (cont.)

VariablesVariables

Research Question 1 answered:Research Question 1 answered: 16 independent variables 16 independent variables 1 dependent variable - question accuracy1 dependent variable - question accuracy Influence on question accuracyInfluence on question accuracy Observed in content analysis of chat Observed in content analysis of chat

transcriptstranscripts Derived from RUSA guidelinesDerived from RUSA guidelines Derived from literature reviewDerived from literature review

Quantitative IVsQuantitative IVs

1. Librarian’s initial contact time (1. Librarian’s initial contact time (hold timehold time, in , in seconds)seconds)

2. Total time of transaction (2. Total time of transaction (service timeservice time, in seconds), in seconds)

3. Longest time gap by librarian (in seconds)3. Longest time gap by librarian (in seconds)

4. Number of URLs co-browsed with the patron4. Number of URLs co-browsed with the patron

5. Keystrokes by librarian5. Keystrokes by librarian

6. Keystrokes by patron6. Keystrokes by patron

7. Keystrokes by both7. Keystrokes by both

Qualitative IVsQualitative IVs

1. The question’s difficulty (seven-point scale)1. The question’s difficulty (seven-point scale)2. Response to a patron’s “are you there” statements (scored 2. Response to a patron’s “are you there” statements (scored

as present, not present, not applicable or ambiguous when as present, not present, not applicable or ambiguous when coders disagreed)coders disagreed)

3. Librarian’s friendliness 3. Librarian’s friendliness 4. Lack of jargon 4. Lack of jargon 5. Use of open-ended questions 5. Use of open-ended questions 6. Use of closed and/or clarifying questions 6. Use of closed and/or clarifying questions 7. Librarian maintains objectivity 7. Librarian maintains objectivity 8. Asking if the question was answered completely 8. Asking if the question was answered completely 9. The type of question (seven categories: ready reference, 9. The type of question (seven categories: ready reference,

research question, library technology, request for materials, research question, library technology, request for materials, bibliographic verification, other and ambiguous for bibliographic verification, other and ambiguous for disagreements among coders)disagreements among coders)

Dependent VariableDependent Variable

Coders Qualitative JudgmentsCoders Qualitative Judgments Service Service QualityQuality

8 8 Librarian gave (or referred) patron to a single Librarian gave (or referred) patron to a single source with an accurate answer source with an accurate answer Excellent Excellent 7 7 Librarian gave (or referred) patron to more than Librarian gave (or referred) patron to more than one source, one of which provided an accurate one source, one of which provided an accurate answer answer Very good Very good 6 6 Librarian gave (or referred) patron to a single Librarian gave (or referred) patron to a single source which does not lead directly to an source which does not lead directly to an accurate answer but did serve as a preliminary accurate answer but did serve as a preliminary source source GoodGood 5 5 Librarian gave (or referred) patron to more Librarian gave (or referred) patron to more than one source, none of which leads directly than one source, none of which leads directly to an accurate answer but one which served asto an accurate answer but one which served as a preliminary source a preliminary source SatisfactorySatisfactory

Dependent Variable (cont.)Dependent Variable (cont.)

Coders Qualitative JudgmentsCoders Qualitative Judgments Service QualityService Quality 44 No direct accurate answer given, No direct accurate answer given,

referred to another person or institution referred to another person or institution Fair / poorFair / poor 33 No accurate answer (or referral) given No accurate answer (or referral) given

(e.g., “I don’t know”) (e.g., “I don’t know”) FailureFailure 22 Librarian gave (or referred) patron to aLibrarian gave (or referred) patron to a

single source which did not answer the single source which did not answer the question question UnsatisfactoryUnsatisfactory

11 Librarian gave (or referred) patron to Librarian gave (or referred) patron to more than one source, none of which more than one source, none of which answered the question answered the question Most Most

unsatisfactoryunsatisfactory

(Richardson and Reyes, 1995)(Richardson and Reyes, 1995)

Significant Findings Significant Findings SummarySummary

Research Question 2 answered: yesResearch Question 2 answered: yes 30 significant relationships (p < .05)30 significant relationships (p < .05) From 9 of 16 variablesFrom 9 of 16 variables 5 found in RUSA guidelines5 found in RUSA guidelines 4 found in other strategies or nature of 4 found in other strategies or nature of

online chat online chat

Significant Findings Significant Findings Answer AccuracyAnswer Accuracy

Answer Accuracy as Judged by Coders (N=331) Answer Accuracy as Judged by Coders (N=331) CriteriaCriteria Point Frequency Point Frequency % % Cum. %Cum. %Accurate Answer (single source) Accurate Answer (single source)

ExcellentExcellent 8.08.0 8888 26.626.6 26.6 (1/4)26.6 (1/4)Accurate Answer (mult. sources) Accurate Answer (mult. sources) 7.5 7.5 1818 5.45.4 32.032.0

Very goodVery good 7.07.0 6464 19.319.3 51.3 (1/2)51.3 (1/2)Preliminary Source (single source) 6.5Preliminary Source (single source) 6.599 2.72.7 54.054.0

GoodGood 6.06.0 3333 10.010.0 64.0 (2/3)64.0 (2/3)Preliminary Source (mult. sources)Preliminary Source (mult. sources) 5.55.5 77 2.12.1 66.166.1

SatisfactorySatisfactory 5.05.0 3535 10.610.6 76.7 (3/4)76.7 (3/4)No Accurate Answer, referredNo Accurate Answer, referred 4.54.5 1111 3.33.3 80.080.0

Fair / poorFair / poor 4.04.0 5757 17.217.2 97.297.2““I don’t know,” no referralI don’t know,” no referral 3.53.5 22 0.60.6 97.897.8

FailureFailure 3.03.0 33 0.90.9 98.798.7Not Accurate (single source)Not Accurate (single source) 2.52.5 22 0.60.6 99.399.3

UnsatisfactoryUnsatisfactory 2.02.0 22 0.60.6 99.999.9Not Accurate (multiple sources)Not Accurate (multiple sources) 1.51.5 00 0.00.0 99.999.9

Most unsatisfactoryMost unsatisfactory 1.01.0 00 0.00.0 99.999.9

Significant FindingsSignificant FindingsBest PracticesBest Practices

Research Question 3 answered: yesResearch Question 3 answered: yesA Conceptual Model for Reference Chat AccuracyA Conceptual Model for Reference Chat Accuracy

minor plus est (less is more)minor plus est (less is more)

1. Keep time gaps between sending responses to 1. Keep time gaps between sending responses to patrons to no more than one-and-a-half minutespatrons to no more than one-and-a-half minutes

2. Maintain a total chat transaction time of eight 2. Maintain a total chat transaction time of eight minutes or lessminutes or less

3. Keep total keystrokes per transaction to within six 3. Keep total keystrokes per transaction to within six and-a-half lines of text (or 480 characters).and-a-half lines of text (or 480 characters).

4. Expect to type twice as many characters as the 4. Expect to type twice as many characters as the patronpatron

Significant FindingsSignificant FindingsBest Practices (cont.)Best Practices (cont.)

5. Be careful about beginning the question 5. Be careful about beginning the question negotiation segment of the reference interview negotiation segment of the reference interview with an open question unless the nature of the with an open question unless the nature of the patron’s question explicitly calls for one. patron’s question explicitly calls for one.

6. Ask closed or clarifying questions when 6. Ask closed or clarifying questions when appropriateappropriate

7. At the end of the reference transaction, ask “Does 7. At the end of the reference transaction, ask “Does this completely answer your question?”this completely answer your question?”

8. Even moderately difficult questions decrease 8. Even moderately difficult questions decrease answer accuracy and not just the medium to high answer accuracy and not just the medium to high difficult questionsdifficult questions

Significant FindingsSignificant Findings1. Gaps1. Gaps

Keep time gaps between sending responses Keep time gaps between sending responses to patrons to not much more than one-and-to patrons to not much more than one-and-a-half minutesa-half minutes

Reinforces RUSA’s Reinforces RUSA’s interestinterest guideline (2.6), guideline (2.6), time away from the patron short, maintain time away from the patron short, maintain “word contact” (RUSA, June 2004)“word contact” (RUSA, June 2004)

Anything nearing two minutes or higher is Anything nearing two minutes or higher is likely to decrease answer accuracylikely to decrease answer accuracy

Significant Findings Significant Findings 1. Gaps1. Gaps

Longest Librarian Gap Longest Librarian Gap

Quartiles (min.) Acc. Mean Sig. of Diff. (p)_____ Quartiles (min.) Acc. Mean Sig. of Diff. (p)_____

1st 0 -- 1st 0 -- 1.851.85 6.686.68

2nd 2nd 1.87 – 2.831.87 – 2.83 5.975.97 .016.016 (1st & 2nd) (1st & 2nd)

3rd 2.85 – 4.453rd 2.85 – 4.45 6.31 6.31 .403 (1st & 3rd, no sig.) .403 (1st & 3rd, no sig.)

4th 4th 4.47 --4.47 -- 6.036.03 .036.036 (1st & 4th) (1st & 4th)

Diff=.71

Diff=.65

Diff=.71

Significant Findings: Significant Findings: 2. Service Time2. Service Time

Maintain a total chat transaction time of Maintain a total chat transaction time of eight minutes or less eight minutes or less

Average = 16.0 minutes (n = 331)Average = 16.0 minutes (n = 331) 7 minutes more than Richardson’s (2002) 7 minutes more than Richardson’s (2002)

8.9 minutes (n = 20,000)8.9 minutes (n = 20,000) However, similar to six f2f studies with mean However, similar to six f2f studies with mean

service time ranging from 10 to 20service time ranging from 10 to 20

Significant FindingsSignificant Findings2. Service Time2. Service Time

Service Time of TransactionsService Time of Transactions

Quartiles (min.) Accuracy Mean Sig. of Diff. (p)Quartiles (min.) Accuracy Mean Sig. of Diff. (p)

1st 0 – 8.31st 0 – 8.3 6.826.82

2nd 8.32 – 13.082nd 8.32 – 13.08 6.026.02 .005 (1st & .005 (1st & 2nd)2nd)

3rd 13.1 – 20.753rd 13.1 – 20.75 6.046.04 .007 (1st & 3rd).007 (1st & 3rd)

4th 20.77 --4th 20.77 -- 6.126.12 .020 (1st & 4th).020 (1st & 4th)

Diff=.78

Diff=.70

Diff=.80

Significant Findings: Significant Findings: 3. Keystrokes3. Keystrokes

Keep total keystrokes per transaction Keep total keystrokes per transaction to within six and-a-half lines of text (or to within six and-a-half lines of text (or 480 characters)480 characters)

Application to virtual software vendors Application to virtual software vendors (add a timer)(add a timer)

Anything over 15 lines of text will Anything over 15 lines of text will decrease accuracydecrease accuracy

Significant FindingsSignificant Findings3. Keystrokes3. Keystrokes

KeystrokesKeystrokesKeystroke Quartiles Accuracy Mean Sig. of Diff. (p)Keystroke Quartiles Accuracy Mean Sig. of Diff. (p)LibrarianLibrarian1st 0 – 480 (6.5 lines)*1st 0 – 480 (6.5 lines)* 6.586.584th 1128 (15 lines) --4th 1128 (15 lines) -- 5.935.93 .041 (1st & 4th).041 (1st & 4th)PatronPatron1st 0 – 188 (2.5 lines)1st 0 – 188 (2.5 lines) 6.656.654th 545 (7.5) --4th 545 (7.5) -- 5.965.96 .023 (1st & 4th).023 (1st & 4th)Both Librarian & PatronBoth Librarian & Patron1st 0 – 690 (9 lines)1st 0 – 690 (9 lines) 6.636.63 4th 1668 (22.5 lines) --4th 1668 (22.5 lines) -- 5.995.99 .041 (1st & 4th).041 (1st & 4th)

*measured at 74 keystrokes per line of text*measured at 74 keystrokes per line of text

Significant FindingsSignificant Findings4. Twice the Typing4. Twice the Typing

Expect to type twice as many characters as Expect to type twice as many characters as the patronthe patron

Appeared across all four quartile segments Appeared across all four quartile segments between librarian and patron.between librarian and patron.

Significant Findings Significant Findings 5. Open-ended Questions5. Open-ended Questions

Be careful about beginning the question negotiation segment of the reference interview with an open question unless the nature of the patron’s question explicitly calls for one

Frequency of Open-ended QuestionsCategoryCategory Frequency Frequency Percent PercentPresentPresent 112 112 3333Absent (but should) 75Absent (but should) 75 22.522.5Not ApplicableNot Applicable 76 76 22.822.8AmbiguousAmbiguous 6767 20.120.1

Significant Findings Significant Findings 5. Open-ended Questions5. Open-ended Questions

Open-ended QuestionsOpen-ended Questions

Category Accuracy Mean Sig. of Diff. (p)Category Accuracy Mean Sig. of Diff. (p)

Not ApplicableNot Applicable 6.726.72

PresentPresent 5.975.97 .008 (3 & 1).008 (3 & 1)

Significant Findings: Significant Findings: 6. Closed-ended Questions6. Closed-ended Questions

Ask closed or clarifying questions when appropriate

Frequency of Closed-ended and/or Clarifying Questions Frequency of Closed-ended and/or Clarifying Questions

CategoryCategory Frequency Percent Frequency Percent

PresentPresent 183183 55.055.0

AbsentAbsent 4848 14.414.4

Not ApplicableNot Applicable 5656 16.816.8

AmbiguousAmbiguous 4444 13.213.2

Significant Findings Significant Findings 6. Closed-ended Questions6. Closed-ended Questions

Closed and/or Clarifying Questions Closed and/or Clarifying Questions

CategoryCategory Accuracy Mean Sig. Of Diff. Accuracy Mean Sig. Of Diff. (p)(p)

3. Not Applicable3. Not Applicable 6.69 6.69

2. Absent2. Absent 5.91 5.91 .065 (3 & 2, not .065 (3 & 2, not sig.)sig.)

Ambiguous filteredAmbiguous filtered

3. Not Applicable3. Not Applicable 6.696.69

2. Absent2. Absent 5.91 5.91 .040.040 (3 & 2) (3 & 2)

Significant Findings Significant Findings 7. Follow-up Question7. Follow-up Question

At the end of the reference transaction, ask “Does this completely answer your question?”

Frequency of the Librarian Asking If the Question Frequency of the Librarian Asking If the Question Had Been Answered Completely Had Been Answered Completely

CategoryCategory Frequency Frequency PercentPercentPresentPresent 125125 37.5 37.5AbsentAbsent 4242 12.6 12.6Not ApplicableNot Applicable 108108 32.432.4AmbiguousAmbiguous 5656 16.8 16.8

Significant Findings Significant Findings 8. Question Difficulty8. Question Difficulty

Question Difficulty Question Difficulty Criteria Point Frequency %Criteria Point Frequency % Cum. %Cum. %LowLow 1.01.0 5959 17.7 17.7 17.8 17.8

1.51.5 4141 12.3 12.3 30.2 30.22.02.0 6666 19.8 19.8 50.2 1/250.2 1/22.52.5 5555 16.5 16.5 66.8 66.83.03.0 2727 8.1 8.1 74.9 3/474.9 3/4

MediumMedium 3.53.5 2626 7.8 7.8 82.8 82.84.04.0 1717 5.1 5.1 87.9 87.94.54.5 1818 5.4 5.4 93.4 93.45.05.0 66 1.8 1.8 95.2 95.25.55.5 44 1.2 1.2 96.4 96.46.06.0 7 7 6.6%6.6% 2.1 2.1 98.5 98.56.56.5 33 0.9 0.9 99.4 99.4

HighHigh 7.07.0 22 0.6 0.6 100.0 100.0

Significant FindingsSignificant Findings8. Question Difficulty8. Question Difficulty

Question Difficulty and Accuracy (reporting only significance) Question Difficulty and Accuracy (reporting only significance) Criteria Criteria Points Accuracy Mean Sig. of Diff. (p)Points Accuracy Mean Sig. of Diff. (p)LowLow 1.01.0 7.247.24

2.02.0 6.366.36 .035 (1.0 & 2.0).035 (1.0 & 2.0)2.52.5 5.945.94 .000 (1.0 & 2.5).000 (1.0 & 2.5)

MediumMedium 3.53.5 5.425.42 .000 (1 & 3.5).000 (1 & 3.5)4.04.0 5.535.53 .001 (1.0 & 4.0).001 (1.0 & 4.0)4.54.5 5.225.22 .000 (1.0 & 4.5).000 (1.0 & 4.5)5.05.0 4.924.92 .009 (1.0 & 5.0).009 (1.0 & 5.0)5.55.5 4.04.0 .001 (1.0 & 5.5).001 (1.0 & 5.5)

HighHigh 6.06.0 4.834.83 .006 (1.0 & 6.0).006 (1.0 & 6.0)LowLow 1.51.5 6.956.95

2.52.5 5.945.94 .041 (1.5 & 2.5) .041 (1.5 & 2.5) MediumMedium 3.53.5 5.425.42 .002 (1.5 & 3.5).002 (1.5 & 3.5)

4.04.0 5.535.53 .032 (1.5 & 4.0).032 (1.5 & 4.0)4.54.5 5.225.22 .001 (1.5 & 4.5).001 (1.5 & 4.5)5.55.5 4.04.0 .005 (1.5 & 5.5).005 (1.5 & 5.5)

HighHigh 6.06.0 4.834.83 .038 (1.5 & 6.0).038 (1.5 & 6.0)

5. Conclusions5. Conclusions

Virtual reference lacks a statistically sound conceptual Virtual reference lacks a statistically sound conceptual model to guide the library profession toward improving the model to guide the library profession toward improving the reference interview through empirical studies which informs reference interview through empirical studies which informs best practices in professional training and assessment.best practices in professional training and assessment.

This study addresses that knowledge void by its discovery This study addresses that knowledge void by its discovery of several statistical relationships between nine behavioral of several statistical relationships between nine behavioral factors and an acurate answer in the reference interview.factors and an acurate answer in the reference interview.

It is hoped that the suggested eight-point rubric and other It is hoped that the suggested eight-point rubric and other results of this project can be a catalyst for practical results of this project can be a catalyst for practical application toward improving the practice of the global application toward improving the practice of the global community of professionals and stakeholders in the field of community of professionals and stakeholders in the field of library and information studies. library and information studies.

6. Questions6. Questions&&

AnswersAnswers