ada emmett and judith emde university of kansas libraries assessing chemical information literacy...
Post on 21-Dec-2015
218 views
TRANSCRIPT
Ada Emmett and Judith EmdeUniversity of Kansas Libraries
Assessing Chemical Information Literacy Skills using the ACRL Standards
ALA 2006 Annual Meeting, New OrleansSTS Research Forum
Sunday June 25th
2
Overview
Chemistry bibliography course
Assessment tool development to measure learning outcomes
Assessment results and observations
ACRL standards: advantages
3
Bibliography of Chemistry : CHEM 720
One hour credit offered for graduate students
Major chemistry and biomedical research tools
Grading: satisfactory/unsatisfactory
4
Questions
What should students learn?
Does teaching produce the desired learning outcomes?
How can we assess student learning?
5
Used to develop:
Learning outcomes
Teaching activities
Assessment tool
An “information literate” student
ACRL’s Information Literacy Competency Standards for Higher Education
6
ACRL’s Information Literacy Competency Standards for Higher Education
Alternatives to ACRL literacy standardsACSSTS
7
ACRL’s Information Literacy Competency Standards for Higher Education
1. Determine the extent of information needed 2. Access the needed information effectively and
efficiently 3. Evaluate information and its sources critically and
incorporate selected information into his or her knowledge base and value system.
4. Use information effectively to accomplish a specific purpose
5. Understand the economic, legal, and social issues surrounding the use of information, and access and use information ethically and legally
http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm
8
Year One: 2004
MethodologyDeveloped assessment tool Conducted pre- and post-course
assessment interviewsUsed control groupQuantified data
9
Year One: 2004
Results
Averagescore
Pre-Test Score
Post-TestScore
Possible Percent Improvement
PointsImproved
Studentsin Class
N=26
45 65 72 44% 20
“Control” N=4
48 56 72 17% 8
10
Year One: 2004
45 48
6556
7272
0
10
20
30
40
50
60
70
80
Students in Class Control
Pre-Pre-
Possible Possible
Post- Post-
11
Year Two: 2005
Assessment tool: redesign and refinement
SPSS Data Entry Builder to enter data One-on-one interviewsPre- and post-test given
students 16small control group 5
12
Year Two: 2005
Results
Averagescore
Pre-Test Score
Post- TestScore
Possible Percent Improvement
PointsImproved
Studentsin Class
N=16
61 99 120 62% 38
“Control” N=5
73 85 120 16% 12
13
Year Two: 2005
ResultsAverage 2005 Scores in Pre- and Post- and Total
Possible Points by Group
6173
9985
120 120
0
20
40
60
80
100
120
140
Students In Class Control
Pre- Post- Possible
14
Year Three: 2006
Used “backward design”– starting with ACRL learning outcomes
Develop assessment questionsDesign course lectures/assignmentsCreate tool – web-based questionnaire
Run as pre- and post-testNo controls
15
Year Three: 2006
Results
Points Improved: 27Percent Improved: 57%
Average scores in Pre- Post- and Total Possible Points
47.5
74.5 80.0
-
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
1
Pre-
Post-Possible
16
Comparison by Assessment Questions
Comparison of Pre- and Post-survey Average Scores by Question
0.00
1.00
2.00
3.00
4.00
5.00
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Question Number
Ave
rag
e P
oin
ts
Per
Qu
esti
on
Total Possible Per Item Scores Before Instruction Scores After Instruction
17
2004-2006 Improvement
2004: 44% improvement (“control” 16%) 2005: 62% improvement (“control” 17%) 2006: 57% improvement
44%
57%62%
0%
10%
20%
30%
40%
50%
60%
70%
2004 2005 2006
18
Assessment questions grouped by standards
Standard #2:The information literate student accesses needed information
effectively and efficiently.
Performance indicator #5:The information literate student extracts, records, and manages
the information and its sources. Outcome “e”:Uses various technologies to manage the information selected
and organized.
Assessment questions: Select among the following tool(s) that manage references or
citations and therefore assist in writing research papers.
Describe two features of a software program that assist in managing citations.
19
Assessment questions grouped by standards
Average Scores on Survey Questions as Associated with Information Literacy Standards
0.00
1.00
2.00
3.00
4.00
5.00
6.00
1.2 1.3 2.1 2.2 2.4 2.5 3.2 3.7 5.1 5.2
Information Literacy Standards and % of Improvement from Pre- to Post-
Po
ints
Per
Qu
esti
on
Pre Score Post Score Points Possible
28.2% 15.9% 44.8% 43.8% 44.8% 36.1% 26.9% 44.8% 81.3% 6.3%
Average Scores on Survey Questions as Associated with Information Literacy Standards
0.00
1.00
2.00
3.00
4.00
5.00
6.00
1.2 1.3 2.1 2.2 2.4 2.5 3.2 3.7 5.1 5.2
Information Literacy Standards and % of Improvement from Pre- to Post-
Po
ints
Per
Qu
esti
on
Pre Score Post Score Points Possible
28.2% 15.9% 44.8% 43.8% 44.8% 36.1% 26.9% 44.8% 81.3% 6.3%
20
Students’ perceptions
Overall Average Student Scores from 2006 Survey
Score Perception
Score
Perception
-
10
20
30
40
50
60
70
80
Pre-Instruction Post-Instruction
Tota
l Ave
rage
Sco
re
(80
Pos
sibl
e)
21
Observations
Mapping standards is subjectiveConsultation on mapping questions to standards
Some standards such as 3 and 4 are difficult to assess
Strong subject expertise needed (faculty/instructor collaboration)
Mechanism of delivering assessment tool needs improvement
Ongoing review of assessment tools’ strengths/weaknesses
22
Observations
Assessing all learning outcomes difficultKeep it simple to several specific outcomes.
Did the class and the test incorporate most important learning outcomes for that student group?
Faculty/instructor collaboration
23
ACRL standards: advantages
Provides mechanism to assess information literacy skills via learning outcomes (using “backward design”)
Assists in the development of course content through backward design
24
Discussion, Questions and Comments
25
Contact information
Ada Emmett
785-864-8831
Judith Emde
785-864-4931
Supplemental information at http://www.people.ku.edu/~jemde/
26
27
Mapping lectures / exercises to standards
Potential learning outcome of lecture and exercise mapped to standards at beginning of semester
Comparison of lectures/exercises to results of post assessment by grouped standards.
28
Mapping lectures / exercises to standards
Correlation Between Number of Lectures/Exercises by Standard and Associated Student Score Improvement
-
5
10
15
1.2 1.3 2.1 2.2 2.4 2.5 3.2 3.7 5.1 5.2
Information Literacy Standards
Total Lectures and/or Exercises associated with each
0%
100%
Pre to Post % Improve-
ment by Standard
# L-E's by Std % Improvement