the evaluation of the web-based artist ann ooms, joan garfield, bob delmas – university of...
TRANSCRIPT
The Evaluation of the Web-basedThe Evaluation of the Web-based ARTISTARTIST
Ann Ooms, Joan Garfield, Bob delMas Ann Ooms, Joan Garfield, Bob delMas – University of Minnesota– University of Minnesota
Assessment Resource Tools for Improving Statistical Thinking (ARTIST)Assessment Resource Tools for Improving Statistical Thinking (ARTIST)
The goal of the Web-based ARTIST is to help teachers asses statistical literacy, statistical reasoning and statistical thinking.
The ARTIST website provides a variety of assessment resources for teaching first courses in statistics and contains:
Item Database: a collection of over 1,200 items organized by topic and learning outcome
11 Assessment Topic Scales and the Comprehensive Assessment of Outcomes in a first Statistics Course (CAOS)
Information, guidelines, and examples of alternative assessments
Information on implementation issues related to designing, administering, and evaluating assessments
Direct links to or copies of articles on assessment in statistics
References and links to other related assessment resources
Copies of conference papers and presentations on the ARTIST project and handouts from ARTIST mini-courses
Information on upcoming ARTIST events
Information about ways to participate as a scale tester
Funded by NSF CCLI-ASA-0206571
The Iterative Evaluation Model for Improving Online Educational ResourcesThe Iterative Evaluation Model for Improving Online Educational ResourcesComponents Methods and instruments
Component 1Evaluation planning
Meetings to plan evaluation
Creation of the pre evaluation interview protocols
Pre evaluation interviews ARTIST users and non ARTIST users
Reporting on pre evaluation interview results
Component 2aEvaluation of web design
Creation of the survey for ARTIST users
Creation of the observation protocol for frequent Item Database users
Creation of the observation protocol for first time Item Database users
Reporting on web design
Component 2bEvaluation of web content
The survey for ARTIST users
Reporting on web content
Component 3Evaluation of use of the educational resource
The survey for ARTIST users
Creation of the interview protocol for frequent ARTIST users
Reporting on use of the educational resource
Component 4Evaluation of educational impact
Creation of the interview protocol for frequent ARTIST users
Reporting on educational impactEvaluation InstrumentsEvaluation Instruments
Pre Evaluation Interview Protocol ARTIST Users and
non-ARTIST Users
Survey for ARTIST Users Survey for Non-ARTIST Users Interview Protocol for Frequent ARTIST Users
Observation Protocol for Frequent Item Database Users
Observation Protocol for First-Time Item Database Users
The interviews were conducted to collect information about today’s instructors of statistics assessment practices. The information was used to tailor the other evaluation instruments. Participants were statistics instructors known by the educational resource directors.
The survey was created and administered to collect information about the web design, the web content, and implementations in educational settings. All registered ARTIST users teaching introductory statistics in the US, who had accessed the website since August 2004, and who were teaching that semester were invited.
The survey was created and administered to collect information about the non-users’ characteristics. Invitations for submitting the survey were sent by e-mail to the following list serves: isostat group, ASA Stat ed, and the sigmaa listserv.
The interviews for frequent ARTIST users were conducted to collect information about implementations in educational settings and educational impact. Participants were selected based on the number of ARTIST topic scales and ARTIST items they had administered to their students.
The observations were conducted to collect information about the navigatability of the Item Database. We wanted to learn where the Assessment Builder was difficult to use, even for experienced users. Participants were asked to conduct a think-aloud while performing certain tasks, such as create a test, add a question to that test, remove a question.
The observations were conducted to collect information about the navigatability of the Item Database. We wanted to learn where the Item Database was difficult to use for first-time users. Participants were asked to conduct a think-aloud while performing certain tasks, such as create a test, add a question to that test, remove a question.
Evaluation FindingsEvaluation Findings1. Web design: concrete recommendations have been made to improve the web design
2. Web content: recommendations have been made to include large data sets to the Item Database and to add additional item topics.
3. Use: Many statistics educators are actively using ARTIST and value the resource. The main reasons for statistics educators to not use ARTIST are time and misconceptions about what ARTIST has to offer. Because of the evaluation, several non-users became more aware of goal of ARTIST and started using ARTIST since.
4. Educational impact: instructors are changing assessment practices but there is no impact yet on student outcomes because, so far, newly developed ideas have not been implemented by instructors.
Contact information Ann Ooms
ARTIST
https://ore.gen.umn.edu/artist//
Component 2Evaluation of educational
value
Component 4Evaluation of educational
impact
Component 4aEvaluation of impact on instructors’
perceptionsComponent 4b
Evaluation of impact on educational environment
Component 4cEvaluation of impact on student outcomes
Component 4dEvaluation of sustainability of impact
Component 1Evaluation planning
Component 2aEvaluation of web design
Component 2bEvaluation of web content
R e p o r t
R e p o r t
Component 3Evaluation of use of the
educational resourceR e p o r t