placement testing with open source software . case study: sapporo gakuin university 1.mixed level...
TRANSCRIPT
Placement Testing with
Open Source Software
Case Study: Sapporo Gakuin University
1.Mixed level students in two year general English program. Six full-time teachers agreed in a staff workshop that this was the greatest problem
- bright students not challenged, lost interest- weak students over-challenged, lost interest
Case Study: Sapporo Gakuin University
2. Commercial testing software was expensive and would take 1-2 years to get budget approval. We had doubts about its usefulness. We wanted a system where we could control and improve the the test items.
- Calus Testing Service 3000 yen/student/year 1000 freshmen/year. Total budget = 3,000,000 yen
Case Study: Sapporo Gakuin University
3. We wanted to adapt our current progress test system (50 questions--30 listening, 20 reading) for placement of incoming freshmen.
But paper test marking services 300,000 yen /year
We wanted a “testing machine” that would not give an audio-based test, but also analyze the results so we could improve the test questions.
Testing Machine version 2.13
50 User friendly questions
Reusable stainless steel interface
User adaptable hood harness
Wrist and leg straps not necessary
I am level 3!
Questions
Case Study: Sapporo Gakuin University
4. Tool SelectionWe had experience administering computer-based m/c tests in large English classes using an open source CMS with quiz-making tools (Moodle).
- Why not do a computer-based placement test- 10 computer rooms available in beginning of term.- Rooms had no headphones. - But quizzes could not do audio. - How would we do listening questions?
Case Study: Sapporo Gakuin University
5. Programming We went to colleagues and the creator of Moodle and asked them to help program Moodle to handle listening questions.
- One colleague contributed 110,000 yen of personal research money for programming.- Sent money to Dougiamas to supervise process.-
Initial Requirements for a placement test programming
Play/Stop Button within each Question: It should be intuitive for student to use
No Popup Player: Avoid confusion for student
Number of Plays Setting: It should be easy to create an online course from existing learner resources,
Simple Login for Student: It should provide a simple screen. provide bilingual menus, help, and input.
No Result Displayed at End
Course Mangement System
Actual Result for a placement test programming
OK Play/Stop Button within each Question: It should be intuitive for student to use
OK No Popup Player: Avoid confusion for student
CUT Number of Plays Setting: We just set a start and end
time for the whole test. We let students play as many times as needed, but warned them to finish in time allowed.
CUT Simple Login for Student: We used standard Moodle login. Teachers turned on machines, logged into main school system. Then we assigned seats, and put unique username and password on a paper on the keyboard of each machine.
CUT No Result Displayed at EndCourse Mangement System
Types of Tests
Criterion Referenced
Norm Referenced *
Evaluating Tests: Four Criteria
Practicality Validity Reliability Washback
Practicality
● Organizing and Achieving Consensus ● Exam Proctoring and Management ● Hardware Test● Software Test
Hardware Test
Server 1 Mac OS X, 933 mhz, singleResult: 140 students, stopped, did paper test
80 students, very slow, successful
Server 2 Mac OS X, 1.2 ghz dualResult: 100 students, stopped
Finder problems, hardware freezes
Server 3 Linux Fedora 2, 3.2ghz singleResult: 100 students, successful
250 students at one time planned for 2005 2 servers, plus two backup servers
Hardware Specifications
Server 3 Linux Fedora 2, 3.2ghz singleBuilt from parts, server engineer made LAMP in free
time.
Parts List and Costs
- Motherboard 13,000- Processor 30,000- Memory, 2gb 45,000- Hard disk, 120gb 9,000- CD drive 3,000- Case 8,000- RAID case 20,000- Install Fee 30,000
RAID with second hard disk will automatically clone whole server
Test Rooms 4 x 60 computers
Reliability
Item Facility - 50 questions X O X
0~30% 30~40% 40~60% 60~70% 70~100%
Item Discrimination X O<1.0 1.0~1.3 >1.3
Distractor Analysis
Validity
Washback
● Effect on teachers
