caa in uk he:
DESCRIPTION
CAA in UK HE:. The State of the Art Bill Warburton University of Southampton Email: [email protected]. Research & Graduate School of Education. The 2003 survey. Introduction Demography Findings: Question banks CAA software Pros & Cons Question types VLEs & MLEs Interoperability - PowerPoint PPT PresentationTRANSCRIPT
CAA in UK HE:CAA in UK HE:The State of the ArtThe State of the Art
Bill WarburtonBill WarburtonUniversity of SouthamptonUniversity of SouthamptonEmail: [email protected]: [email protected]
Research & Graduate School of Education
The 2003 surveyThe 2003 survey IntroductionIntroduction DemographyDemography Findings:Findings:
Question banksQuestion banks CAA softwareCAA software Pros & ConsPros & Cons Question typesQuestion types VLEs & MLEsVLEs & MLEs InteroperabilityInteroperability
SummarySummaryResearch & Graduate School of Education
The 2003 surveyThe 2003 survey
Why do a CAA survey now?Why do a CAA survey now? Increasing prevalence of web-based deliveryIncreasing prevalence of web-based delivery Pervasiveness of Virtual Learning Environments Pervasiveness of Virtual Learning Environments
(VLEs)(VLEs) Emergence of Managed Learning Environments Emergence of Managed Learning Environments
(MLEs)(MLEs) Emergence of Interoperability standards e.g. IMS/QTIEmergence of Interoperability standards e.g. IMS/QTI Four year gap e.g. between 1999 & 1995 CAA Four year gap e.g. between 1999 & 1995 CAA
surveyssurveys
Research & Graduate School of Education
The 2003 surveyThe 2003 survey Similarities with the (1999) CAA Centre Similarities with the (1999) CAA Centre
Survey*Survey* For congruence, a superset of the 1999 For congruence, a superset of the 1999
surveysurvey Dissimilarities with 1999 surveyDissimilarities with 1999 survey
Extra questions on VLEs, MLEs & Extra questions on VLEs, MLEs & InteroperabilityInteroperability
Targeted CAA enthusiasts rather than non-Targeted CAA enthusiasts rather than non-usersusers
Delivered purely by web-based CAA systemDelivered purely by web-based CAA system
*Bull, 1999; Bull & McKenna, 2000; Bull & Hesketh, 2001
Research & Graduate School of Education
The 2003 survey: The 2003 survey: Who were the Who were the respondents?respondents? Targeted people who were likely to be using CAA Targeted people who were likely to be using CAA Incentive- prize draw of two £25 book tokens Incentive- prize draw of two £25 book tokens Survey questionnaire was distributed using three JISC mailbase lists: Survey questionnaire was distributed using three JISC mailbase lists:
CAACAA TOIA TOIA ALTALT
50 responses received by initial deadline of 1850 responses received by initial deadline of 18thth May May Will be issued more widelyWill be issued more widely
Research & Graduate School of Education
The 2003 survey: The 2003 survey: Who were the Who were the respondentsrespondents?? CAA Involvement:CAA Involvement:
A constituency of enthusiastsA constituency of enthusiasts
0%
10%
20%
30%
40%
50%
60%
70%
80%
Not currentlyinvolved with
CAA
Currentlyinvolved with
CAA
Intend to beinvolved with
CAA
No longerdinvolved with
CAA
Research & Graduate School of Education
The 2003 survey:The 2003 survey: Job FunctionJob Function Little more than half the 2003 respondents were Little more than half the 2003 respondents were
academics (1999: academics (1999: 77% were academics)77% were academics) An increased proportion of pedagogic & ICT An increased proportion of pedagogic & ICT
supportsupport
0%
10%
20%
30%
40%
50%
60%
Academicteacher
ICT support other pedagogicsupport
Research & Graduate School of Education
The 2003 Survey: The 2003 Survey: Who supports CAA Who supports CAA now?now?
0%
10%
20%
30%
40%
50%
Solely in acentral unit
Solely withinfaculties
Solely inacademic
departments
In a central unitand within
faculties and/ordepartments
Elsewhere
CAA support is largely centralised Even where support has been devolved to faculties /
departments, there is usually a central CAA service
Research & Graduate School of Education
The 2003 survey: The 2003 survey: Who uses CAA now?Who uses CAA now?Usage by Subject : broadly the same as 1999Usage by Subject : broadly the same as 1999
20032003 199919991 1 Computing/IT2 Biomedical Science
3 Other science
4 Geological science5 Business=8 Psychology=8 Leisure=8 Study skills
1 Computing/IT2 Biomedical Science
3 Geological science4 Maths/Engineering
5 Other science6 Modern languages7 Psychology8 Business9 Social science
10 Humanities
11 Built environment
12 Leisure13 Study skills
Research & Graduate School of Education
2003 Survey: 2003 Survey: The use of question banksThe use of question banks
Do you use question banks?
No41%
Yes59%
Would you be interested in sharing questions with academics at other
institutions?
No12%
Yes88%
Nearly 60% said they now use question banks Nearly 90% would be interested in sharing
questions
Research & Graduate School of Education
The use of question banksThe use of question banks Pros of question sharing- shared loads:
Authoring QA Editing
Cons of question sharing: Initial time penalty difficulty of applying rigorous QA standards across multiple authors & institutions requirement for multiple formats to suit all users
Research & Graduate School of Education
The use of question banksThe use of question banks Example in Science:
The e3an project developed an item bank of more than 1500 questions for use in Electronics and Computer Science
Example in Humanities: A question bank has been developed to pre-course assess modern language
students
Research & Graduate School of Education
2003 Survey: 2003 Survey: What CAA software is used in HE What CAA software is used in HE now?now?
0
2
4
6
8
10
12
14
Perception Blackboard WebCT TRIADS i-assess Other
What CAA software do you use?
Commercial CAA systems are more dominant Many respondents used multiple CAA systems
2003 Survey: 2003 Survey: What CAA software is used in HE What CAA software is used in HE now?now?
Split between users of Commercial and Open systems Commercial systems- traditionally seen as expensive but easier to use Non-commercial systems e.g. TRIADS- free, but support-intensive
Research & Graduate School of Education
2003 Survey: 2003 Survey: Advantages of different CAA Advantages of different CAA softwaresoftware
2003 Advantages of commercial systems ranked by popularity:
2003 Advantages of Open systems:
1 Flexibility2 Support for Interoperability: “QTI…which makes it futureproof and easier to share questions”3 Student tracking- 4 Range of Q. types5 Reporting- “ability to produce wide range of reports“=9 Support=9 Scalability=9 Robustness=9 Ease of use
“TRIADS is very flexible”“AIM is tailored to mathematics. It is open source, rather than a professional package”“Ease of use (as I developed it)”
Research & Graduate School of Education
2003 Survey:2003 Survey: Disadvantages of different CAA Disadvantages of different CAA softwaresoftware
2003 Disadvantages of commercial systems ranked by unpopularity:
2003 Disadvantages of Open systems:
1 Limitations of quiz facilities built into VLEs2 Technical issues: e.g. Windows applications only, poor use of the database, non - standard user interface, cost, poor integration with MIS, poor password encryption, ad hoc patches and fixes3 non-intuitive user interfaces=5 Inflexibility: “need to use the underlying html type code to make it do what i want”=5 Restricted feedback and statistical results
“TRIADS is very complex”
Research & Graduate School of Education
2003 Survey:2003 Survey: Main benefits of CAA Main benefits of CAA Little different from 1999, considering small sample 1999 survey:
Speed of feedback to students Statistical analysis of results Automatic marking
2003 survey2003 survey Speed of feedback: Speed of feedback:
““Available 24/7 - with instant feedback”Available 24/7 - with instant feedback” Speed of automatic marking:
“50 pieces of coursework marked in a flash”” Formative use promotes student learning: Formative use promotes student learning:
““students often retake quizzes and improve their score”students often retake quizzes and improve their score”
2003 Survey:2003 Survey: Main disadvantages of Main disadvantages of CAACAA There appears to have been a shift- Learning curve and up-front investment are now seen to be the most prominent disadvantagesThere appears to have been a shift- Learning curve and up-front investment are now seen to be the most prominent disadvantages 1999 survey:
access to and reliability of hardware and software amount of time needed create and organize delivery difficulty of writing good questions
2003 survey:2003 survey: amount of time needed create and organize delivery access to and reliability of hardware and software difficulty of writing good questions
Research & Graduate School of Education
2003 Survey:2003 Survey: Does CAA save time? Does CAA save time?
If academic staff at your institution use CAA, are they saving their own time?
No49%Yes
51%
The 2003 respondents were evenly split regarding the time saved by CAA
Research & Graduate School of Education
Has this meant a significant shift in the workloads of other staff?
No71%Yes
29%
z
Surprisingly, a large majority of those who reported productivity benefits did not see a corresponding shift in workload to other staff
2003 Survey: 2003 Survey: Does CAA save time?Does CAA save time?
Research & Graduate School of Education
2003 Survey:2003 Survey: Formative vs. Summative testingFormative vs. Summative testing
There appears to be a definite shift towards formative use: less than half the 2003 tests were summative (1999: most tests were summative)
Diagnostic use is emergent: other formative usage is long-established
0%
10%
20%
30%
40%
50%
diagnostic formative self-assessment
summative
Type of assessment
Research & Graduate School of Education
2003 Survey:2003 Survey: Question typesQuestion types
1999: MCQ & MRQ most popular followed by text and numeric input, then graphic hot spot and other types
Little appears to have changed in 2003: MCQ still top
0%
10%
20%
30%
40%
50%
60%
graphicalhot spots
MCQs multi-response
other text input
What question types/formats do you use?
Research & Graduate School of Education
2003 Survey:2003 Survey: Who is doing CAA? Who is doing CAA?
0%
20%
40%
60%
80%
No-oneuses CAA
A smallgroup of
enthusiasts
20% ormore
Some nolonger use
CAA
Some may,but notaware
Which of the following describes the experience of the staff at your institution with CAA?
Moore's Gap has yet to be crossed: CAA appears still to be largely the province of ‘early adopters’
Will CAA follow VLEs and become mainstream technology?
Research & Graduate School of Education
2003 Survey:2003 Survey: The growth, influence and relevance The growth, influence and relevance of VLEsof VLEs
Does your institution already provide an institution-wide VLE?
No19%
Yes81%
Have VLEs crossed Moore’s Gap? Are now pervasive Most respondents (81%) had implemented site-wide VLEs It seems that VLE quiz tools give users a taste for CAA The uptake of VLEs may foreshadow the uptake of CAA
Research & Graduate School of Education
0%5%
10%15%20%25%30%35%40%45%
Discouraged Mildlydiscouraged
Neutral Mildyencouraged
Encouraged
Do you think your VLE has encouraged or discouraged the use of CAA?
VLEs are seen as a largely positive (60%) influence on the uptake of CAA
Cautionary note: Two reports that VLEs apparently discouraged the use of CAA
2003 Survey:2003 Survey: The growth, influence and relevance The growth, influence and relevance of VLEsof VLEs
Research & Graduate School of Education
2003 Survey:2003 Survey: Interoperability & VLEsInteroperability & VLEs
Would you like to be able to export quizzes from your VLE to
a CAA system?
I do not use my VLE's quiz
facility38%No
22%
Yes40%
The majority of VLE quiz users wanted to export their quizzes to a dedicated CAA system
One respondent wanted to import CAA items and tests into a VLE
Research & Graduate School of Education
2003 Survey:2003 Survey: tension between VLE quizzes tension between VLE quizzes and CAAand CAA Many respondents attempting to do CAA using VLE’s Quiz engine
Most commented unfavourably on VLE Quiz engines: “…testing facility is very limited, but is all we have available”
Cultural & technical differences between VLEs & CAA: Words do not always have common meanings-
Blackboard refers to a quiz as an ‘item’ In CAA an ‘item’ is usually a single question
VLE quiz items/tests often restricted to proprietary formats, & may be less secure
Research & Graduate School of Education
2003 Survey:2003 Survey: The emergence of MLEsThe emergence of MLEs
To your knowledge, is your institution aiming to develop an MLE?
Yes68%
Don't know26%
No6%
More than two thirds of respondents’ institutions have stated an intention to develop an MLE
However, the concept of an MLE is contended (JISC 7/99 projects)
Research & Graduate School of Education
2003 Survey:2003 Survey: InteroperabilityInteroperability
0
5
10
15
20
25
30
Not at allimportant
Not reallyimportant
Moderatelyimportant
Veryimportant
How important is Interoperability to you?
The majority of HE users thought interoperability to be important
Research & Graduate School of Education
2003 Survey:2003 Survey: InteroperabilityInteroperability
0%
10%
20%
30%
40%
50%
Not reallyaware
Moderatelyaware
Very aware
How aware are you of interoperability standards?
Although most HE users thought interoperability to be important, awareness of interoperability standards varied widely
Research & Graduate School of Education
SummarySummarySuperset of 1999 survey + extra questions on VLEs, MLEs & Superset of 1999 survey + extra questions on VLEs, MLEs & InteroperabilityInteroperabilityDemography: More support staff involved than in 1999Demography: More support staff involved than in 1999Question banks: Desirable but rareQuestion banks: Desirable but rareCAA software: Commercial CAA systems dominant but inflexibleCAA software: Commercial CAA systems dominant but inflexiblePros: Little changed from 1999- time savingsPros: Little changed from 1999- time savingsCons: Learning curve now more significant than costCons: Learning curve now more significant than costQuestion types: MCQs & MRQs still predominantQuestion types: MCQs & MRQs still predominantVLEs: have crossed Moore’s Gap and appear to drive CAA VLEs: have crossed Moore’s Gap and appear to drive CAA MLEs: Most HEIs intend to implements MLEs (but concept of MLE MLEs: Most HEIs intend to implements MLEs (but concept of MLE contended)contended)Interoperability: thought to be important, but awareness of Interoperability: thought to be important, but awareness of standards variesstandards varies
Research & Graduate School of Education
End of presentationEnd of presentation Questions?
Question banks: desirable but how much effort? CAA software: What features do people most want? Pros & Cons: at what point do the pros outweigh the cons? What question types do people most use? Most want? What roles do Interoperability, VLEs & MLEs play in promoting CAA? Do Interoperability, VLEs & MLEs drive CAA or vice-versa? When will CAA cross Moore’s Gap?
Research & Graduate School of Education