uk euniversities worldwide limited derek morrison 25 november 2003

UK eUniversities Worldwide Limited

Derek Morrison25 November 2003

UK eUniversities Worldwide Limited

Derek Morrison25 November 2003

Assessment Menu [ short ]Assessment Menu [ short ]

What do we assess?What do we assess?

ExamplesExamples

Assessment alignment / integrationAssessment alignment / integration

Is assessment in e-learning different?Is assessment in e-learning different?

FinishFinish

Assessment ExamplesAssessment Examples

Bath’s GroupLogBath’s GroupLog

WIMBAWIMBA

LearnWrightLearnWright

War in the Modern WorldWar in the Modern World

Objective test examplesObjective test examples

ePortfolio examplesePortfolio examples

Leeds/Soton GISLeeds/Soton GIS

What should we / do we assess?


How do we assess?How do we assess?



Knowledge

“… examinations are very poor predictors of any subsequent performance, such as success at work … first degree results explain less than 10% of the variance in postgraduate performance” (Warren 1971) in Gibbs G & Simpson C [publication pending]

“… examinations are very poor predictors of any subsequent performance, such as success at work … first degree results explain less than 10% of the variance in postgraduate performance” (Warren 1971) in Gibbs G & Simpson C [publication pending]




“… the kind of learning that coursework involves has long term consequences while the kind of learning involved in revision for exams does not … in experimental studies in which students have either studied exam-based or assignment-based courses, the quality of the learning has been shown to be higher in the assignment-based courses.” (Gibbs G & Simpson C [publication pending])

“… the kind of learning that coursework involves has long term consequences while the kind of learning involved in revision for exams does not … in experimental studies in which students have either studied exam-based or assignment-based courses, the quality of the learning has been shown to be higher in the assignment-based courses.” (Gibbs G & Simpson C [publication pending])




Is assessment in e-learning different? Is assessment in e-learning different? • There is an assumption that technological

literacy and other prerequisites will be met• Students are not necessarily in the same

time zone/place • Cultural influences and expectations• The informal assessment practices of

face-to-face are impossible• Processes need to be clearly defined.• High quality feedback is critical.• How do we design a range of indicators

or evidence gathering processes for online assessment to measure how students solve ill-defined problems, transfer learning, and achieve higher order outcomes (classification, interpretation, abstraction, rule application, judgement > analysis, synthesis, evaluation?)

“Patterns indicate that freshmen enter ENG105 with relatively high word-processing, editing, email, and web research skills, but with lower skills in online collaboration, multi-media presentation, web analysis, and web design.” (Norris, JM (2003), Assessment of Students in Electronically-Mediated Settings, Northern Arizona University, http://www4.nau.edu/assessment/main/electronic/E-LearningAssessmentReport2002-2003.pdf )

“Patterns indicate that freshmen enter ENG105 with relatively high word-processing, editing, email, and web research skills, but with lower skills in online collaboration, multi-media presentation, web analysis, and web design.” (Norris, JM (2003), Assessment of Students in Electronically-Mediated Settings, Northern Arizona University, http://www4.nau.edu/assessment/main/electronic/E-LearningAssessmentReport2002-2003.pdf )

Should assessment (like learning itself) be a process or an event?

Should assessment (like learning itself) be a process or an event?

How integrated/aligned/blended are your parts?How integrated/aligned/blended are your parts?

Outcomes

Assessment

“Key to the outcomes approach to assessment is the use of ‘authentic assessment’. This approach stresses creating assignments and assessments that simulate as much as possible the situations in which students would make use of the knowledge, skills and values emphasized in the course (Battersby M, So What’s a Learning Outcome Anyway?, Centre for Curriculum, Transfer and Technology http://www.c2t2.ca/GoodPractice/Exchange/curriculum-frameworks/sowhtsa.html

“Key to the outcomes approach to assessment is the use of ‘authentic assessment’. This approach stresses creating assignments and assessments that simulate as much as possible the situations in which students would make use of the knowledge, skills and values emphasized in the course (Battersby M, So What’s a Learning Outcome Anyway?, Centre for Curriculum, Transfer and Technology http://www.c2t2.ca/GoodPractice/Exchange/curriculum-frameworks/sowhtsa.html

“… the standard written tests will seldom be appropriate to assess outcomes – because what the students will do with the knowledge and skills they are learning is not usually writing tests or essays – unless we believe that the point of the course is primarily to succeed in subsequent courses.” (ibid)

“… the standard written tests will seldom be appropriate to assess outcomes – because what the students will do with the knowledge and skills they are learning is not usually writing tests or essays – unless we believe that the point of the course is primarily to succeed in subsequent courses.” (ibid)


Outcomes

Assessment

“The more outcomes which must be passed for the course to be passed … the more magnified are any unreliabilities in assessment. And, from the student’s perspective, each distinct essential learning outcome is an additional opportunity to fail.” (Baume D, 2001, A Briefing on Assessment of Portfolios p15, LTSN Generic Centre. http://www.ltsn.ac.uk/application.asp?app=resources.asp&process=full_record&section=generic&id=6)

“The more outcomes which must be passed for the course to be passed … the more magnified are any unreliabilities in assessment. And, from the student’s perspective, each distinct essential learning outcome is an additional opportunity to fail.” (Baume D, 2001, A Briefing on Assessment of Portfolios p15, LTSN Generic Centre. http://www.ltsn.ac.uk/application.asp?app=resources.asp&process=full_record&section=generic&id=6)

“… learning is rarely the linear process of movement towards immovable goals that the use of learning outcomes can sometimes suggest.” (Baume D, 2001, A Briefing on Assessment of Portfolios p18, LTSN Generic Centre. http://www.ltsn.ac.uk/application.asp?app=resources.asp&process=full_record&section=generic&id=6)

“… learning is rarely the linear process of movement towards immovable goals that the use of learning outcomes can sometimes suggest.” (Baume D, 2001, A Briefing on Assessment of Portfolios p18, LTSN Generic Centre. http://www.ltsn.ac.uk/application.asp?app=resources.asp&process=full_record&section=generic&id=6)


Outcomes

Assessment

Resources

Activities

Events

“Many of the flaws in existing e-learning programs stem from lack of alignment.” (Reeves TC, Aggen WD, Enhancing E-Learning Assessment and Evaluation Strategies, http://www.learnwright.com/ELearn 2002 Reeves Aggen Paper.pdf )

“Many of the flaws in existing e-learning programs stem from lack of alignment.” (Reeves TC, Aggen WD, Enhancing E-Learning Assessment and Evaluation Strategies, http://www.learnwright.com/ELearn 2002 Reeves Aggen Paper.pdf )

“Assessment if most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time.” (American Association for Higher Education, http://www.aahe.org/assessment/principl.htm)

“Assessment if most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time.” (American Association for Higher Education, http://www.aahe.org/assessment/principl.htm)


Outcomes

Assessment

Resources

Activities

Events

What’s the result of a lack of alignment/integration?

What’s the result of a lack of alignment/integration?

“… academic developers are often disappointed by the degree to which learners actually use these [electronic journals, real time databases, and online discussion forums] and similar online resources.” (Reeves TC, Aggen WD, Enhancing E-Learning Assessment and Evaluation Strategies p4, http://www.learnwright.com/ELearn 2002 Reeves Aggen Paper.pdf )

“… academic developers are often disappointed by the degree to which learners actually use these [electronic journals, real time databases, and online discussion forums] and similar online resources.” (Reeves TC, Aggen WD, Enhancing E-Learning Assessment and Evaluation Strategies p4, http://www.learnwright.com/ELearn 2002 Reeves Aggen Paper.pdf )


Outcomes

Assessment

Resources

Activities

Events

“However, learners may simply perceive that these resources have little relationship with assessments in the course (such as writing a paper or taking a final exam) and so they wisely choose to target their cognitive efforts on those learning activities that they perceive as more clearly aligned with attaining the highest scores on assessments.” (Reeves TC, Aggen WD, Enhancing E-Learning Assessment and Evaluation Strategies p4, http://www.learnwright.com/ELearn 2002 Reeves Aggen Paper.pdf)

“However, learners may simply perceive that these resources have little relationship with assessments in the course (such as writing a paper or taking a final exam) and so they wisely choose to target their cognitive efforts on those learning activities that they perceive as more clearly aligned with attaining the highest scores on assessments.” (Reeves TC, Aggen WD, Enhancing E-Learning Assessment and Evaluation Strategies p4, http://www.learnwright.com/ELearn 2002 Reeves Aggen Paper.pdf)


Outcomes

Assessment

Resources

Activities

Events

ASSESSMENT DRIVES STUDENT LEARNING & IS CENTRAL TO THEIR LEARNING EXPERIENCE

• Becker et al (1968)

• Snyder (1971) – the ‘hidden curriculum’

• Miller & Parlett (1974) - cues and rewards

• Rowntree (1987)

• Ramsden (1992)

• MacFarlane (1992)

• Biggs (1999)

• McConnell (2002)

ASSESSMENT DRIVES STUDENT LEARNING & IS CENTRAL TO THEIR LEARNING EXPERIENCE

• Becker et al (1968)

• Snyder (1971) – the ‘hidden curriculum’

• Miller & Parlett (1974) - cues and rewards

• Rowntree (1987)

• Ramsden (1992)

• MacFarlane (1992)

• Biggs (1999)

• McConnell (2002) So how can we make best use of this driver?

So how can we make best use of this driver?

The ‘cue conscious’ got better degree results than the cue-deaf’.

The ‘cue conscious’ got better degree results than the cue-deaf’.

“… students in the UK allocate their time largely to assessed tasks and that this becomes a more narrow focus over time as they become more experienced students, allocating as little as 5% of their time to unassessed study tasks by year three.” (Gibbs G & Simpson C [ publication pending])

“… students in the UK allocate their time largely to assessed tasks and that this becomes a more narrow focus over time as they become more experienced students, allocating as little as 5% of their time to unassessed study tasks by year three.” (Gibbs G & Simpson C [ publication pending])


Outcomes

Assessment

Resources

Activities

Events

PARTICIPATORY / CONTRIBUTIONS ORIENTED LEARNING

Assessment activities:

• Recognise students as contributors

• Involve opportunities for students to communicate, contribute to, and participate in an online community

• Reflect the status of students as contributors to course content, and creators of new knowledge products. McLoughlin C, Luca J (2003), Quality in Online Delivery: What does it mean for assessment in e-learning environments?, Eric Clearinghouse on Information & Technology, March 2003, http://www.ericit.org/fulltext/IR021479.pdf

PARTICIPATORY / CONTRIBUTIONS ORIENTED LEARNING

Assessment activities:

• Recognise students as contributors

• Involve opportunities for students to communicate, contribute to, and participate in an online community

• Reflect the status of students as contributors to course content, and creators of new knowledge products. McLoughlin C, Luca J (2003), Quality in Online Delivery: What does it mean for assessment in e-learning environments?, Eric Clearinghouse on Information & Technology, March 2003, http://www.ericit.org/fulltext/IR021479.pdf

Assessment example: King’s College London – War in the Modern WorldAssessment example: King’s College London – War in the Modern World

Group Exercise [20% of module mark]

“In this learning object you will deepen your knowledge of the Cuban Missile Crisis by playing the role of a number of historical figures – American, Soviet, Cuban, British, and German …You depend greatly on the advice of your cabinet but the final call is yours. Victory conditions:

•Avoid nuclear war, 100 points

•Force the Soviets to withdraw missiles from Cuba, 80 points

•Depose Castro, 80 points

•Cause the Soviets to give up security commitment to Cuba, 50 points” Extract from Group Activity: Simulation of the Cuban Missile Crisis, War in the Modern World, Kings College London

Group Exercise [20% of module mark]

“In this learning object you will deepen your knowledge of the Cuban Missile Crisis by playing the role of a number of historical figures – American, Soviet, Cuban, British, and German …You depend greatly on the advice of your cabinet but the final call is yours. Victory conditions:

•Avoid nuclear war, 100 points

•Force the Soviets to withdraw missiles from Cuba, 80 points

•Depose Castro, 80 points

•Cause the Soviets to give up security commitment to Cuba, 50 points” Extract from Group Activity: Simulation of the Cuban Missile Crisis, War in the Modern World, Kings College London

Two Short Answer Exercises of the Student’s Own Choice[ 2 x 15% of module mark]

“The Student will draft a 750 word answer to the question set for the chosen unit, will post that draft to a discussion forum for peer review, will defend the answer in the forum and will then submit a revised answer for assessment, along with a log and reflective commentary on the peer review process.”

Two Short Answer Exercises of the Student’s Own Choice[ 2 x 15% of module mark]

“The Student will draft a 750 word answer to the question set for the chosen unit, will post that draft to a discussion forum for peer review, will defend the answer in the forum and will then submit a revised answer for assessment, along with a log and reflective commentary on the peer review process.”

Assessment in e-learning resources - 1Assessment in e-learning resources - 1

http://www.qaa.ac.uk/public/dlg/contents.htm http://kerlins.net/bobbi/education/onlineed/24benchmarks.html http://www.ihep.com/Pubs/PDF/Quality.pdf or http://www.nea.org/he/abouthe/Quality.pdf http://assessment.cetis.ac.uk/ http://bsonline.techindex.co.uk/ (BS 7988:2002)

http://www.caacentre.ac.uk/ http://www.ltsn.ac.uk/genericcentre/

Scottish Centre for Research into On-Line Learning and Assessment (SCROLLA)http://www.scrolla.ac.uk/about.html

Assessment in e-learning resources - 2 Assessment in e-learning resources - 2 Computer-Aided Assessment in Mathematics http://ltsn.mathstore.gla.ac.uk/index.asp?cat=137

Scottish Computer Assisted Assessment Network http://www.scaan.ac.uk/

Tools for Interoperable Assessment (JISC X$http://www.jisc.ac.uk/index.cfm?name=project_toia)

McLoughlin C, Luca J (2003), Quality in Online Delivery: What does it mean for assessment in e-learning environments?, Eric Clearinghouse on Information & Technology, March 2003 http://www.ericit.org/fulltext/IR021479.pdf

McConnell D, (2002), Collaborative Assessment as a Learning Event in E-learning Environments, Computer Support for Collaborative Learning (CSCL) 2002, Boulder, Colorado http://newmedia.colorado.edu/cscl/54.pdf

Evaluation and redevelopmentEvaluation and redevelopment

Capture participant feedback from specifiers, authors, tutors, reviewers, etc

Build evaluation into coursequestionnaires

interviews

email analysis

behaviour metrics

assignment performance

Research Centre can advise

Annual evaluation report and action plan required by UKeU

Allow 20% of initial development cost for first annual revision (minimum)

Assessment in e-Learning ExamplesAssessment in e-Learning Examples

GroupLog 1GroupLog 1

LearnWright ExamplesLearnWright Examples

LearnWright: ModelsLearnWright: Models

LearnWright: Virtual Audit (Cam)LearnWright: Virtual Audit (Cam)

LearnWright: Virtual Audit (Doc)LearnWright: Virtual Audit (Doc)

LearnWright: Virtual Audit (Obs)LearnWright: Virtual Audit (Obs)

WIMBAWIMBA

Objective tests and higher level learning? Objective tests and higher level learning?

“The capacity of objective tests to assess a side range of learning is often underestimated … It is commonly assumed that objective tests are only useful for examining the first three or four levels defined by Bloom et al (1956). However, some educationalists including McBeath (1992) suggest that all six levels can be tested using objective test questions.” (Bull J, McKenna C et al, 2001, Blueprint for Computer-assisted Assessment pp 14-21, CAA Centre, http://caacentre.ac.uk )

“The capacity of objective tests to assess a side range of learning is often underestimated … It is commonly assumed that objective tests are only useful for examining the first three or four levels defined by Bloom et al (1956). However, some educationalists including McBeath (1992) suggest that all six levels can be tested using objective test questions.” (Bull J, McKenna C et al, 2001, Blueprint for Computer-assisted Assessment pp 14-21, CAA Centre, http://caacentre.ac.uk )

Knowledge

Comprehension

Application

Analysis

Synthesis

Evaluation

Objective tests and higher level learning? – AnalysisObjective tests and higher level learning? – Analysis

Dr Roger Suthren, Environmental Geology, Oxford Brookes University, http://www.brookes.ac.uk/geology/8307/a40frame.html

Objective tests and higher level learning? – AnalysisObjective tests and higher level learning? – Analysis

Source: TRIADS, Demonstration of Question Styles, Centre for Interactive Assessment Development, University of Derby,, http://www.derby.ac.uk/assess/webdemo/webdemgo.html

Objective tests and higher level learning? – TRIADS Objective tests and higher level learning? – TRIADS

TRIADS, Demonstration of Question Styles, Centre for Interactive Assessment Development, University of Derby, http://www.derby.ac.uk/assess/webdemo/webdemgo.html and

http://www.derby.ac.uk/assess/ukeu/ [local demo][local demo]

ePortfolios in Assessment? ePortfolios in Assessment?

http://www.theospi.org/


How flexible, extensible do these categories need to be to be useful for assessment purposes?

How flexible, extensible do these categories need to be to be useful for assessment purposes?



catdemo 20%off

“ A portfolio project can emphasize the participant’s (student’s) role in constructing understanding, involve participants in the assessment and evaluation of their work, and enhance participant awareness of processes involving writing, problem solving, researching, or analyzing.” (http://catalyst.washington.edu/catalyst/how-to/portfolio/about.html)

“ A portfolio project can emphasize the participant’s (student’s) role in constructing understanding, involve participants in the assessment and evaluation of their work, and enhance participant awareness of processes involving writing, problem solving, researching, or analyzing.” (http://catalyst.washington.edu/catalyst/how-to/portfolio/about.html)


http://catalyst.washington.edu/catalyst/how-to/portfolio/tour.html

ePortfolios in Assessment? - linksePortfolios in Assessment? - links

http://www.eportfolios.ac.uk/

http://faculty.sccd.ctc.edu/jkent/workshops/ePortfolios/

https://catalyst.washington.edu/webtools/ (catdemo 20%off)

Assessment as the driver references Assessment as the driver references Becker et al (1968) Snyder BR (1971), The Hidden Curriculum, MIT Press

Miller CMI & Parlett M (1974), Up to the mark: A study of the examination game, Society for Research into Higher Education

Rowntree (1987),

Ramsden P (1992), Learning to teach in higher education, Routledge

MacFarlane B (1992), The ‘Thatcherite’ generation of university degree results, Journal of Further and Higher Education, 16, pp 60-70

Biggs (1999) McConnell (2002)

Assessment example: Leeds & Southampton – GISAssessment example: Leeds & Southampton – GIS

“Based on the activities undertaken in Unit 6, please compile the following …

1. A map of land suitability for agriculture based on the four factors.

2. A short one page report explaining how the map was derived including a critique of the process.”

“Based on the activities undertaken in Unit 6, please compile the following …

1. A map of land suitability for agriculture based on the four factors.

2. A short one page report explaining how the map was derived including a critique of the process.”

Key featuresActivity driven, supported by (but not led by) resources; plus assignments which are well integrated with (and dependent upon) the actual learning processes.

Key featuresActivity driven, supported by (but not led by) resources; plus assignments which are well integrated with (and dependent upon) the actual learning processes.

TRIADS Examples TRIADS Examples

The End

uk euniversities worldwide limited derek morrison 25 november 2003

Documents

assessment of students

online assessment

knowledge slide

finish slide

outcomes assessment

transfer learning

kind of learning

use of authentic assessment