Evaluating the Impact of a VLE on Learning and Teaching?

Dr Barbara NewlandLearning Design Studio

Academic ServicesBournemouth University

http://www.bournemouth.ac.uk/lds/bnewland@bournemouth.ac.uk

Overview Aims of evaluation Durham context Methodology Results Conclusions

Aims of evaluation International widespread

adoption of VLEs Longitudinal study of impact

of a VLE on learning and teaching

Qualitative and quantitative methods

University of Durham 670 academic and 280

academic-related staff 9400 undergraduates

and 2400 postgraduates Founded in 1832 Split campus Research led MIS – outsourced to

Unisys Learning Technologies

Team (LTT)

Images of Durham

Learning Technologies Team aims to encourage, support and develop

elearning throughout the University by: providing advice, information and training supporting the University's learning

environment - duo (Durham University Online)

developing interactive materials 4 members of staff – Juliette Pavey, Kate

Boardman, Malcolm Murray, Victoria Boyd

Duo Durham University Online http://duo.dur.ac.uk

Uptake of duoEstimated use by Xmas 2000

Actual use by Xmas 2000

Use at June 2003

200 students3 departments

4286 students21 departments211 modules

10,000 students700 academics700 modules

Implementation Varied patterns of departmental

implementation All courses All first year courses Individual academics

Administration courses Duo is now embedded into

learning and teaching

Methodology for duo surveys Quantitative and qualitative Online surveys Analyse factors such as gender, year of

study and faculty 3 faculties – Arts and Humanities, Science,

Social Science and Health Relate to C&IT skills in learning and

teaching Data collected from students and staff

in all faculties and both campuses

Response rate All students and staff are in duo so hard

to know the response rate in relation to active users which has increased over the years

Staff April 01 (n = 91)May 02 (n = 153)May 03 (n = 283, c. 40%)

Students May 01 (n = 793)May 02 (n = 2435)May 03 (n = 3668, c. 37%)

Survey questions Similar questions for staff and students

in all 3 years to enable comparisons : between years between staff and students

Software ease of use levels of confidence features used

Survey questions Frequency of use and access

during term time and vacations Learning and teaching

contribution effect

Student responses – support, frequency and access Help documents

19%, 15%, 10% Small and decreasing use

Accessed duo at least once a week 70%, 71%, 71% Consistent pattern but increasing numbers over time

Accessed duo out of term time 60%, 61%, 64% Consistent pattern but increasing numbers over time

Students: Overall, how do you rate the contribution duo has made to your learning?

6.5

53.1

26.6

2.1 1.4

0

10

20

30

40

50

60

%

excellent good nodifference

poor very poor

Contribution

Staff: Overall, how do you rate the effectiveness of duo as a learning resource?

16.3

48.1

11.7

3.2 1.1

0

10

20

30

40

50

60

%

excellent good average poor very poor

effectiveness

Staff: Has duo affected your approach to teaching?

10.90.58.6

80

My basic approach to teaching seems to be changing, in part because of duo

My basic approach to teaching has not changed, but duo helps me do certain things better

There is minimal potential for change

I can't teach as w ell w hen I rely on using duo

91.6

8.4

yes no

Students: If duo is not currently being used for all your modules, would you recommend that it should be?

Staff use of duo features - content 62%, 49% staff have used more features 79%, 76% staff have developed content

since the previous year Course information – 74%, 84%, 92% Course documents – 77%, 89%, 95% External links – 42%, 56%, 55% Bibliography – 35%, 53%, 43%

(plus 17%, 21%, 21% link to OPAC) Staff information – 49%, 63%, 72%

Staff use of duo features - communication Group work – 25%, 23%, 26% Email – 72%, 77%, 80% Announcements – 75%, 84%, 86% Discussion board – 16%, 32%, 31% Chat – 10%, 4%, 6%

Staff use of duo features - assessment Assignments – 53%, 63%, 67% Quizzes – 11%, 20%, 26% Surveys - 15%, 13%, 21% Digital drop box – 10%, 6%, 15%

Reflect current ways of teaching Context – relevant, flexible (start from

where they are at) Collis also found at the University of

Twente that “instructors are most likely to begin by choosing aspects of a system that reflect their current ways of teaching, and then gradually move to new instructional approaches and new features.” (Collis and Messing, 2001)

Lecture notes - staff Is it good practice to put lecture

notes online? Depends on what the lecturer does

with them 2/3 put lecture summaries, handouts,

full notes and/or Powerpoint slides after the lecture

Lecture notes - students Depends how students use them

73% listen more 81% clarify information 72% improve lectures 93% study for exams

2/3 (64%, 64%) students note-taking had stayed the same but for ¼ (27%, 26%) it had decreased

Only 1%, 1% stated no longer attended lectures

Gender No statistical significance between

gender and Staff with effectiveness as learning

resource Students with contribution to learning Both with levels of confidence

Higher percentage of females think duo helps them to manage their learning

Staff by faculty – skills More Science faculty found it very

easy/easy to use compared to Arts Some Arts staff took longer to gain

confidence (6-10 visits) Fewer Science staff chose to have

training Reflects the levels of staff

confidence in C&IT skills found in Audit June 2000

Staff by faculty – teaching 2002 - no significant difference 2003

Basic approach is changing Science and social science 12% compared to

3% arts Basic approach has not changed but

duo helps me do certain things better Science 79%, social science 76% compared

to 98% arts

Students by faculty Contribution that duo has made to their

learning overall 2002 - no significant difference 2003 – higher percentage of science (69%)

and social science (52%) rate “good” or “excellent” compared to 42% arts students

Higher percentage of social science students think duo helps them to manage their learning “greatly” or “quite a lot” compared to science students

Students by year group Contribution that duo has made to

their learning overall 2001 and 2002 - 2nd and 3rd year

students rates more higher than 1st years (1st years have always had duo)

2003 – 1st year students rate slightly higher than 2nd and 3rd years

C&IT skills survey of new undergraduates – 2002 n = 2656 82% of students perceive themselves to

be confident/very confident computer users

60% have brought a computer with them to University

69% have been using a computer for 5+ years (including 21% for over 10 years) and only 2% have been using a computer for <2 years

C&IT skills survey of new undergraduates - 2002 79% perceive themselves to have

good/advanced email skills 65% perceive themselves to have

good/advanced web browser skills 70% stated a preference for a

combination of web and paper based for learning support materials

C&IT skills survey of new undergraduates - 2002 95% own a mobile phone 5% have a PDA 38% take part in online discussions 59% take part in online chat 49% have a qualification in IT 54% regularly switch between several

applications when using a computer

C&IT skills survey of academic staff - 2000 n = 576 (53%) 90% perceive themselves to have

good/advanced email skills 67% perceive themselves to have

good/advanced web browser skills only 17% of staff had taken part in online

discussions only 12% of staff had taken part in online

chat

Catalyst Introduction encouraged

continual reflection on teaching Postgraduate Certificate in HE

compulsory for new academics from Jan 2001

Academics working together within courses

Factors influencing enhancement How used How integrated into curriculum Quality of the parts eg content,

assessment, communication (earlier research)

Enhance learning if the parts enhance learning

Flexibility allows lecturers to choose parts

Is the whole greater than the sum of its parts? But is a VLE more than the pieces

put together?

Evaluating the impact of a VLE on learning and teaching Incredible uptake of duo which has become

embedded into the University Staff and students believe duo has enhanced

teaching and learning Academics are gradually using more features

so this may result in more interactive and innovative teaching

Students expectations are increasing as the basic use of a VLE is assumed

Further analysis is being undertaken

References Collis, B. and Messing, J. (2001) Usage, attitudes and

workload implications for a web-based learning environment, Association of Learning Technology Journal 9 (1)

Collis, B. and Moonen, J. (2001) Flexible Learning in a Digital World, London, Kogan Page  

Laurillard, D. (1993). Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. London: Routledge.

Richardson, J., & Turner, A. (2000). A large-scale ‘local’ evaluation of students’ learning experiences using virtual learning environments. Educational Technology and Society, 3 (4).

Salmon, G. (2000). E-moderating: the Key to Teaching and Learning Online. London: Kogan Page.