Benchmarking eLearningDayle Soong, eLearning Adviser

Benchmarking eLearningThis project will identify benchmarks and methods to measure performance in regard to eLearning at The

University of Adelaide in order to inform planning and to report on change and improvement in the use of MyUni to

the LTSG (Learning Technologies Strategy Group), LTOG (Learning Technologies Operations Group) and LTT (Learning

Technologies Team).

Usage

Policy Strategy

Engagement

Support

In the beginning…

Sub Cruce Lumen: light under the Cross. Against the vast Australian skies the University of Adelaide shines as a beacon bright as the Southern Cross: it illuminates new knowledge and is a lodestar of enlightened learning for South Australia and the world.

“

”

Internationally higher education is now being reshaped by globalisation and thedigital revolution… (p5)

We will reach out to a still broader student body • providing flexible learning and e-learning to meet new student needs (p6)

students need confidence their learning will be better supported by digital learning resources. Where high-quality content can be effectively delivered online with demonstrable pedagogical integrity it will be, to free staff time for small-group discovery where the focus can shift to learning and problem solving.

the University will treble its expenditure on IT learning support and e-learning,

˃ Online learning is available in 100% of undergraduate programs

˃ MyUni is actively used in 80% or all courses in all faculties

˃ All lectures are recorded and available for download

˃ We should provide an online delivery option for all student-focused administrative and transactional services, including learning administration services, such as the online submission of assignments. (from SeE)

Operational Plan

How will we know if we are realising the Beacon??What do we want to measure?

What needs to be done to improve?

Defined set of benchmarks and measures

Usage

• Content• Communication and collaboration• Assessment• Grade and monitor progress

User Engagement

• Distinct users• Frequency• Duration

Support

• Requests• Incidents• Resolution

Strategy and Policy

• ACODE benchmark 1

Getting the data

SELECT *FROM [Users]WHERE [Clue] > 0No records found.

What data are we getting?MyMedia Data Description Example RecordColumn A MyUni Course ID associated with MyMedia sessions 3410_PSYCHOL_1000Column B MyUni Course Description Psychology IAColumn C Count of the available MyMedia Sessions (individual

lecture recordings and any uploaded media items) associated with the Course ID

25

     MyUni Data Description Example RecordCourse ID MyUni Course ID is a combination of the Term Code,

Subject Area Code and Catalogue Number3410_PSYCHOL_1000

Course Name The Course Name Psychology IACourseDesc The course description. Useful to identify the 'child'

courses in the COMBINED coursesThis course, together with PSYCHOL 1001, provides an introduction to the basic concepts and core…

Discipline The Subject Area PSYCHOLUnit N.A. can be looked up from Peoplesoft data  School N.A. can be looked up from Peoplesoft data  Faculty N.A. can be looked up from Peoplesoft data  Institution The University of Adelaide uofaPS Year Year the course ran measured by years after 1980

(for example, 34 refers to courses that are ran in year 2014)

34

Approx Level Year level of the course within its program. Derived from first digit of Catalogue Number.

1

Class Size Number of students enrolled in the course 687

Available Whether the course has been made available to students (Y = Yes, N = No) Y

AllContent Count Count of content items contained in the course i.e. uploaded files or text/media items entered directly into MyUni 230

AvailableContent Count Count of content items which are available/visible to students 219

Pdf Count Count of all PDF files which are available/visible to students 93

PowerPoint Count Count of all PowerPoint files (.ppt/.pptx) which are available/visible to students 5

Word Doc Count Count of all Word files (.doc/.docx) which are available/visible to students 1

Image Count Count of all image files (added from Build Course -> Image) which are available/visible to students. Does not include count of images which are added as 'File' or 'Item'.

0

Audio Count Count of all audio files (added from Build Course -> Audio) which are available/visible to students. Does not include count of audio files which are added as 'File' or 'Item'.

0

Video Count Count of all video files (added from Build Course -> Video) which are available/visible to students. Does not include count of video files which are added as 'File' or 'Item'.

0

ExtLink Count Count of all web links (added from Build Course -> Web Link) which are available/visible to students.

2

SCORM Count of SCORM items (learning objects produced outside of MyUni that are scorm compliant e.g. an Articulate presentation including a quiz that will interface with GradeCenter)

2

MyMedia PlaylistCount Count of links to MyMedia playlist feeds (one feed is likely to contain many individual session recordings, but individual sessions are not counted here)

3

MyMedia SessionCount Count of links to MyMedia sessions (added with Embed code) - this is NOT a count of the number of sessions recorded and available for this course

14

TLE Links to Online Content items managed by the DRMC (copyright journal articles and book chapters) Note: TLE (The Learning Edge) is now called Equella

0

Cal Count Semester Count of Calendar entries which has a start date within the current semester 0

Task Count Semester Count of Task entries which has a due date within the current semester 0

Avail Announcement Count

Count of Announcement which has a start date within the current semester 28

avail forum count Count of Discussion forums which are available/visible to students 0

Forum Post Count Count of posts in the available discussion forums (does NOT include Group discussion boards) 0

Forum Auth Count Count of distinct/unique authors in the posts (does NOT include Group discussion boards) 0

Blog Count Count of Blogs which are available/visible to students. Includes both Individual and Group blogs. 0

Blog Post Count Count of posts in the available blogs 0

Journal Count Count of Journals which are available/visible to students. Includes both Individual and Group journals.

0

Journal Post Count Count of posts in the available journals 0

Wiki Count Count of Wikis which are available/visible to students. Includes both Course and Group Wiki. 0

Collaborate Count Number of Blackboard Collaborate session that have been deployed in a course and availalble/visible to students

0

Group Count Count of Groups 5

Group Gt2 Count Count of Groups which has at more than 2 students. 5

Voice Podcast Count Count of Voice Podcaster which are available/visible to students 0

Voice Board Count Count of Voice Board which are available/visible to students 0

Voice Pres Count Count of Voice Presentation which are available/visible to students 0

Adaptive Rel Rule Count Count of Adaptive Release Rules 0

Assessment Count Count of MyUni Tests, Surveys and Mobile Compatible Test which are deployed in a course 6

Test Count Count of MyUni Tests which are deployed in a course 6

Survey Count Count of Surveys which are deployed in a course 0

Mob Assign Count Count of Mobile Compatible Tests which are deployed in a course 0

MapleTa Assign Count Count of Maple T.A. Assignments which are deployed in a course 0

MyUni Assign Count Count of MyUni Assignments which are deployed in a course 6

Icc Assign Count Count of ICC Printed Assignments which are deployed in a course 0

TurnItIn Assign Count Count of Turnitin Assignments which are deployed in a course 0

Grade Centre Column Count

Count of Grade Centre Columns 35

Grade Centre Student Visible Count

Count of Grade Centre Columns which are visible to students 35

Early Warning Rule Count

Count of Early Warning System rules which are available (4 are available and active by default) 4

Rubric Count Count of MyUni Rubrics (Does NOT include Turnitin Rubrics) 0

Access Count Sem Count of number of ‘hits’ or times that a page within a course was accessed over the semester 230098

User Count Sem Count of active users who have accessed the course over the semester. Includes students, academic and professional staff.

775

sum of all assignment types

Sum of eSubmission assignments (sum of ICC, Turnitin and MyUni Assignments) 6

sum of all test types Sum of MyUni tests and Maple TA tests 6

hits per user Access Count Sem / User Count Sem 296.9006452

MyMedia Available Session Count

Count of the available MyMedia Sessions (individual lecture recordings and any uploaded media items) associated with the Course ID

25

Current Results

Why?

• Some areas have other LMSs

• Some areas run courses with very low

numbers of students

• Some areas have course structures that do

not align with PeopleSoft course codes

• Some areas create courses for administration

purposes

• Some areas have practical courses

• Some areas teach in spaces without AV

• Some areas ARE SPECIAL

Recommendations from 2013 baseline report1. Guidelines for minimum usage of MyUni are developed

and proactively disseminated to and applied by staff.2. Administration processes that create MyUni courses

should be examined in order to improve the accuracy of reporting.

3. Specific strategies should be employed in 2014 to increase utilisation of MyMedia lecture recordings and eSubmission of assignments to meet targets set by the Beacon of Enlightenment and Student e-Experience Strategy.

4. Consideration should be given to increasing support for browsers and devices that are most commonly used by staff and students.

5. The new Closure Code Matrix (refer to Appendix C) should be implemented in LANDesk and utilised by the new MyUni Support Team to provide better information on support issues.

6. Policies are developed that specify the use of technologies to support learning and teaching covering all aspects and stakeholder perspectives.

7. Consideration should be given to undertaking a comprehensive ACODE or eLearning Maturity Model assessment in the future.

8. Combined or merged courses in MyUni should have a clear identifier that enables the source courses and year levels to be automatically determined.

9. Reporting on eLearning performance should be sought from areas of the University who are using alternative tools and systems.

10. The Benchmarking eLearning project should be re-established in 2014 with the following objectives:– Automation of eLearning reporting mechanisms

with the implementation of an integrated business intelligence reporting application;

– Selection of a suitable benchmarking partner in order to compare our performance with our peers.

eMM/ACODE

• eLearning Maturity Model• Assess the institutions eLearning capability maturity using

the eMM framework• Compare with other Australian Universities• Provide a visual representation of strengths and gaps• Identify strategies for improvement that will have the most

impact

The future… automated and systematic

Data Warehouse

MyUni and MyMedia