effective application of quality assurance & enhancement procedures to e-learning courses
DESCRIPTION
Effective application of quality assurance & enhancement procedures to e-learning courses. Workshop at University of Reading 7 July 2008. Welcome. Project team. Workshop background. IOE Pathfinder Pilot – PREEL Included revision of QA procedures for e-learning courses - PowerPoint PPT PresentationTRANSCRIPT
Effective application of quality assurance & enhancement
procedures to e-learning courses
Workshop at
University of Reading
7 July 2008
Welcome
Project teamInstitute of Education Dr Harvey Mellar
Dr Magdalena Jara
University of Reading Maria-Christiana Papaefthimiou
University of Derby Dr Dave O'Hare
UoL External System Brian Sayer
University of Teesside Eileen Webb
Quality Assurance Agency
Dr David Cairns
Workshop background IOE Pathfinder Pilot – PREEL
Included revision of QA procedures for e-learning courses
IOE research study on QA/QE in e-learningExamined effectiveness of internal quality assurance procedures for online courses
Reading Pathfinder DIRECTDriving Institutional Reform: Exploring Change with Technology
Quality Assurance AgencyIn 2006 the QAA began a new cycle of audits with a more enhancement-focused approach
Purposes of the workshop Identify any issues in quality assurance/enhancement
procedures that arise from the use of technology in teaching and learning
Identify how these issues are reflected in your own current institutional procedures
Identify how the internal QA/QE procedures of other institutions deal with these issues
Identify ways to make your institutional QA/QE procedures more effective in assuring and enhancing the quality of TEL.
Day overview10.00 Introduction
10.15 Part 1: Making terms explicit
10.55 Part 2: Internal QA/QE procedures for TEL
11.35 Coffee
11.45 Part 3: Issues affecting effective implementation of internal QA/QE procedures for TEL
12.45 Lunch
13.30 Part 4: Planning actions to improve the effective application of QA/QE procedures to TEL
14.40 QA/QE in e-learning SIG
15.00 Close
Part 1
Making terms explicit
Technology Technology enhanced learning E-learning On-line teaching
QAA’s framework - IQAPs
Internal quality assurance proceduresEach [HEI] is responsible for the standards and quality of its academic awards and programmes. Each has its own internal procedures for attaining appropriate standards and assuring and enhancing the quality of its provision.In particular, institutions address their responsibilities for standards and quality through:
the assessment of students; their procedures for the design, approval, and the
monitoring and review of programmes.
QAA (2003) A brief guide to quality assurance in UK higher education. www.qaa.ac.uk
Quality enhancement.. the process of taking deliberate steps at institutional level to improve the quality of learning opportunities....[…] … an aspect of institutional quality management that is designed to secure, in the context of the constraints within which individual institutions operate, steady, reliable and demonstrable improvements in the quality of learning opportunities.
QAA (2006) Handbook for institutional audit: England and Northern Ireland
The definition of 'enhancement' QAA has adopted …. leaves room for institutions to follow their own definitions of 'enhancement'. Some institutions may define enhancement as 'continuous improvement', others as 'innovation' and there may be other definitions.
QAA (2007) Audit and enhancement: further guidance for institutions and QAA audit teams
QE and TEL Enhancement
improvement or innovation
TEL Is an innovation, hence enhancement? Need to improve TEL?
Task 1(15 minutes)
In each group…
What do you mean by QE? How can you use the internal quality assurance
procedures to achieve QE?
Report back – one point per group
Swinglehurst, D. (2006) Peer Observation of Teaching in the Online Environment: an action research approach. Final Report. Centre for Distance Education
QA/QE – the differenceQuality assurance and quality enhancement are not the same as the first is concerned with determining that objectives and aims have been achieved, while quality enhancement is concerned with making improvements.
Quality enhancement is part of a wider framework in which quality control, quality assurance, quality enhancement and transformation are stages in the management of quality
Common belief that quality assurance leads naturally to quality enhancement is not correct, as most quality assurance efforts are by and large concentrated in accountability
Middlehurst, R. (1997), 'Enhancing Quality'. In F. Coffield and B. Williamson (eds), Repositioning Higher Education. Buckingham: SRHE and Open University Press.
Retrospective vs prospectiveQuality assurance may be either ‘retrospective’ or ‘prospective’ depending on the type of quality it is aiming to assure.
Retrospective QA looks into the past to make a judgement with a focus on accountability
Prospective QA is concerned with the present and future, focusing on quality as fit for purpose, and encouraging improvement
Defines QE as the internal mechanisms that an institution puts in place to continually review and improve practice
Biggs, J. (2001), 'The reflective institution: assuring and enhancing the quality of teaching and learning'. Higher Education, 41, 221 - 238
Accountability vs improvement… external evaluations have accountability and compliance focuses rather than the encouragement of continuous quality improvement of the student experience. In most institutions where it occurs, improvement of the student experience is a function of internal review and monitoring processes, usually heavily reliant, nowadays, on student feedback, examiners reports, internal improvement audits, periodic revalidation of programmes of study and staff teams critically self-reflecting on their everyday practice.
All internal processes of quality monitoring have a greater effect on the quality of the provision than the external monitoring processes
Harvey, L. (2005), 'A history and critique of quality evaluation in the UK'. Quality Assurance in Education, 13 (4), 263 – 276.
Part 2
Internal QA/QE Procedures for TEL
Internal QA procedures
1. Validation
2. Module evaluation
3. Annual course review
4. Periodic course review
5. Student representation
6. External examiner
7. Peer observation
8. Other…
Task 2(15 minutes)
In each group…
Considering the internal QA procedures in place in your own institution, discuss:
Does TEL require new/changed internal QA procedures?
Does TEL enable new/changed internal QA procedures?
Report back
National overviewHEIs' Audit Reports overview regarding QA of e-learning
Total = 129
Mentions particular QA procedures of e-learning
(80) 62%
Mentions QA of e-learning in general
terms only(10) 8%
Does not mention QA of e-learning
(8) 6%
Reports no e-learning, (31) 24%
(QAA audits 2003-2006)
Percentage of HEIs mentioning each QA procedure for e-learning
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
External Examiners
Student feedback
Student Representative
Staff feedback
Team meetings
Annual Review
Periodic Review
Validation/approval
Peer review
Other QA procedures
HEIs reporting variations in the application of the procedures to e-learning
ProcedureTotal
(n=129)
External Examiners
Student feedback 14 11%
Student representatives
Staff feedback
Team meetings/staff development
Annual Review 11 9%
Periodic Review 10 8%
Validation/Approval 35 27%
Peer Review
Other QA procedures
VariationsStudent feedback
Specially tailored forms Implementation of online surveys and discussion
forums to collect feedback (to improve low response rates)
AnnualReview
Additional issues to be considered by review team Additional scrutiny Additional section to be discussed
PeriodicReview
External assessor with experience in e-learning To include an evaluation of the development and
effectiveness of teaching, learning and assessment strategy and method
Validation Additional consultations/approval steps: Consultation/approval by specialised units Proposals seen by additional committees
Sustainability: Additional scrutiny of economic viability and availability
of resources (risk assessment, contingency plans)
E-learning mode of delivery: Demonstrate appropriateness of delivery method,
explain how student discussions will be facilitated
Materials: Provide sample materials for review Internal and external peer review
Validation Panel members: At least one member of the panel needs to have
expertise in the area
Academic staff: Extra checks on staff development needs
Coffee break
Part 3
Issues affecting effective implementation of internal QA/QE procedures for TEL
QA/QE for TEL
The literature identifies differences between TEL and campus-based learning:
Disaggregated processes
Distributed teams
Distant location of students
Openness to review
Research study Four case studies
Postgraduate courses in a range of universities Explore how dual-mode HEIs approach the application of their
internal QA procedures to their e-learning courses
For each case study QA documentation
Interviews with stakeholders
Comparative examination of data: Map of issues not captured by the QA procedures Identification of those aspects of the courses which were
impacting on the implementation of the procedures
Research resultsThe application of the QA procedures was affected by:
Organisational context in which courses are located
Disaggregated processes
Distributed teams
Distant location of students
Openness to review
1. Organizational context - detachmentOnline courses were often in a ‘detached’ position in their institutions, which created both a sense of autonomy and isolation
On-line courses off senior management’s agendas Courses without central oversight This led to failure to collect relevant information
2. Disaggregated processesDistributed organisation of teams in terms of roles affected the levels of coordination and communication among team members
Unclear distribution of responsibilities Collaboration with external and specialised units added further complexity
3. Distributed teams - coordination
Online courses were often taught by a mixture of full time tutors, tutors with fee-based contracts and tutors working from home or elsewhere. Course teams were scattered and course leaders were often not aware of the coordination requirements of a distributed team
On campus team with face-to-face meetings, tutors fully integrated Course leaders tend to use the same coordination and feedback mechanisms used for on-campus staff
4. Distant location of studentsDistant location of students also affected the quality assurance mechanisms, as it obstructed the implementation of some of the procedures in their current form (e.g. the use of student representatives)
Compensated for by a strong and trusting relationship between students and tutors Opportunity was only occasionally taken up by teams
Effect on internal procedures
Annual Reviews Module evaluations Team meetings
Annual Reviews Approached as an administrative burden, which had to
be written up just for accountability purposes Senior management is perceived as not prepared/not
able to understand the relevance of TEL issues
Annual review often detached from its enhancement function Perceived as an event rather than a process Teams render two different accounts, one for external
consumption and another to be used internally
Module evaluations Distant location of students…
low response rates teams discarding the results as invalid, regardless
of their content
Distributed organisation of teams… unclear allocation of responsibilities
led to responses being left untouched or only superficially analysed
Team meetings Distributed teams
absence of a structure of formal meetings put at risk the team’s capacity to deal with the
issues identified and the monitoring of their resolution
Evidence: consistency in the information managed by team members mechanisms for coordination and feedback relying on informal
encounters
Adjustments to overcome the limitations usually took the form of increasing the formalisation of the communication and coordination channels either online or face-to-face
Task 3(35 minutes)
Select one internal QA procedure
In each group: Quickly review the procedure’s purpose and how should it work Quickly review how the procedure is intended to aid quality
enhancement Describe any issues impacting on the use of this procedure for
QA/QE arising from the use of TEL Suggest solutions
Produce a one page written report: List issues identified affecting the procedure selected List suggestions for changes/improvement
Lunch break!
Part 4
Planning actions to improve the effective application of QA/QE
procedures to e-learning courses
Techniques - research & practice Validation criteria
Evaluation of e-learning courses
Embedded evaluation
PROPP Peer Observation
Periodic review framework
1. Additional validation criteria (IoE) Staff experience of, or training in, use of on-line teaching
methods Evaluation plan Arrangements for the support of students with disabilities Explicit course management procedures (including the
role of administrators, and a named person responsible for quality assurance)
Arrangements for obtaining feedback from students Explicit arrangements for tutor peer observation
2. Recommendations for evaluation Obtain feedback from all stakeholders: students, tutors,
administrators, technical support
Organise frequent formal staff meetings, face-to-face and/or online, with agendas defined by the staff, and covering all key issues
Carry out evaluation as an integral part of the activities of the course, both during and after the course
Ensure that responsibility for collection and analysis of results is clearly assigned
Take advantage of the technology in use in the course to collect feedback
Evaluation of e-learning courses WLE Occasional Paper 4
3. Embedded evaluation Project studying e-learners’ experiences in a mixed-
mode course
Student learning is enhanced by evaluation which is concurrent with teaching Embedding evaluation tasks as part of the activities of the course
encouraging students to think about their own learning and how the course design, materials and/or activities have supported them (or not) in this pr
Daly, C. Pachler, N., Pickering, J. & Bezemer, J. (2006)
A study of e-learners’ experiences in the mixed-mode professional degree programme, the Master of Teaching.
Examples of evaluation questionsWriting online or in face to face seminar
MASTER OF TEACHINGOnly two of you had prior experience of learning online before joining the MTeach, so this has been a new way of communicating to learn for everybody else in the group. These discussions form a significant alternative to talking about issues at the face-to-face days or in traditional seminars. What has it been like to ‘discuss’ by using writing to communicate with each other like this?
RESEARCH SYNTHESIS FOR POLICY AND PRACTICE
Conducting a systematic review is a collaborative process, involving both face-to-face meetings and online discussions/activities. The course has been designed to mimic this process - carrying out activities both in the workshops and online - how has it been to learn like this? Please offer your thoughts…
4. PROPP Peer Observation
Project studying the implementation of peer observation of teaching for tutors of an online course
Development & piloting of approach to peer observation regular meetings course tutors are held
at each meeting, individual tutors bring an example of a problem, issue or a project (e.g. feedback prepared for a student, transcript of a virtual seminar, marks awarded for student’s work)
this material is used as the basis for discussion with other tutors supporting reflection, and challenging assumptions
Swinglehurst (2008) Peer Observation of Teaching in the Online Environment: an action research approach
Example of a PROPP meeting Topic How do we assess our students?
Evidence: Two student tutor marked assignments with the marks and tutor feedback
Main outputs The instructions in the assignment task were ambiguous, and so needed to
be altered The task needed to be broken down into smaller components Written feedback should include specific examples rather than relying on
general comments which may not be well understood by students Feedback should refer back to previous feedback given to students, it is
worth exploring the possibilities for enabling this more easily within the VLE
5. Periodic review frameworkUniversity of Reading
Pathfinder DIRECT - Driving Institutional Reform: Exploring Change with Technology
Plenary discussion
Summaries of discussions from Task 3
Discussion:
How can your institutional procedures be made more effective in assuring and enhancing the quality of TEL?
What obstacles stand in the way of improving these institutional procedures ?
QA-QE SIG JISCmail list: ‘QA-QE SIG’
to join go to http://jiscmail.ac.uk and sign up
More information: www.lkl.ac.uk/research/qa-elearning
Evaluation of workshop External evaluator (Charlotte Creed)
By email: link to online evaluation (10 min max)