virtual schools success criteria draft final

iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 1

Success Criteria for Virtual Schools

- Pick&Mix

Professor Paul Bacsich

Sero Consulting Ltd and Matic Media Ltd


Topics

1. Introduction, disclaimers and acknowledgements

2. History of Pick&Mix in universities

3. Why is he telling us this?

4. Pick&Mix

5. More recent history: application to colleges and high schools

6. Reflections on this process


1. Introduction, disclaimers and acknowledgements

iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012

Who is this talk for?

Want to know about comparing your virtual school with other ones?

Want to know about the “tradecraft” of benchmarking and quality reviews?

Want to adapt or update a benchmarking system?

Want to learn some of the underlying principles of such schemes?

4


Pick&Mix is an “Open Educational Methodology” for benchmarking online learning, developed

by Paul Bacsich and available for all to develop and modify, but the names

“Pick&Mix” and “ELDDA” are reserved for the “main sequence” of development

Thanks to many, including UK HE Academy, JISC, EU Lifelong Learning Programme

(Re.ViCa and VISCED), Manchester Business School, University of Leicester and

Sero Consulting Ltd for support


2. History of Pick&Mix in universities

6


Benchmarking online learning

At national level, started in 2004-05 in UK and New Zealand– Soon spread to Australia– Not closely linked initially to quality agenda

At European level, developments include E-xcellence and UNIQUe– Some earlier work from OBHE, ESMU etc– Later, developments in other projects– Increasingly, links made to quality agenda and to

critical success factors


Pick&Mix history

Initial version developed in early 2005 in response to a request from Manchester Business School for an international (largely US) competitor study

Since then, refined by literature search, discussion, feedback, presentations, workshops, concordance studies and four phases of use – sixth and seventh phases now

Forms the basis of the wording of the Critical Success Factors scheme for the EU Re.ViCa project and now being used to develop a similar scheme for Virtual Schools in VISCED


Benchmarking e-learning (UK)

Foreseen in HEFCE 2005 e-learning strategy for universities in England (later for Wales via HEFCW)

Higher Education Academy (HEA) oversaw itFour phases – 82 institutions – 5 methodologiesTwo consultant teams – one run by myself My team benchmarked over 40 institutions using

4 methodologiesIncluding 24 using Pick&Mix – now well over 30


What HEFCE wanted

“Possibly more important is for us [HEFCE] to help individual institutions understand their own positions on e-learning, to set their aspirations and goals for embedding e-learning – and then to benchmark themselves and their progress against institutions with similar goals, and across the sector”


Methodologies in UK HEOf the five methodologies used in the UK from 2005 on, only

one survives that is actively under development, refined annually, public domain and available for supervised or self-applied use in institutions, via funded projects and commercially:

Pick&Mix

In other countries’ HE systems:– eMM is in a similar situation in New Zealand– Quality Matters is widespread in US– ACODE is used somewhat in Australia– On the continent of Europe there are a few methodologies but

fostered purely by EU-funded projects


Pick&Mix overview

Focussed on online learning, not general pedagogyDraws on several sources and methodologies – UK and

internationally (including US – especially Quality on the Line) and from college sector

Not linked to any particular style of online learning (e.g. distance or on-campus or blended)

Oriented to institutions with notable activity in online learning

Suitable for desk research as well as “in-depth” studiesSuitable for single- and multi-institution studies


BECAUSE IT CAN BE USED FOR COLLEGES AND SCHOOLS ALSO

3. But why is he telling us this?

13


4. Pick&Mix

Criteria and metrics


Criteria

Criteria are “statements of practice” which are scored into a number of performance levels from bad/nil to excellent

It is crucial that these statements are in the public domain – to allow analysis & refinement

The number of criteria is also crucial: 24Pick&Mix originally had a core of 20 – based on

analysis from the literature (ABC, BS etc) and experience in many senior mgt scoring meetings


Pick&Mix: 20 core criteriaRemoved any not specific to online learning

– Including those in general quality/accreditation schemesCareful about any which are not provably success factorsLeft out of the core were some criteria where there was

not yet UK consensus Institutions will wish to add some to monitor their KPIs and

objectives. Recommended no more than 6.– Pick&Mix now has over 70 supplementary criteria to choose from– more can be constructed or taken from other schemes

These 20 have stood the test of four phases of benchmarking with only minor changes of wording– originally 18 - two were split to make 20


Pick&Mix Scoring

Use a 6-point scale (1-6)– 5 (cf Likert, MIT90s levels) plus 1 more for

“excellence”Contextualised by “scoring commentary”There are always issues of judging

progress especially “best practice”The 6 levels are mapped to 4 colours in a

“traffic lights” system – red, amber, olive, green


Pick&Mix System: summary

Has taken account of “best of breed” schemes

Output and student-oriented aspectsMethodology-agnostic but uses underlying

approaches where useful (e.g. Chickering & Gamson, Quality on the Line, MIT90s)

Requires no long training course to understand


P01 “Adoption” (Rogers)

1. Innovators only2. Early adopters taking it up3. Early adopters adopted; early majority

taking it up4. Early majority adopted; late majority taking

it up5. All taken up except laggards, who are now

taking it up (or retiring or leaving)6. First wave embedded, second wave under

way (e.g. BYOD-learning after e-learning)


P10 “Training”

1. No systematic training for e-learning2. Some systematic training, e.g. in some projects

and departments3. Institution-wide training programme but little

monitoring of attendance or encouragement to go4. Institution-wide training programme, monitored

and incentivised5. All staff trained in VLE use, training appropriate to

job type – and retrained when needed6. Staff increasingly keep themselves up to date in a

“just in time, just for me” fashion except in situations of discontinuous change


Supplementary criteria - examples

IT reliabilityMarket researchCompetitor research IPRHelp DeskManagement of student expectationsStudent satisfactionWeb 2.0 pedagogy


Adaptation to colleges and high schools (both virtual and blended)

5. More recent history

22


Virtual post-secondary institutions

The EU Re.ViCa project 2007-09 (Review of Virtual Campuses) did a great deal of work to refine and “Europeanise” the Pick&Mix criteria, focussing on – Critical Success Factors for virtual campuses

A slightly revised scheme was produced, oriented to institutions where distance learning was the prevalent approach

23


Distance Learning Benchmarking Club 2009-11

In summary: Universities, encouraged by a UK-funded project, formed a group to benchmark themselves:– University of Leicester (UK)– Royal Swedish Institute of Technology (KTH)– Lund University (Sweden)– University College Gotland (Sweden)– Thompson Rivers University (Canada)


VISCED 2011-12

A further iteration is in final stages to produce a scheme of Critical Success Factors for virtual schools– With initial focus to European virtual schools– Since these are smaller and few have ever

failed, this is a hard task

25


Draft key success factors for European virtual schools

1. Usability (for all participants)

2. Strategy (for online learning)

3. Recruitment and training (fused?)

4. Evaluation (of programmes) – often informal but effective

5. Reliability (of system)

6. Leadership (with knowhow, down and up – fused?)

7. Organisation (formal and informal)

8. Learning outcomes (often individualised)d

9. Use of resources (not key in universities)

10. Market research?

26


6. Reflections on this process

27


Too many concepts

Benchmarking

StandardsQuality

Accreditation/approval

/kitemarking

(Critical) Success Factors

Online learning is only a small part of the quality process – how can agencies and assessors handle five variants of the concept across many separate methodologies?


My view for ENQA – the pyramid

Critical Success Factors, a selection of:

Benchmarking, which split into sub-criteria:

Quality, split into

Detailed pedagogic guidelines ----------

Criteria are placed at different layers in the pyramid

depending on their “level”

Leadership level

Senior managers


Adaptability

The transition of a basically university-originated methodology to schools is feasible, particularly for larger high schools (virtual or blended)

The sectors are not as different as territorial experts like to believe

But the methodology has to be theoretically sound and well-researched

There are other extensions - eg for OER !

30


Merging US and EU practice?But could we cross the transatlantic divide?There were some discussions between EU

institutions and QM over a common HE system (UNESCO would love that)

For schooIs, we need to look at iNACOL guidelines – meeting after the Symposium

Since Pick&Mix was heavily influenced by US experience (Ehrmann; Quality on the Line) there should be hope – at first sight it looks promising

A hierarchical (pyramid) approach will be key

31


A view after 7 years of Pick&MixMethodologies do not survive without regular

updating by a design authority– this is difficult in a leaderless group context

Forking of methodologies needs dealt with by folding updates back to the core system– otherwise survival is affected

Complex methodologies do not survive well A public criterion system allows confidence,

transparency, and grounding in institutionsSectoral boundaries can be overcome


ReferencesThe Pick&Mix system is detailed at http://www.matic-media.co.uk/benchmarking/PnM-2pt6-beta3-full.xlsx

A key paper on the international aspects is:

“Benchmarking E-Learning in UK Universities: Lessons from and for the International Context”, in Proceedings of the ICDE conference M-2009, Open Praxis – http://www.openpraxis.com/files/Bacsich%20et%20al..pdf

A specific chapter on the UK HE benchmarking programme methodologies is:

“Benchmarking e-learning in UK universities – the methodologies”, in Mayes, J.T., Morrison, D., Bullen, P., Mellar, H., and Oliver, M.(Eds.) Transformation in Higher Education through Technology-Enhanced Learning, York: Higher Education Academy, 2009 – http://www.heacademy.ac.uk/assets/documents/learningandtech/Transforming-07.pdf

virtual schools success criteria draft final

Design

critical success

wide training

higher education

benchmarking

online learning

pickamp

virtual schools

systematic