building moocs: scalable course development & delivery
TRANSCRIPT
Building MOOCs: Scalable Course Development &
Delivery
Learning E-ssentials 2013
Ken CurrieOpus Learning
12th December 2013
1
This Talk
• Three main themes:– Why Design is critical, but not necessary complex– Why an Engineering approach is the only way to
build and deliver– The implications of Delivery, Support and
Assessment• Fourth theme?
– Implementation– MOOC specifics
2
Useful Quote
• “Large scale online learning could provide continuity for learners throughout their lives between different education systems.” – Association of Learning Technology Draft Green
Paper 2013.
3
Useless Quote
• “Enterprise technology for authoring, publishing and distributing is still largely in an immature state.”– Steve Barden, Lead Consultant, LINE
Communications, Inside Learning Technologies & Skills Magazine - November 2013
4
MOOCs or Programmes?
• Some of you may be here to hear about MOOCs– The story doesn’t really change, though there are
some delivery and operational extras• An example: Swim Coaching
– http://ties.courseworker.com– University of Iceland offering– Exploitation of the output of an EU LdV Project
5
History of Opus
• A spinout from CAPDM.com– Developing and delivering Masters programmes
for over 20 years• Opus delivers highly interactive courses,
delivered via platforms such as Moodle– With tools to manage and monitor learning with
or without tutor support. • An SQA Approved Centre
– A wholly online College
6
Show of hands
• How many people here build and deliver online distance education?
• What scale? Is it a challenge?• Opus has 99 SQA SCQF Level 7/8 courses
– All content full• Edinburgh Napier University’s Global Online
– 18 masters courses @ 20 credits– All content full
• Sound online pedagogy?7
Case: Heriot-Watt University
• Edinburgh Business School Online MBA– Europe’s largest DL MBA; the world's most international– 14,000 successful MBA graduates from 180 countries– Twice awarded the Queen's Award for Exports– Students from over 50% of Fortune 100 companies– 4,400 study hours; English, Spanish, Chinese, Arabic. – 800 different deliverables for 37 released courses in print and online.– 20 million words of English; 8,000 multi-lingual vector diagrams, etc.– 14 year unbroken record of revision controlled single source XML
– Has earned over £150 million– Now on its third VLE – but not a single character has had to change to
accommodate these moves!8
MOOCs vs Programmes
• Not a lot of difference?– Opus aims for tutor-lite courses (so high quality)– MOOCs take this to the limit
• Content full– Socially constructed content? No.– Opus delivers all the content a student needs– Complete, interactive, interesting!
• But develop social learners
9
Theme 1: the Design Challenge
• To build, deliver and manage– 99 HN courses for Opus– 18 masters level courses for Edinburgh Napier– … + others!
• To manage > 100 academics/teachers/authors• To manage very large domains of content• To deliver to many platforms
– Moodle, Blackboard, ePub, tablet/iPad, PDF, …
10
Strategic , not Tactical
• These challenges must be planned for– A strategy
• Many (large) organisations have no such strategy for this scale of development– Does yours? Most universities don’t, for example
• Structure & Content Management is critical– Communicate a consistent and understood
message to all parties– Minimise the entropy
11
Three Linked Parties
• Authors, engineers, students– Three links in a chain
• Many authors write great works– Few write good distance learning materials– Learning materials have to be fit for purpose
• Numerous authors– They must all hear the same message– They must all understand it
12
Talk in simple terms - components
13
Standard programme design
• Standardise course design within programmes
14
Simple message
• This structure carries a message that is:– Familiar: all components understood– Understood: can be communicated safely– Scoped: easy to put a scale to components– Quality Assured: all content must comply
structurally, in scope, academically, pedagogically
15
The Three Links
• Communicating the message– Authors work to a common template– Students consume a consistent product
• Know how to use it• Appreciate the clarity?• Keep retention rates up?
– Engineers, such as CAPDM, can build the product• Cost effectively• Consistently
16
SQA: the Start Point
• SQA provide ‘arrangement’ documents for all– Frameworks (e.g. HND Business) and – Units (e.g. Business Accounting)
• Unit specifications of:– Outcomes– Knowledge and/or Skills– Evidence Requirements– + Assessment Guidelines
• Universities have similar, perhaps less rigorous, guidelines 17
Theme 2: Engineering
• These courses are complex (in a structural sense)– Content-full, pedagogically rich?– Highly integrated and linked– Delivered to multiple platforms– Highly functional and interactive– Rich navigation and search
• Too complex to build by hand– The typical ‘bricolage’ approach cannot work
18
Standards
• All course materials are designed for their learning potential– Completely agnostic wrt technology– Single source masters of all content– Standards based: XML, MP3/4, etc.
• Structural descriptions of the assembly of this content through …– An XML manifest
• Focus on learning19
Publishing Goals
• To go from the single source master to full delivery– On all chosen platforms (e.g. Moodle)– In about 5 minutes– All published forms (xHTML, PDF, …)– Automatically
• Delivery to be– Complete and fully integrated– Fully functional
20
Where’s the middle bit?
• Where’s the instructional design, the delays, the faffing about with the VLE?– Eliminated!– This is all in the engineering, so fully automated– Styles, structures, etc., are all parameterised– Consistency of interpretations and delivery
• The Authors/Module Leaders can be in control– … in theory!
21
Semantic Mark-up
• Word, HTML and many other formats are ‘dumb’– XML can embody rich metadata and semantics– Though “File | Save As” XML is also dumb
• Our job? – To capture all learning content in a rich mark-up– To apply a consistent and rich semantic
interpretation to that content for delivery
22
Empowering Authors
• Authors write what they need– If the mention a ‘Forum’ then we guarantee its
presence and a link to it• It is our job to interpret their needs and to
engineer a solution– You will never hear “we can’t do that as the
platform doesn’t offer that”– A challenge at times
• We interpret good learning content23
Flexibility
• Single source masters with an ‘engineered’ delivery provides flexibility– Changes made in one place– Style(s) added during publishing– New platforms added non-disruptively– Multiple instances easily supported– Freed from specific technologies
• Smart, agile content– Standards in action
24
Theme 3: Delivery to Assessment
• Assessment can take many forms– Formative, e.g. through topic quizzes– Summative, e.g. exams and assignments– Ipsative, e.g. through comparison of self with
earlier marks• Students build up a Reflective Log, over time
– A longitudinal measure (Digital Work Book)– Continually improve it– Shared with the tutor (peers potentially)
25
Assessments & the SQA
• Why are we trying to ‘change’ things?• Units are stand alone
– Can be delivered in a very flexible manner– But very clear pointers as to how assessment
should be carried out• No longer appropriate to base an assessment
of competence on a final three hour written examination.
26
HND Business: Unit by Unit
• HND Business: each student has to– Sit 6 formal examinations– Deliver 6 documented portfolio exercises– Complete 22 other assessment reports
• total word count of 34,500
• An HE Diploma?– 12 modules each with final examinations or
assignment coursework
27
Need: Learning Analytics
• MOOCs are gathering global analytics– Measure everything but the individual
• Opus is looking to move towards developing the ‘Digital Self’– How you model yourself as an individual, and
access your data– How you are measured at all times– The DWB is our first step in recognising the
potential of ‘always-on’ learning
28
A Response: a Digital Work Book
• Longitudinal assessment– Augments examination and assignment
• Part of our ‘Social Learner’ aim– No faith in social learning (students creating the
learning content)– Great belief in breeding social learners– Many levels from ‘none’ to
• Interact with content for own ends (e.g. DWB)• Comment locally with others (e.g. Scotsman)• Interact and discuss with all peers (e.g. Forums)
29
Interacting with Content Objects
Learner interacting with content, leaving traces
Student interacts with content
Makes notes, responds to quiz, send messages, etc
Capture notesStore in MLE
Print off record of learning The Digital Workbook
30
Social Networks
• Opus is looking to build 000s of small groups– MOOCs are looking for a group of 000s
• Our social networks are small scale but highly linked (see Anderson 2003)
• Our goal: 5,000 students, but– 100 centres with 50 students– Model the individual, or a small group
31
How do students interact when learning
Student
Environment
Content Tutor
Student/Student
Content/ContentTutor/Tutor
Environment/Environment
Student/Tutor
Tutor/Environment
Student/Environment
Student/Content
Content/Environment
Tutor/Content
Based on diagram inAnderson 2003 32
Big Data?
• The analytics of the individual are of more interest to us– Not specifically looking at group behaviour– Individuals, in depth and across courses
• Digital Self profiles may be limited– E.g. me to Facebook and Linked-In (no pedometer
around my waist!)– How do we enhance profiles?
33
The Role of the Learning Materials
• Authors are encouraged to develop clear ‘concept maps’ for students to follow– Learning paths must go through key ‘concept
gateways’– Evidence of competent/critical thinking gathered
• The core is learning content that– Is interactive– Measures its own consumption– Builds ‘Digital Self’ data
• How to make it more intelligent? 34
Student Learning Aims
• Our aims for our learners– To see how they change over time– To teach these learners how to learn
• Their interface– Our content, systems and support– Must be smart– Back to the interpretation of the semantically rich
XML base
35
Analytics: Some Conclusions
• Analytics and tracking, via the DWB, etc., appears to– Provide the potential of personalised learning
within mechanical presentation of study– Permit formative assessment in the content,
driving tracking and learning progress– Provide feedback on Tutor effectiveness and the
effectiveness of the pedagogy underpinning the content
36
Theme 4: Implementation Details
• Note– No mention of Authoring Tools or proprietary
technologies– No real mention of e-learning– But massive exploitation of the technology of
delivery• Standards imply
– Technology agnosticism– Development outside of deliver environment(s)– Delivery into that/these environment(s)
37
A complete System
• Our final systems integrate– All assignment statements, requests and
submissions– Online tutor support and a Help Desk– Payment clearance and immediate access to study
materials• Our MOOCs include
– 100% automation– Social interaction– Auto generation of a Certificate of Completion
38
Back to Opus Learning
• Opus is a wholly online college– SQA Approved– Offering HNC/Ds and other courses– With its own students and tutors
• Opus is also a B2B provider of– Ready packaged courses and programmes– A full own-brand ‘College in a Box’– Consultancy for aspiring SQA centres
• Opus /CAPDM is a custom publisher & developer of ODL 39
Strategic?
40
Quote Revisited: Mature?
• “Enterprise technology for authoring, publishing and distributing is still largely in an immature state.”– Steve Barden, Lead Consultant, LINE
Communications, Inside Learning Technologies & Skills Magazine - November 2013
41
What people think we do
42
What we think we do
43
The Digital Workbook is
• A portfolio embedded in the content
• A dashboard for learning • Rich seam of data and
behaviours for mining• Ongoing formative
assessment, and potential summative assessment
• A pdf for the student45
Case study HND Business Unit by Unit Assessment
In a typical instance of the HND Business each student has to:– Sit 6 formal examinations– Deliver 6 documented portfolio exercises , and – 22 other assessments - reports with a total word
count of 34,500.
A university Dip HE might only have 12 modules with 12 final examinations or courseworks
46
Managing Learning with Technology
• At Opus we track student behaviour. • We use 2 levels of tracking
– Moodle Based (the VLE) and – Content Based.
47
Tracking through the learning content
• Opus content is massively connected to the students workbook
• Every reflection, exercise, quiz, short response and so on are logged within the learning content and in the workbook
• The tutor can view, track and compare workbooks
48
DWB Examples – Source Page
49
50
51
DWB Dashboard –Managing groups of students
52
The Commercial Imperative
We wish to – drive down cost of assessment– Drive up the quality and efficiency of assessment
within the context of a well resourced learner.
We have to manage costs, and use of human resource
53
Assessor/Tutor Activity Model in Distance Learning
1. Monitor VLE activity, identify any candidate with lack of progress and contact course leader for action.
2. Monitor Digital Workbook (if used) and identify any outliers in progress, contact course leader with details.
3. Respond to direct tickets from candidate, making use of FAQs for responses, if new response is required, copy response to course leader.
4. Responding to assessments; remediation advice
54
Learning Centre Cost Model
• Based on the previous models– Time per tutor/unit/student can be estimated– Payment to Tutor/assessor is known– Viability of the Distance Course can be calculated.
• We need to know these numbers and to watch them
55
Potential Cost Model
Possible costs/charges per studentawarding body charges = £250;tutoring = £1000; tech + online content licence = £300; agent fees = £250Centre overheads per student = £500
Total cost per student = £2300
56
Potential Centre Surplus
• In UK Charge to student £3500-£4500• In India Charge to student £1500-£2500
We cannot serve one of largest markets in the world, with this model.
We have to be able to manage costs.
57
Learning from MOOCS
• Massive size of cohort• Highly programmed week by week• Clearly signposted weekly targets• Using the student body as the main tutor
toolset• Building a community of activists and lurkers• Managing and controlling costs• Generate revenue by conversions
58
Learning from MOOCS
• Do not do assessments – Until students join full programme
• Keep the granularity of the course fairly small – not full degrees
• Set up an honour system – make the students police themselves.
59
Applying all these lessons
• Manage the cohort size• Make sure investment reflects assessment and
tutor support provided – the added value of a non-free course
• Integrate assessments across units; manage the assessment overload
• Make assessments social, encourage group help. Track this activity!
60
Opus StrategySmall Private Open Courses
• Organise cohorts in groups of 10s, not groups of 10k’s or 100k’s
• Localise tutors and assessors• Build communities including assessors• Track activity and behaviour at keyboard level and at
cognitive level• Record activity – perhaps award a badge for partially
completed activity• Guide the learners towards completion, intervene as
necessary61
Creating the Social Learner
• Our DWB is part of a social learner strategy– Rich interaction spaces– Personal interaction with the content (DWB)– Interaction on specific issues (traversing the web)– Social learner (Forums, Facebook, etc.)
• Manage forum and virtual space activity
62
The Future
• The DWB is sharable, including – with the Tutor – with other students, groups
• In the future it will– support many content types– integrate with other portfolio components
• The DWB will be our main assessment toolset• The DWB will be our main behaviour manager
63
Conclusion
• Do not just set exams and give badges– Track and Assess– Intervene and support– Audit and learn lessons
• Build a community for the whole qualification• Record learning and make available for future
generations• Continually improve learning content.
64