iwwua presentacion ppt

aDeNu Research Group http://adenu.ia.uned.es

Usability and Accessibility Evaluationsalong the eLearning Cycle

Ludivine Martin, Emmanuelle Gutiérrez y Restrepo, Carmen Barrera,Alejandro Rodríguez Ascaso, Olga C. Santos, Jesús G. Boticario

Adaptive Dynamic online Educational systems based oN User modellingArtificial Intelligence Department, Computer Science School, UNED, Madrid

2aDeNu Research Group http://adenu.ia.uned.es

1. Introduction

2. E-learning, LMS, Usability

3. Adaptation in the e-learning lifecycle

4. Usability and accessibility in the learning life cycle

• Interactions

• Methodologies

• Results

5. Conclusion and Challenges

PLAN


1. Introduction


1. Introduction

• aLFanetDevelopment of an adaptive LMS (Learning Management System),which combined design and runtime adaptation along the full lifecycle of the learning process with a pervasive use of educationalstandards from IMS (Instructional Management Systems).

• EU4ALLEuropean Unified Approach for Accessible Lifelong Learning.Development of a framework to support the delivery of learningservices along the life cycle of the learning process to cover theneeds of people with disabilities.

• ALPEAccessible eLearning Platform for Europe: a validation projectinvolving an accessible, standard based e-learning solution forvisually impaired, hearing impaired and adult learners.


2. E-learning, LMS, usability


E-learning

• E-learning = learning through electronic media, orcomputer-enhanced learning

• Currently a concept assimilated in educational centres atall stages, especially in higher education institutions tryingto support the life-long learning paradigm.

• It removes time and space barriers --> benefit for peoplewith disabilities although it is not always the case.

• Current research in e-learning strongly focuses onproviding adaptation support to users. Personalizedlearning adapted to individual needs and preferencesimproves the learning process effectiveness.


E-learning

• Disappointment and frustration are common in e-learning,due to– poor features– lack of pedagogical guidance– ineffective evaluation procedures– little support to fulfil administrative tasks.– ….usability issues

• A direct correlation has been established between thekey aspects in the usability field and their effect onlearning.

• In the case of e-learning, the main task for the user islearning (P. Zaharias)


E-learning• Combination of usability and instructional design (L. Lohr):

– Usability (ISO, 1993)• Effectiveness: The user's ability to achieve specific goals in

the environment• Efficiency: The resources used when performing a system-

supported task• Satisfaction:The user's comfort level and acceptance of the

system overall

– Formative evaluation (Instructional Design)• Effectiveness: The attainment of instructional objectives• Efficiency: How quickly and cost-effectively learning

objectives were achieved• Satisfaction: The user's interest in the content and the desire

to continue to learn


E-learning

• Combination of instructional design and usability (L.Lohr):– Definition for the instructional interface design process

• Effectiveness: Learner interprets instructionalinterface function correctly; instructional interfacefunction performs according to the learner'sexpectations

• Efficiency: Learner experiences minimal frustrationinterpreting instructional interface function; learnerexperiences minimal obstacles in usinginstructional interface element

• Satisfaction: Learner seems comfortable in theenvironment overall


E-learning

• Double persona: learner-user (D. Smulders)Using a e-learning site = making functional connectionsLearning = making cognitive connections

• Usability as a precursor of learnability

• Usability shouldn’t be a barrier to learning, especiallybecause it can impact motivation.

• Usability as a basic condition for usable e-learningsystems


E-learning

• Definition of usability in e-learning includes morecomponents than interface design (S. Shilwant)– usefulness : utilizing knowledge gained during the

needs assessments to ensure usefulness– ease of use: employing interface design principles to

achieve ease of use– learnability : using instructional design to attain

learnability– motivation: focusing on motivational principles

(feedback, curiosity, challenge). Success of a trainingexperience largely dependant in student ownmotivation. Importance of affective dimension.


E-learning

• What is usable e-learning? (M. Feldstein):Starting point: learnability.Definition of usability in e-learning = the ability of alearning object to support or enable a very particularconcrete cognitive goal.

--> which specific presentation or interface features havemeasurable impact on specific cognitive tasks or goal ?

example: does the course navigation design teaches thelearner about the course structure? Or does it createvisual static that interferes with the learner’s ability toprocess the main points on a page?


E-learning

• How to measure usability in e-learning?– Expert methods based on heuristics or adapted

heuristics (motivation to learn)– Empirical methods based on satisfaction

questionnaires.– On-going research on methods addressing both

technical and pedagogical issues– No out-of-the box widely accepted usability evaluation

in e-learning exists.– Need for usability specialists to move “out of the

comfort zone” and get familiar with methods frominstructional design and learning theories. (M.Feldstein)


Learning Management Systems (LMS)

• Large variety of LMS, also called VLE, Virtual LearningEnvironments

• Commercial or open source• Different categories :

– general LMS with tools for create and managecourses,

– collaborative learning support systems– questionnaire and test authoring systems– people and institutions resources management

systems– virtual classroomsOne LMS can belong to several categories at the sametime.


Learning Management Systems (LMS)

• Wide variety of integrated features:– school management– course management– class management– content management– communication tools– student tools– assessment tools– etc.


dotLRN

• Before adopting a LMS: identification of the needs andcomparative analysis

• UNED chose ArsDigita Community System (ACS)developed at the MIT. Along time, ACS evolved intodotLRN (officially named .LRN), open source enterprisee-learning platform.

• dotLRN today is used by over a 1/2 million users in morethan 18 countries.

• complete portal framework along with out-of-the-boxcapability for course management, online communities,content management, and learning management.

• Used as a framework in our projects and as acollaborative tool.


LMS particularities

• Out-of-the-box product.Usually an institution will adopt an existing LMS, notbuild a new one. Therefore the LMS already exists. It isnot built for a very specific purpose and audience.

• High level of customization.LMS usually offer a wide range of configuration optionsat the admin level, the tutor level and the student level.This flexibility affects the overall usability of the system

• Variety of needs.LMS have to be flexible enough to address a variety ofteaching, learning and collaborative styles.


LMS particularities when evaluatingusability and accessibility

• Educational standards.To facilitate the reusability of the author work and theadaptability of the course to the learner.Educational standards (IMS, SCORM), usually complex,impact the LMS user interface, therefore the userexperience


LMS particularities when evaluatingusability and accessibility

• Container/contained relationship.– the platform, or “shell” or “receptacle”, including

“modules”, where the course materials are stored anddelivered

– the formal content, or packaged course materials,compliant to educational standards. Here the linebetween interface design and course design beginsto blur

– content generated by users, or content produced bythe entire community


3. Adaptation in the e-learninglifecycle



• Learning ideally should be a personalised and adaptiveprocess which from beginning to end should consider thelearner’s specific needs.

• aDeNu research group (aLFanet project):deliver adapted courses based on– use of educational standards– user-modelling techniques– multi-agent architecture : machine learning techniques

to learn the model attributes and providerecommendations to learners.

• Objective: adapt the user interface to the learners’background, interests, special needs, functional abilitiesand interaction preferences.


4. Usability and accessibility in thee-learning life cycle


Interactions


Interactions

(a) Authoring / Design phase / LMS developmentphase

• integrates the A&U requirements defined viarequirements gathering and elicitation methods.

• integrates the evaluation results from the auditing phasethat closes the cycle. Results become requirements ornew guidelines for redesign.

• course design:author needs to take into account A&U guidelines. Beforedelivering the course, a strict evaluation on accessibility,usability and standards compliance is performed

• LMS development:A&U evaluations (manual and validators) are performedat this phase, documented in a Bug Tracker.


Interactions

• (b) Administration or publication phaseU&A evaluations to detect persistent failures or desirableimprovements that come up when the course design isimported in the LMS. U&A inspection methods (expertreviews): heuristic evaluation and cognitive walk through(task-based scenarios).

• (c) Use phaseReal life with real users (students and tutors) in realsituations of use.Empirical testing or user-based methods: directobservation of a sample of users (thinking aloud andcontextual inquiry), with task based scenarios.We also engage users to report failures or suggestimprovements. (contact forms)Tracking of the users interactions (web logs)


Interactions

• (d) Auditing or feedback phaseResults from experts, users evaluations and automaticprocessing of interactions are analyzed and translatedinto technical requirements or guidelines forimprovements to be integrated in the next release.


Accessibility Methodologies

• LMS accessibility evaluation based on– Web Content Accessibility Guidelines (WCAG)– but also the other accessibility guidelines in order to

cover all LMS processes, components, including anyintegrated interaction elements or applications (chatfor example):• ATAG: Authoring Tool Accessibility Guidelines 1.0• UAAG: User Agent Accessibility Guidelines 1.0• GDALA: Guidelines for Developing Accessible

Learning Applications 1.0• Comprehensive accessibility evaluation = technical

assessment + evaluation with end users (including userswith disabilities).


Accessibility Methodologies

Technical Assessment• Evaluation all templates, content, and templates and

content combined.• In-house accessibility expert extensive manual review• Audit from external experts (Tenuta)• 2 Web accessibility evaluation tools: HERA , AccMonitor

End-users assessment• Continuous conformance assessment with specific users

with different disabilities and using different assistivetechnologies

• Upcoming large testing as part of the ALPE project: 300users over 3 countries.


Usability methodologies

Expert Assessment• In house expert: Heuristic evaluation• External experts from Tenuta project: task scenarios,

evaluation based HHS guidelines.


Usability methodologies

Evaluation with users• Short sessions with involved users. Task based scenarios. Direct

observation. Iterative design.• ALPE project

market validation project --> focus on the user satisfaction /acceptance of the product.1. Large-scale survey with 300 users over 3 countries (Spain, UK,

Greece). Online trial including task based scenarios and aquestionnaire (background and feedback questionnaires).

2. Small-scale survey: Real course over 3-month period in realconditions with a real tutors. Online questionnaires.

3. Laboratory sessionDirect observation. Task based scenarios. Thinking Aloud.


Results

Accessibility Results

• The initial expert accessibility assessment detected 22problems: 27 % of Priority 1, 64% of Priority 2 and 9% ofPriority 3. 75% of those accessibility problems have beenfixed. The remaining issues are under study for the nextrelease.

• The accessibility evaluation by Tenuta experts was alsopositive:

“The website has been well coded, offers a number ofaccessibility features and no major accessibility issueswere identified.”


Results

Usability Results

• Heuristics study showed that the platform complied particularly wellon the following categories:– Aesthetic and Minimalist Design– Pleasurable and Respectful Interaction with the User– Visibility of System Status.Improvements must be made in the following categories:– Match Between System and the Real World– Help and Documentation.

• Positive conclusions from the Tenuta usability audit :"This website has been thoughtfully produced. There hasobviously been a thorough consideration of the user throughout.There are several minor usability issues that could be addressedto improve its usability.“


5. Conclusions and challenges


Conclusions and Challenges

• From the point of view of usability, LMS have specificitiesthat must be taken into account in a user-centredframework.

• LMS are complex systems and A&U need to beconsidered for the entire e-learning life-cycle.

• Evaluations along the cycle in a continuous process.• ALPE and EU4ALL projects as a framework to dotLRN

development



• Accessibility of the content generated by usersor how to support accessible collaborative learning.

Need for accessible authoring tools + automaticassessment of created content.

The LMS can provide certain support, such as WYSIWYGeditor that forces the learners to include alt and longdesctext for every image linked, but there is no guarantee thatthe text provided fully describes the image.Often, WYSIWYG editors are not accessible themselves.Therefore, we need to increase the accessibilityawareness level of all users, by creating some guides forexample.



• Usability of the content generated by user --> editorialquality (instructive design)

• Ensure usability of a highly flexible systemlot of customization options at every levelwhich moreover integrates– components (the platform, the course, the content

generated by users)– generated from different sources (developers,

administrators, authors, students, tutors)– for different purposes.



• Future work– ALPE: 300 users trials– Integration of wiki and blogs in dotLRN.

iwwua presentacion ppt

Documents