Agenda

1. Purposes for digital learning resources (DLR) evaluation systems

2. Development and use of a classification system for a math-related subgroup of DLR

3. Online-evaluations and discussions (Workshop)

4. Support form for evaluation, provided by Norwegian Centre for ICT in Education

5. Criteria for the subgroup of serious games (Workshop)

3

DLR = Digital Learning Resources

Learning Goals

After the presentation and workshops, you should be able to

• be aware of the variety of (sets of) criteria for quality classification of DLRs

• compare and contrast specific types of mathematics related DLRs by using sets of evaluation criteria

• apply a set of classification statements for serious games for quality evaluation

4

Usability/Quality Evaluation Criteriafor Digital Learning Resources

Development/Purchase

ImplementationAnalysis

Ideas

DLR

Usability/Quality Evaluation Criteriafor Digital Learning Resources

What is usability?

• the ease of use and learnability of a human-made object

Good usability of a DLR means …

Usability/Quality Criteriafor Digital Learning Resources

Use

r experi

ence

fun

emotional fulfilling

rewarding

supportive of creativity

aesthetically pleasing

motivating

helpful

entertaining

enjoyable

satisfying

Pedagogic

al usa

bili

ty

learner control

learner activity

cooperative learning

goal orientation

applicability

added value

motivation

valuation of previous knowledge

flexibility

feedback

1 2 3 4 5 6 7 8 9 10

Interface heuristics and technical usabilityNielsen (1993) Nielsen (1994) Reeves and

Harmon (1994)Nokelainen (2004, 2005)

Usability Heuristics Usability Heuristics Interface dimensions

Technical usability of digital learning materials

1. Simple and natural dialogue

2. Speak the user’s language

3. Minimize the user’s memory load

4. Consistency5. Feedback6. Clearly marked

exits7. Shortcuts8. Good error

messages9. Prevent errors10. Help and

documentation

1. Aesthetic and minimalist design

2. Match between system and the real world

3. Recognition rather than recall

4. Consistency and standards

5. Visibility of system status6. User control and freedom7. Flexibility and efficiency

of use8. Help users recognize,

diagnose, and recover from errors

9. Error prevention10. Help and Documentation

1. Ease of use 2. Navigation 3. Cognitive

load4. Mapping 5. Screen

Design6. Knowledge

Space Compatibility

7. Information Presentation

8. Media Integration

9. Aesthetics 10. Overall

functionality

1. Accessibility2. Learnability and

memorability3. User control 4. Help 5. Graphical layout 6. Reliability 7. Consistency 8. Efficiency of use9. Memory load 10. Errors

 

Reference for tables presented: Brodahl & Smestad (2009)

Pedagogical usability criteria, part 1

Reeves and Harmon (1994), Reeves (1994)

Quinn (1996) Albion (1999) Squires and Preece (1999)

Pedagogical dimensions Educational design heuristics

Content Heuristics Learning with software heuristics

1. Epistemology 2. Pedagogical

Philosophy3. Underlying

Psychology 4. Goal orientation 5. Instructional

Sequencing (*) 6. Experiential Value

(Authenticity)7. Role of Instructor 8. Value of Errors9. [Origin of] Motivation 10. Structure (*) 11. Accommodation of

Individual Differences (Scaffolding)

12. Learner Control 13. User activity14. Cooperative Learning (*) Reeves (2004): Flexibility; Cultural Sensitivity

1. Clear goals and objectives

2. Context meaningful to domain and learner

3. Content clearly and multiply represented, and multiply navigable

4. Activities scaffolded5. Elicit learner

understandings6. Formative

evaluation7. Performance

should be 'criteria-referenced'

8. Support for transference and acquiring 'self-learning' skills

1. Establishment of context

2. Relevance to professional practice

3. Representation of professional responses to issues

4. Relevance of reference materials

5. Presentation of video resources

6. Assistance is supportive rather than prescriptive

7. Materials are engaging

8. Presentation of resources

9. Overall effectiveness of materials

1.Match between designer and learner models

2.Navigational fidelity3.Appropriate levels of

learner control4.Prevention of

peripheral cognitive errors

5.Understandable and meaningful symbolic representation

6.Support personally significant approaches to learning

7.Strategies for the cognitive error recognition, diagnosis and recovery cycle

8.Match with the curriculum

Pedagogical usability criteria, part 2

Horila, Nokelainen, Syvänen and Överlund (2002)

Vargo, Nesbit, Belfer and Archambault (2003)

Nesbit, Belfer and Leacock (2003)

Nokelainen (2004, 2005, 2006)

Pedagogical usability of digital learning environments

Pedagogical heuristics of Learning Objects

Pedagogical heuristics of Learning Objects

Pedagogical usability of digital learning material

1. Learnability2. Graphics and layout3. Technical requirements4. Intuitive efficiency5. Suitability for different

learners and different situations

6. Ease of use: Technical and pedagogical approach

7. Interactivity8. Objectiveness9. Sociality10. Motivation11. Added value for

teaching

1. Presentation Aesthetics

2. Presentation Design for Learning

3. Accuracy of Content4. Support for Learning

Goals5. Motivation 6. Interaction: Usability7. Interaction:

Feedback and Adaption

8. Reusability9. Standards

Compliance10. Accessibility

1. Content Quality2. Learning Goal

Alignment3. Feedback and

Adaptation4. Motivation5. Presentation

Design6. Interaction

Usability7. Accessibility8. Reusability9. Standards

Compliance

1. Learner control2. Learner Activity3. Cooperative/

collaborative learning

4. Goal orientation5. Applicability6. Added value for

learning7. Motivation8. Valuation of

previous knowledge

9. Flexibility10. Feedback

Types and levels for interactivity and interaction

Schulmeister (2003) El Saddik (2001) Sedig and Liang (2006) Sedig and Sumner (2006)

Level of interactivity offered to the user

Degree of interaction with a visualization system

Interactivity framework for analyzing mathematical presentations

Interaction framework for characterizing mathematical presentations

1. Viewing objects and receiving

2. Watching and receiving multiple representations

3. Varying the form of representation.

4. Manipulating the component content

5. Constructing the object or representation contents.

6. Constructing the Object or Contents of the Representation and Receiving Intelligent Feedback from the System through Manipulative Action.

1. Still images2. Animated Pictures3. Visualization with

display adjustments

4. Visualization selection and arrangement capabilities (VCR)

5. Visualization with changing input, zooming and panning

6. Visualization with interactive decision points, e.g. changing data while running

7. Visualization generated by students

1. Affordance 2. Cognitive offloading 3. Constraints 4. Distance 5. Epistemic

appropriateness 6. Feedback 7. Flexibility 8. Flow 9. Focus 10. Involvement 11. Scaffolding 12. Time-Space

communication

Basic-interaction1. Conversing2. Manipulating3. NavigatingTask-based interaction4. Animating5. Annotation6. Chunking7. Composing8. Cutting9. Filtering10. Fragmenting11. Probing12. Rearranging13. Picturing14. Scoping15. Searching

Other aspects in the literatureDraft Standard for LOM 1.3 (IEEE_LTSC, 2005b)

Draft Standard for LOM 1.3 (IEEE_LTSC, 2005b)

Heinich, Molenda, Russell and Smaldino (2002)

Gadanidis and Schindler (2003)

Types of interactivity

 

Learning resource types

 

Types of instructional methods in pedagogical design

Thinking level potential of mathematical learning objects

1. Active (learning by doing)

2. Expositive (passive learning)

3. Mixed

1. Exercise2. Simulation3. Questionnaire4. Diagram5. Figure6. Graph7. Index8. Slide9. Table10. Narrative text11. Exam12. Experiment13. Problem

statement14. Self assessment15. Lecture

1. Presentation2. Demonstration3. Discussion4. Drill-and-practice5. Tutorial6. Cooperative

learning7. Gaming8. Simulation9. Discovery10. Problem solving

1. Focus on recalling mathematical facts and definitions

2. Focus on applying mathematical procedures

3. Focus on understanding mathematical relationships

4. Focus on mathematical extensions and generalizations

Purposes/Goals for learning object evaluation systems

1. Aid for searching and selecting

2. Guidance for use

3. Formative evaluation

4. Influence on design practices

5. Professional development and student learning

6. Community building

7. Social recognition

8. Evaluation for economic exchange

9. Fostering communication

Brodahl & Smestad (2009)

Nesbit, Belfer & Vargo (2002)

DEVELOPING AND USINGUSABILITY CRITERIA FOR DIGITAL LEARNING RESOURCES (DLR)

Case: Mathematical Visual Dynamic/Interactive Applications

Development phase 1:

• Find terms describing the particular group of DLRs

• Collect examples of all kinds within this group

• Build a taxonomy with descriptive terms of statements to classify these DLRs

Development phase 2:

• Explore if the use of a taxonomy will provide better selection of DLRs for L&T activities give higher quality of DLRs to be developed

Development of a classification system

Case: Mathematical Visual Dynamic/Interactive Applications

Steps of Analysis

DLRno. k

Classificationdocument

version no. n

Jointanalysis

Need

for terms

Jointdiscussion

Classificationdocument

version n+1

DLRno. k+1

Jointanalysis

Needfor terms

Jointdiscussion

Classificationdocument

version n+2

Jointdiscussion

Development of a classification system

Case: Mathematical Visual Dynamic/Interactive Applications

Classificationdocument

version no. n

Need for terms

DLRno. k

Jointdiscussion

Jointanalysis

Development of a classification system

Case: Mathematical Visual Dynamic/Interactive Applications

Steps of Analysis

DLRno. k

Classificationdocument

version no. n

Jointanalysis

Need

for terms

Jointdiscussion

Classificationdocument

version n+1

DLRno. k+1

Jointanalysis

Needfor terms

Jointdiscussion

Classificationdocument

version n+2

Jointdiscussion

Development of a classification system

Case: Mathematical Visual Dynamic/Interactive Applications

One example – Building houses with side views

Discussion on the feedback

• The feedback can differ in many, pedagogically important, ways: in timing, level of detail, frequency…

• The classification document should take this into account

DLR no. k Joint analysis

DLRno. k

Classificationdocument

version no. n

Jointanalysis

Need for terms

Jointdiscussion

Discussions triggered

• the way the applet gives feedback

• objective or subjective criteria

• the possibility to navigate back to earlier tasks

• the possibility to input new tasks

• “landscapes of investigation” and “exercise

discourses” (or: how to include theoretical

terms)

Steps of Analysis

Steps of Analysis

DLRno. k

Classificationdocument

version no. n

Jointanalysis

Need

for terms

Jointdiscussion

Classificationdocument

version n+1

DLRno. k+1

Jointanalysis

Needfor terms

Jointdiscussion

Classificationdocument

version n+2

Jointdiscussion

Steps of Analysis

Jointanalysis

DLRno. k

Classificationdocument

version no. n

Jointanalysis

Need

for terms

Jointdiscussion

Classificationdocument

version n+1

DLRno. k+1

Jointanalysis

Needfor terms

Jointdiscussion

Classificationdocument

version n+2

Jointdiscussion

DLRno. k+2

Result

A faceted taxonomy

with 11 main areas

220 statements

1. Concepts from mathematics education and pedagogy

2. Target group and learning goals 3. Type of teaching and learning

activity 4. The user's influence and control 5. Type of interactivity 6. Content components 7. Dynamic way of presentation 8. Time aspects 9. Appearance, language and

accessibility 10. Technical concepts 11. Overall evaluation of the quality

From research and development

1. Developing a taxonomy/classification system- a comparative framework for analysis

• for teachers to evaluate DLRs they consider to use • for educators and developers to communicate when developing DLRs

2. Testing the system as an instrument in practice to facilitate/strengthen• academic exchange in e-learning oriented teacher-training programs• communication in outsourcing development of DLRs• evaluation of DLRs in higher and post-qualifying education,

and by groups of pupils in primary/secondary school

3. Developing the system further (“permanent beta”)

4. Using the system for practical purposes• analysis of and choice between DLRs for T/L activities• description and self-evaluation of students work• initiation of students' discussions in a university setting

Use classification criteria for teaching and learning

• Preparations

Make careful selections of DLRs

For each DLR, choose statements from the taxonomy

Include them in an electronic questionnaire

• Voting and discussion

Show the result on the screen

Discuss the results

• Discuss the analytic dimensions of the evaluation criteria

… that is what I want us to do now in the

workshop!

WORKSHOP – Evaluation of Mathematical Visual Dynamic/Interactive DLRs

Voting and discussion

No. DLR Form Number of questions

1 Kids and cockies, fractions 1 # 15

2 Decanting puzzle, jugs 2 # 20

3 Number patterns 3 # 20

4 Building with blocks 4a, 4b # 15, # 11

5 Rabbits and wolves 5 # 16

6 Linear equations 6 # 11

7 Tangram (English version) 7 # 23

Evaluation at http://home.uia.no/cornelib/dlr/

Form 1

Learning outcome

DLR 1

Can you, for instance, divide 11 chocolate bars among five people?

Kids and cookies

Evaluation at http://home.uia.no/cornelib/dlr/

DLR 2Form 2

InteractivityLearning outcomeCan you measure exactly 4 litres using the jugs?

Decanting puzzle

Evaluation at http://home.uia.no/cornelib/dlr/

Interactivity

Control activities

• stop/start, forwards and back• navigation• display pace• sound on/off, volume• Choose layout: color, text size

• e.g. by control buttons

31

Didactical interaction

offer context-related dialogue on the user’s demand

e.g. related to be able to

• make choices and influence the presentation of the content(level of difficulty, type of activity, etc.)

• ask for a hint

• move elements to see what will happen

• to choose how data are proceeded (e.g. choose a mode)

Mathematics learning outcome

• facts • skills• concepts• strategies• attitudes

32

DLRs can support learning, if they provide • relevant challenges• meaningful choices• good responses

Form 3

Interactivity

Learning outcome

DLR 3

Find the missing number. Find the rules.

Number patterns

Evaluation at http://home.uia.no/cornelib/dlr/

DLR 4

Can you use the side views to construct a building in a 3d environment - with as few cubes as possible?

Form 4aForm 4b

Mathematics educational questions

Building with blocks

Evaluation at http://home.uia.no/cornelib/dlr/ Demo

Terms from the didactics of mathematics

• Exercise discourses vs. landscapes of investigations

• Misconceptions and providing cognitive conflicts

35

DLR 5

Build a simple ecosystem with grass, rabbits, and wolves.

Form 5

Mathematics educational questions

Rabbits and wolves

This DLR is about probabilities, chaos

and populations.

Evaluation at http://home.uia.no/cornelib/dlr/

DLR 6

Use the balance strategy to solve the equations.

Form 6

Position in learning process

Linear equations

Evaluation at http://home.uia.no/cornelib/dlr/

Terms from the didactics of mathematics

Position in the learning process

• Lecture

• Instruction

• Step-by-step guidance

• Demonstration

• Example

• Illustration

• Game

• Extensive practice driving

38

DLR 7

Build classic Tangram figures suggestedor more freely by using all of the shapes

Form 7

Overall evaluation of the quality

Tangram

Evaluation at http://home.uia.no/cornelib/dlr/

The classification criteria are supposed to be

an aid for teachers and others who want to evaluate the suitability of digital learning resources in an educational context

The classification criteria are supposed to be

a living taxonomy with educational and technical aspects

Guidelines/forms upon the classification criteria are tools/aids that may be useful

samples in the selection/evaluation process and can be utilized, adapted, and/or modified to meet specific needs.

QUALITY CRITERIAFOR DIGITAL LEARNING RESOURCES (DLR)

The quality criteria are intended as a guide in the development and evaluation of digital learning resourceswww.iktsenteret.no

The quality criteria is a living document and will be updated and expanded as needed and changes in the underlying laws, regulations and guidelineswww.iktsenteret.no

Metadatatagging

Distinctiveness of DLR

Subject + education dimension

User orientation

Technicalinteroperability

Paralleleditions

Accessibility

Reference: The Norwegian Centre for ICT in Education. (2010). Quality Criteria for Digital Resources.

Download the newest document here: https://iktsenteret.no/ressurser/kvalitetskriterier-digitale-laeringsressurser

DLR

Technical aspects

Educational aspects

Quality Criteria for DLR

Quality criteriaregardless of whether or not

the LR is digital

Key question…Description In-depth questions Evaluator’s comments… … …

Educational evaluation

Key question…Description In-depth questions Evaluator’s comments… … …

……

Quality requirements…

Metadatatagging

Distinctiveness of DLR

Subject + education dimension

User orientation

Technicalinteroperability

Paralleleditions

Accessibility

Reference: The Norwegian Centre for ICT in Education. (2010). Quality Criteria for Digital Resources.

Download the newest document here: https://iktsenteret.no/ressurser/kvalitetskriterier-digitale-laeringsressurser

DLR

Technical aspects

Educational aspects

Quality Criteria for DLR

Quality criteriaregardless of whether or not

the LR is digital

: the interface between user and resource

User orientation

Quality requirements Yes Partly No

The learning resource enables individual differentiation

The learning resource uses the digital medium with the pupil’s learning in mind

The learning resource addresses the target audience in an engaging manner

The learning resource has a universal design

The learning resource is self-explanatory

: the interface between user and resource

User orientation

Key question 1

Does the DLR create interest?Description In-depth questions

The resource

• should help activate and motivate

• must be relevant to the subject being taught

a) What type of functionality helps create interest, and how does the design and layout contribute to this?

b) How are images, graphics, video, sound, and so forth exploited to create interest?

c) What choices have been made to ensure that the resource will promote learning and not merely activity and entertainment?

: the interface between user and resource

User orientation

Key question 2

Is the DLR inclusive and accessible?Description In-depth questions

The resource should

• be easy to use and as self explanatory as ‐possible

• follow familiar patterns for navigation and retrieval

• not exclude users on the basis of ethnic or social background or gender

a) How efficiently can the pupil start working on the subject matter (not waste time on navigation)?

b) How does the resource enable as many as possible pupils to use it?

c) To what extent can the resource be adapted to the individual pupil, and how is adequate differentiation facilitated?

d) How can the pupils’ media skills be utilized and developed?

Metadatatagging

Distinctiveness of DLR

Subject + education dimension

User orientation

Technicalinteroperability

Paralleleditions

Accessibility

Reference: The Norwegian Centre for ICT in Education. (2010). Quality Criteria for Digital Resources.

Download the newest document here: https://iktsenteret.no/ressurser/kvalitetskriterier-digitale-laeringsressurser

DLR

Technical aspects

Educational aspects

Quality Criteria for DLR

Quality criteriaregardless of whether or not

the LR is digital

: the possibilities and limitations of the DLR

Distinctiveness of DLR

Quality requirements Yes Partly No

The learning resource enables interaction with the pupil

The learning resource uses communication to reinforce the learning work

The learning resource allows its contents to be updated

The learning resource is inspirational

The learning resource utilizes the specific capabilities of various media in the learning work

: the possibilities and limitations of the DLR

Distinctiveness of DLR

Key question 1

How does the DLR utilize the inherent possibilities of digital media?

Description In-depth questions

The resource should• include various media

forms such as text, images, video, animation, simulations, etc.

• select and incorporate the media forms on an educational basis

a) In what ways can the user use a variety of media forms in the resource?

b) How does interactivity (if such exists) help capture the pupils’ interest in the subject?

c) How is it ensured that the presented information is up to date?

: the possibilities and limitations of the DLR

Distinctiveness of DLR

Key question 2

Is the DLR adaptable?Description In-depth questions

The resource should

• be able to be adapted to different contexts

• enable the use of individual components in different contexts (modular use)

a) How does the DLR enable individual components to be used independently?

b) To what degree does the resource allow the pupil to adapt the contents to his or her educational context?

: the possibilities and limitations of the DLR

Distinctiveness of DLR

Key question 3

How does the DLR enable new educational possibilities that are lacking in traditional learning resources?Description In-depth questions

The resource should

• facilitate a varied teaching and learning practice through the use of various

media forms the use of

communication resources

solid access to updated Information

a) Does the resource enable communication between pupils, between teacher and pupil, and between others?

b) How does the resource challenge the pupils in regard to the subject matter?

c) To what degree are links provided to alternative sources and updated information?

d) How are the specific capabilities of various media forms used in the learning work?

e) To what degree is the resource innovative, and how can it help teach the subject matter?

Metadatatagging

Distinctiveness of DLR

Subject + education dimension

User orientation

Technicalinteroperability

Paralleleditions

Accessibility

Reference: The Norwegian Centre for ICT in Education. (2010). Quality Criteria for Digital Resources.

Download the newest document here: https://iktsenteret.no/ressurser/kvalitetskriterier-digitale-laeringsressurser

DLR

Technical aspects

Educational aspects

Quality Criteria for DLR

Quality criteriaregardless of whether or not

the LR is digital

: the educational potential

Subject and education dimension

Quality requirements Yes Partly No

The educational resource is relevant for current curricula

The educational resource supports collaborative learning

The educational resource is well-suited to the target group

The educational resource has built-in evaluation capabilities

The educational resource can be supplemented with user content and contains a teacher guide

: the educational potential

Subject and education dimension

Key question 1

How is the DLR relevant for the curriculum?Description In-depth questions

The resource should

• be able to be adapted to different contexts

• enable the use of individual components in different contexts (modular use)

a) How is the learning resource relevant for the curriculum and the competence objectives?

b) Is the learning resource suitable for achieving the goals that have been defined?

c) Is the resource intended for a certain age or target group? How is it designed so as to reach any such target groups?

: the educational potential

Subject and education dimension

Key question 2

Does the DLR enable an evaluation that is adapted to the education setting?Description In-depth questions

The resource should

• have built-in possibilities for evaluation, for example tests

• be able to be used for or be incorporated in formative and/or summative evaluation

a) To what degree does the resource support different forms of evaluation?

b) To what extent does the resource enable the pupils to evaluate their own work themselves?

c) How can the resource help the pupils to reflect following the evaluation?

d) Does the resource allow the pupils to give feedback to and evaluate one another?

e) How does the resource support the pupils in their further work following the evaluation (in the event of such an evaluation)?

: the educational potential

Subject and education dimension

Key question 3

In which education settings is the DLR suitable? Description In-depth questions

The resource should be suited for

• individual work

• teacher-led activity

• group work, etc.

The resource should

• contain a teacher guide

a) How does the DLR facilitate use in a variety of learning contexts?

b) How does the DLR support collaborative learning?

c) To what degree can pupils and teachers add their own contents to the resource?

d) To what degree does the teacher guide (if such exists) provide useful ideas?

Support form for evaluation

For support form for evaluation of the educational suitability and technical and formal requirements of digital learning resources will be made available for print as a separate PPT attachment.

Group activity

1. Find descriptive terms/ statements to describe your serious game. Make a list.

2. Sort your list and publish as a blog entry (check category ALICT Classification)

3. Present your list.

Thank you for your attention! Thank you for your attention!

cornelia.brodahl@uia.no

• Brodahl, C. & Smestad, B. (2009). A Taxonomy as a Vehicle for Learning. Interdisciplinary Journal of E-Learning and Learning Objects, 5, 111-127.

• Brodahl, C. & Smestad, B. (2007). The MathWiz Project. Design, Technologies and Future Perspectives. Presentations at Beijing Normal University and Beijing Institute of Education.

• The Norwegian Centre for ICT in Education. (2012). Quality Criteria for Digital Resources.

• The Norwegian Centre for ICT in Education. (2012). Støtteskjema for vurdering av pedagogisk egnethet og tekniske og formelle krav ved digitale læringsressurser. Download the newest version: https://iktsenteret.no/ressurser/kvalitetskriterier-digitale-laeringsressurser

• Skovsmose, 0. (2001). Landscapes of investigation. Zentralblatt für Didaktik der Mathematik, 3(4), 123-132.

Referances and further readings (#1/3)

Referances and further readings (#1/3)

• Nesbit, J. C., Belfer, K., & Vargo, J. (2002). A Convergent Participation Model for Evaluation of Learning Objects. Canadian Journal of Learning and Technology, (3), 105-120. Retrieved October 15, 2008, from http://www.cjlt.ca/index.php/cjlt/article/viewArticle/110/103

• Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In Wild, M. & Kirkpatrick, D. (Eds.) Computer education: New Perspectives, Perth, Australia: MASTEC, 219-246.

• Reeves. T. C. & Harmon, S.W. (1994): Systematic Evaluation Procedures for Interactive Multimedia for Education and Training, i Sorel Reisman: Multimedia Computing, Idea Group Publishing.

• Quinn, C. (1996). Pragmatic Evaluation: Lessons from Usability, http://www.ascilite.org.au/conferences/adelaide96/papers/18.html

• Squires D., & Preece, J. (1996). Usability and Learning: Evaluating the Potential of Educational Software. Computers & Education, 27 (1), 15-22.

• Albion, P. (1999). Heuristic evaluation of educational multimedia: From theory to practice. Paper presented at the 16 th Annual conference of the Australasian Society for Computers in Learning in Tertiary Education, http://www.usq.edu.au/users/albion/papers/ascilite99.html.

• Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11, 467-483.

• Horila, M., Nokelainen, P., Syvänen, A., & Överlund, J. (2002). Criteria for the pedagogical usability, version 1, Hämeenlinna, Finland: Häme Polytechnic and University of Tampere.

• Nielsen, J. (2000). Designing Web Usability: The Practice of Simplicity. New Riders.

• Shneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human-Computer Interaction (3rd Ed.), Menlo Park, CA, USA. Addison Wesley.

• Jensen, J.F. (1998). Interactivity. Tracking a New Concept in Media and Communication Studies. Nordicom Review, 19(1).

Referances and further readings (#3/3)