d5.2.3 - deployment of final inspiring science education...

Inspiring Science Education

Project Acronym: ISE

Grant Agreement number: 325123

Project Title: Inspiring Science: “Large Scale Experimentation Scenarios to Mainstream eLearning in Science, Mathematics and Technology in Primary and Secondary Schools”

D5.2.3 - Deployment of Final Inspiring

Science Education System Revision: 1.0

Editor(s): UPRC Panagiotis Zervas

Lead Author(s): UPRC Panagiotis Zervas INTRA Lamprini Kolovou VELTI Andreas Drakos FRAUNHOFER Constantin Brosda COVUNI Petros Lameras Learnit3D Neville Ruud SIVECO Ana-Maria Baldea, Diana Niculescu

Project co-funded by the European Commission within the Competitiveness and Innovation framework programme

Dissemination Level

P Public X

C Confidential, only for members of the consortium and the Commission Services


D5.2.3 - Deployment of Final Inspiring Science Education System of 74

17/05/2016 v.1.0

Task/WP related to the Deliverable:

Task 5.1, Task 5.2, Task 5.3, Task 5.4, Task 5.5 / WP5

Type 1: P

Reviewer(s): Chris North (Cardiff), Yvonne Crotty (DCU)

Approved by: EA

Contractual Date of Delivery to the EC:

30/11/2015 (M32) Actual Date of Delivery to the EC: 17/05/2016 (M38)

Abstract:

This deliverable reports on the work done in WP5 with respect to the development and deployment of the final version of the Inspiring Science Education (ISE) System following the feedback received from the usage of the ISE System during Phase B of the ISE Project‟s pilots. Phase B of ISE project pilots started on November 2014 and it ended on August 2015. To this end, this deliverable deals with (a) the additional technical development of different components of the ISE System based on the feedback received from ISE Pilot users, namely teachers and students, (b) a study that was conducted to received feedback about the usability of the different components of the ISE System and (c) summative data (until the end of M36) about the usage of the different components of the ISE System, so as to provide evidence about the large scale capacity of the ISE System.

Keywords: Inspiring Science Education System, Authoring Tool, Delivery Tool, eLearning Tools, ISE Portal, ISE Tools Repository, Mobile Delivery, Assistive Tools

Project Co-ordinator

Company Name: INTRASOFT International S.A. Name of Representative: Kostas Thiveos Address: Rue Nicolas Bove 2b Phone Number: + 30 310 6876482 Fax Number: + 30 210 6876478 E-mail: [email protected] Website: http://www.inspiringscience.eu/

1Deliverable Type: P (Prototype), R (Report), O (Other)

mailto:[email protected]

http://www.inspiringscience.eu/



17/05/2016 v.1.0

Versioning and Contribution History

Revision Date Author Organization Description

0.1 November 5, 2015 Panagiotis Zervas UPRC First draft of Table of

Contents and Section 3.1

0.2 November 23, 2015 Andreas Drakos VELTI Section 3.2

0.3 November 25, 2015 Lamprini Kolovou INTRA

Section 2.1 and consolidation of Chapter 2

Diana Niculescu SIVECO Section 2.2

0.4 November 26, 2015

Constantin Brosda FRAUNHOFER Section 4.1 and

consolidation of Chapter 4

Neville Ruud Learnit3D Section 4.2

Petros Lameras COVUNI Section 4.3

0.5 February 10, 2016 Andreas Drakos VELTI Section 3.2.3

0.6 March 7, 2016 Ana-Maria Baldea SIVECO Section 2.2.3

0.7 March 31, 2016 Panagiotis Zervas UPRC Section 3.1.3

0.8 April 7, 2016 Panagiotis Zervas UPRC

Chapter 1, consolidation of Chapter 3 and revisions to Section 2.2.3 and section 3.2.2, Section 3.1.4 and Section 3.2.4, Chapter 5, Executive Summary and

Abstract.

0.9 April 20, 2016

Panagiotis Zervas UPRC Final editing and

preparation of complete final version

Constantin Brosda FRAUNHOFER Section 4.1.3

Neville Ruud Learnit3D Section 4.2.3

1.0 May 13, 2016 Panagiotis Zervas UPRC

Revisions based on the comments received by the internal reviewers, namely Chris North (Cardiff) and

Yvonne Crotty (DCU)



17/05/2016 v.1.0

Table Contents

1. Executive Summary 10

2. Introduction 12

2.1 Scope 12

2.2 Audience 12

2.3 Structure 12

3. ISE Portal and ISE Tools Repository 13

3.1 ISE Portal 13

3.1.1 Overview of New Features and Changes since the Intermediate Release 13

3.1.2 New Features and Changes 13

3.1.2.1 “How to Use” Guide 13

3.1.2.2 Portal and ISE Authoring Tool Integration 14

3.1.2.3 Training Academies: Connect with Communities and Resources 16

3.1.2.4 Pin Events & Activities between Communities and Training Academies 17

3.1.2.5 Search Educational Resources 17

3.1.2.6 ISE Demonstrators Page 21

3.1.2.7 New Layout of the Home Page 21

3.1.2.8 ISE Portal Infrastructure 22

3.1.3 Usability Study 23

3.1.3.1 The Online Questionnaire 23

3.1.3.2 Experts‟ Usability Test 25

3.1.4 Summative Usage Data 27

3.2 ISE Tools Repository 27



3.2.2.1 Improved Layout of eLearning Tools Metadata Presentation 28

3.2.2.2 Enhanced Searching/Browsing of eLearning Tools 28


3.2.3.1 Sample 29

3.2.3.2 Procedure 31

3.2.3.3 Results 32


4. ISE Authoring and Delivery Environment 33

4.1 ISE Authoring Environment 33



4.1.2.1 Enhanced Integration with ISE Portal 33



17/05/2016 v.1.0

4.1.2.2 Upload and Visualize MS Office Documents 33

4.1.2.3 Authoring Lessons/Educational Scenarios without Problem Solving Questions 34


4.1.3.1 Sample 35


4.1.3.3 Results 38


4.2 ISE Delivery Environment 39



4.2.2.1 Enhanced Class Assessment Dashboard for Teacher 39

4.2.2.2 Multilingualism 41


4.2.3.1 Sample 42


4.2.3.3 Results 46


5. ISE Educational Tools 48

5.1 Augmented Reality Tool: Science Center to Go 48



5.1.2.1 Adapted Web Server Landing Page 48

5.1.2.2 New App Dialog 49

5.1.2.3 Double Slit Experiment 50

5.1.3 Usability and Quality Assurance 51

5.2 Online Labs and Simulations: Learnit3D 52



5.2.2.1 Light Pollution 52

5.2.2.2 Introduction to Telescopes 53

5.2.2.3 Primary Mechanics 54


5.2.3.1 Light Pollution 55

5.2.3.2 Introduction to Telescopes 55

5.2.3.3 Primary Mechanics 56

5.3 Educational Game: SIMAULA 56



5.3.2.1 The Electromagnetic Spectrum 57



17/05/2016 v.1.0

5.3.2.2 The Foucault Pendulum 59

5.3.2.3 Feedback Mechanics 61


5.3.3.1 Questionnaire Design 62

5.3.3.2 Findings 64

5.3.3.3 Conclusions 72

6. Conclusions 73

7. References 74



17/05/2016 v.1.0

Index of Figures

Figure 1: How to Use Guide – Home Page ..................................................................................................... 14

Figure 2: The ISE Portal and ISE Authoring Environment Integration Workflow ............................................ 14

Figure 3: Access of ISE Authoring Environment through the User‟s Profile ................................................... 15

Figure 4: Access of ISE Authoring Environment from Community‟s Resources Page ................................... 15

Figure 5: Default Educational Scenario/Lesson Plan ...................................................................................... 16

Figure 6: Connecting Training Courses of ISE Academy with Communities and Resources ......................... 16

Figure 7: Share Communities‟ Activities and Events like Sharing/Pinning Resources ................................... 17

Figure 8: Architecture of the Search Engine.................................................................................................... 18

Figure 9: Search Engine Components and their Communication ................................................................... 19

Figure 10: The ISE Demonstrators Page ........................................................................................................ 21

Figure 11: ISE Portal New Home Page Layout ............................................................................................... 22

Figure 12: ISE Portal Infrastructure ................................................................................................................. 23

Figure 13: Online Questionnaire of the ISE Portal .......................................................................................... 24

Figure 14: Results from the Online Questionnaire of the ISE Portal ............................................................... 24

Figure 15: Presentation of the eLearning Tools Metadata after Improving the Layout ................................... 28

Figure 16: Search eLearning Tools that Match Teacher‟s ICT Competences ................................................ 28

Figure 17: Gender Distribution ........................................................................................................................ 30

Figure 18: Years of Teaching Experience ....................................................................................................... 30

Figure 19: Level of Education .......................................................................................................................... 30

Figure 20: Previous Knowledge on ICT ........................................................................................................... 31

Figure 21: ICT Usage During Teaching ........................................................................................................... 31

Figure 22: ISE Authoring Environment – Upload MS Office Documents for an Inquiry Activity ...................... 33

Figure 23: MS Office Document Viewer Integrated in the ISE Authoring Environment .................................. 34

Figure 24: ISE Authoring Environment – The Preview of an Example Lesson without Problem Solving Questions ......................................................................................................................................................... 35

Figure 25: Gender Distribution ........................................................................................................................ 36


Figure 27: Level of Education .......................................................................................................................... 37

Figure 28: Previous Knowledge on ICT ........................................................................................................... 37

Figure 29: ICT Usage During Teaching ........................................................................................................... 37

Figure 30: ISE Delivery Environment Teacher View – Class Assessment Feature ........................................ 39

Figure 31: ISE Delivery Environment Teacher View – Subject-Domain Knowledge Questions Statistics ...... 40

Figure 32: ISE Delivery Environment Teacher View – Subject-Domain Knowledge Questions Results ........ 41

Figure 33: ISE Delivery Environment Teacher View – Changing the Language of the Interface ................... 41

Figure 34: Gender Distribution (STEM Teachers) ........................................................................................... 43


Figure 36: Level of Education (STEM Teachers) ............................................................................................ 43

Figure 37: Previous Knowledge on ICT (STEM Teachers) ............................................................................. 44

Figure 38: ICT Usage During Teaching (STEM Teachers) ............................................................................. 44

Figure 39: Gender Distribution (Students) ....................................................................................................... 45

Figure 40: Age Range (Students) .................................................................................................................... 45

Figure 41: Experience on Using Computers (Students) .................................................................................. 45

Figure 42: Science Center To Go Android Version - New Landing Page ...................................................... 49

Figure 43: Science Center To Go Android App's New Dialog ........................................................................ 50

Figure 44: Science Center To Go Android - Double Slit in Particles mode ..................................................... 51



17/05/2016 v.1.0

Figure 45: Light Pollution Experiment - Showing Interaction View .................................................................. 53

Figure 46: Introduction to Telescopes - Launch Page ..................................................................................... 53

Figure 47: Introduction to Telescopes - Different Telescopes Side-by-Side ................................................... 54

Figure 48: Primary Mechanics Screenshots .................................................................................................... 54

Figure 49: SIMAULA – Level Objectives ......................................................................................................... 57

Figure 50: SIMAULA - Electromagnetic Spectrum Nested Dialogues Process .............................................. 58

Figure 51: SIMAULA - Responding to a Question Raised by a Student ......................................................... 58

Figure 52: SIMAULA – The Light Experiment ................................................................................................. 59

Figure 53: SIMAULA - The Foucault Pendulum .............................................................................................. 60

Figure 54: SIMAULA - The Foucault Pendulum Experiment ........................................................................... 61

Figure 55: SIMAULA - Intrinsic Feedback mechanic ....................................................................................... 61

Figure 56: SIMAULA Usability Questionnaire.................................................................................................. 63

Figure 57: Subject of Study ............................................................................................................................. 64

Figure 58: SIMAULA Game Responsiveness ................................................................................................. 65

Figure 59: SIMAULA Engagement Levels ....................................................................................................... 65

Figure 60: SIMAULA Game Dialogue Mechanism .......................................................................................... 66

Figure 61: Selecting a Dialogue using the Game Controls ............................................................................. 66

Figure 62: Involvement in Game Play ............................................................................................................. 67

Figure 63: Adjustment of the Experience of Playing the Game ...................................................................... 67

Figure 64: Level of Proficiency in Interacting with the Game .......................................................................... 68

Figure 65: Feedback Mechanics ..................................................................................................................... 68

Figure 66: Level of Focus in Responding to Questions ................................................................................... 69

Figure 67: Game Objectives ............................................................................................................................ 69

Figure 68: Winning Conditions ........................................................................................................................ 70

Figure 69: Fail Conditions ................................................................................................................................ 70

Figure 70: Overall Reactions to the GUI .......................................................................................................... 71

Figure 71: Text on the Screen ......................................................................................................................... 71

Figure 72: Overall Experiences of Interaction with the Game Play ................................................................. 72



17/05/2016 v.1.0

Index of Tables

Table 1: Facets & Sorting Criteria ................................................................................................................... 20

Table 2: Summary of the Results from Focus Groups‟ Testing....................................................................... 25

Table 3: Summary of the Results from 1st Users‟ Tests .................................................................................. 26

Table 4: Search Page Usability Questionnaire ................................................................................................ 27

Table 5: Data from the Use of the ISE Portal .................................................................................................. 27

Table 6: ISE Tools Repository Usability Study - Distribution of participants in different European Member States ............................................................................................................................................................... 29

Table 7: ISE Tools Repository Usability Questions ......................................................................................... 31

Table 8: ISE Tools Repository Usability Results ............................................................................................. 32

Table 9: Data from the Use of the ISE Repository .......................................................................................... 32

Table 10: ISE Authoring Environment Usability Study - Distribution of participants in different European Member States ................................................................................................................................................ 36

Table 11: ISE Authoring Environment Usability Questions ............................................................................. 38

Table 12: ISE Authoring Environment Usability Results ................................................................................. 38

Table 13: Data from the Use of the ISE Authoring Environment ..................................................................... 38

Table 14: ISE Delivery Environment Usability Study - Distribution of participants (STEM Teachers) in different European Member States .................................................................................................................. 42

Table 15: ISE Delivery Environment Usability Study - Distribution of participants (Students) in different European Member States ................................................................................................................................ 42

Table 16: ISE Delivery Environment Usability Questions ................................................................................ 46

Table 17: ISE Delivery Environment Usability Results (STEM Teachers) ...................................................... 46

Table 18: ISE Delivery Environment Usability Results (Students) .................................................................. 46

Table 19: Data from the Use of the ISE Delivery Environment ....................................................................... 47



17/05/2016 v.1.0

1. Executive Summary

This deliverable reports on the work done in WP5 with respect to (a) the final developments and deployment of the final version of the Inspiring Science Education (ISE) System that is currently being used for the Phase C of the ISE Project‟s pilots. Phase C of ISE project pilots started on October 2015 and it will last for ten (10) months, namely until July 2016. To this end, this deliverable deals with (a) the technical development and new features/functionalities of the different components of the ISE System since the intermediate release of the ISE System at M24 (described in previous deliverable D5.2.2), (b) an extensive study that was performed for evaluating the usability of all ISE System‟s components during the first six months of the Pilot Phase C, (October 2015 to March 2016) and (c) the final deployment of the ISE System and summative data about its usage until the end of M36 (31/03/2016). More specifically, the main outcomes of this deliverable can be summarized as follows:

Final developments, deployment and a usability study (with N=695 teachers) of the ISE Portal (http://portal.opendiscoveryspace.eu/ise) for supporting registered teachers to organize and manage national, thematic or school communities, as well as for sharing their lessons and educational scenarios created with the ISE Authoring Environment. The ISE Portal also provides to the ISE pilot teachers access to aggregated educational resources from various repositories. This is technically supported by a metadata harvesting process, which has resulted to harvest educational resources from eleven (11) different web-based repositories. The ISE Portal currently

2 includes 6.715

teachers and 210 teacher communities. Final developments, deployment and an extensive usability study (with N=61 teachers) of the ISE

Tools repository (http://portal.opendiscoveryspace.eu/repository-tool) for storage, search and retrieval of the different eLearning Tools that are offered by the project partners, as well as by external tool providers to be used by teachers during the lesson/educational scenario authoring process. The ISE Repository currently

2 includes 131 eLearning Tools, along with their related

metadata. Final developments, deployment and a usability study (with N=324 teachers) of the ISE Authoring

Environment for supporting teachers (a) to design their lessons as well as their educational scenarios by following a variation of the inquiry based teaching model (as proposed by WP3) and (b) to develop and include in the different phases of the inquiry based teaching model assessment tasks following the PISA 2012 problem solving framework (as proposed by WP8) towards assessing their students‟ problem solving competences. The ISE Authoring Environment has been used

2 by 1.169

teachers, who have developed 831 lessons/educational scenarios, as well as 2249 clones of these lessons/educational scenarios.

Final developments, deployment and a usability study (with N=121 teachers and N=136 students) of the ISE Delivery Environment for supporting teachers (a) deliver their lessons as well as their educational scenarios in their classrooms with their students and with the use of different devices (i.e., desktop and tablets) and (b) collect assessment data towards monitoring the development of their students problem solving competences. The ISE Delivery Environment has been used

2 by 238

teachers and 7715 students who executed 400 classroom runs of 83 ISE Demonstrators at 290 European Schools from 12 European Member States.

Final developments and deployment of external educational tools to be used in lessons/educational scenarios developed by teachers. More specifically, the following tools (from the intermediate release of the ISE System at M24, described in previous deliverable D5.2.2) have been further developed: (a) an augmented reality tool, namely the Science Center To Go Mobile with three different experiments, namely Bernouli Effect (http://goo.gl/ndzphf), Doppler Effect (http://goo.gl/TTzfWd), Double Slit (http://goo.gl/YkDfrJ), (b) Online Labs and Simulations, namely Learnit3D‟s Light Pollution (http://goo.gl/it8n6x), Learnit3D‟s Introduction to Telescopes (http://goo.gl/9Mj2Qx) and Learnit3D‟s Primary Mechanics (http://goo.gl/vnAVa4). Finally, the SIMAULA Educational Game (http://goo.gl/8i5JNi), which supports STEM teachers training on using inquiry-based teaching in their classrooms, has been further developed and tested about its usability with (N=30 university students).

2 Data retrieved on 31/3/2016

http://portal.opendiscoveryspace.eu/ise

http://portal.opendiscoveryspace.eu/repository-tool

http://goo.gl/ndzphf

http://goo.gl/TTzfWd

http://goo.gl/YkDfrJ

http://goo.gl/it8n6x

http://goo.gl/9Mj2Qx

http://goo.gl/vnAVa4

http://goo.gl/8i5JNi



17/05/2016 v.1.0

Based on the above, the use of the ISE System is expected to be greatly increased until the end of Phase C of the ISE Project‟s pilots (namely, 31/7/2016). This clearly demonstrates the large-scale deployment capacity of the ISE System and provides strong evidence about its sustainability after the project end.



17/05/2016 v.1.0

2. Introduction

2.1 Scope

This document stands for deliverable “D5.2.3 Deployment of Final Inspiring Science Education System” and its scope is to describe: (a) the technical development and new features/functionalities of the different components of the ISE System since the intermediate release of the ISE System at M24 (described in previous deliverable D5.2.2), (b) the study that was performed for evaluating the usability of all ISE System‟s components during the first six months of the Pilot Phase C, namely from October 2015 until March 2016 and (c) the final deployment of the ISE System and summative data about its usage until the end of M36 (31/03/2016).

2.2 Audience

As well as the WP5 partners, this document also targets the various ISE partners in other WPs, so that they can be aware of (a) the final functionalities and features of the different components of the ISE System (b) feedback received from the pilot users regarding its usability and (c) the data from the use of the different components of the ISE System until the end of M36 (31/03/2016). Moreover, this deliverable can provide insights about the technical development and the use of the ISE System to other interested parties external to the ISE Project consortium.

2.3 Structure

The deliverable is structured as follows. Following this introductory chapter, chapter 2 presents the new features/functionalities of the ISE Portal and the ISE Tools Repository (implemented since the intermediate release in M24), as well as results from the usability study and summative data from the use of ISE Portal and the ISE Tools Repository by pilot users until the end of M36 (31/03/2016). Chapter 3 describes similar information to chapter 2 but this is for ISE Authoring and Delivery Environment. Chapter 4 presents the new features/functionalities of (a) Science Center to Go augmented reality tool and the Learnit3D online labs/simulations, as well as (b) the SIMAULA educational game, which supports science teachers training on using inquiry-based teaching in their classrooms. Finally, chapter 5 presents our conclusions regarding the overall deployment capacity of the ISE System and its sustainability after the project end.



17/05/2016 v.1.0

3. ISE Portal and ISE Tools Repository

3.1 ISE Portal

3.1.1 Overview of New Features and Changes since the Intermediate Release

This section presents the new features and developments regarding the ISE Portal since the intermediate release in M24. During this period the following developments took place aiming to enhance and improve the experience of the users and the entire provided services:

Online users and teachers guidelines on the ISE pedagogical framework concepts and the use of the ISE portal services for search, create, modify and use of educational resources

Enhancement of the integration between the ISE portal and the ISE Authoring Environment. Enhancement of the ISE Academy in order to support the connection of the training

activities with other content in the portal Enhancement of the community services for more effective use and share of the content Enhancement of the search mechanism for the educational resources Support more effective presentation of the ISE Demonstrators through the portal Final version of the ISE Portal home page to integrate and support all the new

implementations Finalize the ISE Portal infrastructure

These new/enhanced features of the ISE portal are presented in details in the following sections.

3.1.2 New Features and Changes

3.1.2.1 “How to Use” Guide

In order to better support the introduction and guidance of the teachers on the ISE concepts and the use of the ISE portal services in the use, creation, modification and search of educational resources, the full and the short version of the “Teacher Guide” produced by WP6 was made available through the portal. A specific set of pages was implemented making all the designed material accessible through the following address: http://portal.opendiscoveryspace.eu/ise/how-to-use. The full guide was made available also in PDF format

3.

The figure below presents the home page of the relevant guide.

3 http://portal.opendiscoveryspace.eu/sites/default/files/u16/ise_guide_test_1-48p_160615_lr2.pdf

http://portal.opendiscoveryspace.eu/ise/how-to-use

http://portal.opendiscoveryspace.eu/sites/default/files/u16/ise_guide_test_1-48p_160615_lr2.pdf



17/05/2016 v.1.0

Figure 1: How to Use Guide – Home Page

3.1.2.2 Portal and ISE Authoring Tool Integration

The supported process for accessing the ISE Authoring Environment through the portal and to edit/create/clone a new educational scenario or lesson plan is presented in Figure 2. The coloured steps that are marked in Figure 2 represent the new features implemented to better support the users for easy and effective access of the tools and shorten the path of the entire process.

Figure 2: The ISE Portal and ISE Authoring Environment Integration Workflow



17/05/2016 v.1.0

These new steps include:

(a) The possibility of creating new lesson plans and educational scenarios through the user profile, by following the option depicted in Figure 3. In the next step, the user needs to select also one of the communities that he/she is a member of, where the new resource will be published.

(a) The user profile

(b) Select community

Figure 3: Access of ISE Authoring Environment through the User’s Profile

(b) Direct access to the “new resource” metadata editing form of the ISE Authoring Environment from

the profile of the user in this Environment, eliminating the intermediate steps as depicted in Figure 4.

Figure 4: Access of ISE Authoring Environment from Community’s Resources Page

(c) Provide in each (existing and new) community in the ISE portal a default scenario/lesson plan that is

used as “template” or “guide” for the users that attempt to create their own lesson plans or educational scenarios (see Figure 5).



17/05/2016 v.1.0

Figure 5: Default Educational Scenario/Lesson Plan

3.1.2.3 Training Academies: Connect with Communities and Resources

To better support the presentation of the training activities under the ISE Academy (http://portal.opendiscoveryspace.eu/topic-courses/inspiring-science-education-academy) and enable the user to find more related content uploaded in the portal there is a the possibility of connecting each course with selected communities and educational resources. These are defined by the authors of the courses and uploaded during the editing. Figure 6 presents such an example of the “SIMAULA Educational Game” training course connected with a relevant community.

Figure 6: Connecting Training Courses of ISE Academy with Communities and Resources

http://portal.opendiscoveryspace.eu/topic-courses/inspiring-science-education-academy



17/05/2016 v.1.0

3.1.2.4 Pin Events & Activities between Communities and Training Academies

The ISE portal supports the possibility for the community managers to “share” already existing events and activities uploaded in various communities to their own communities. The process is similar to this of “pinning” educational resources. This is the “Pin” option presented in the page of an activity or event. The activities and events are not actually transferred from the one community to the other, but simply referenced. The original source community is always named in the communities to which a resource is “pinned”. Figure 7 presents the workflow that the users follow in order to share such content with this functionality. The same process is also supported for sharing events among the communities and the ISE Academy, to support the visitors of the Academy to follow the relevant events implemented by the ISE portal communities‟ members.

Figure 7: Share Communities’ Activities and Events like Sharing/Pinning Resources

3.1.2.5 Search Educational Resources

The current section presents all the features and technical aspects of the search mechanism of the ISE portal in their final/enhanced version. This mechanism utilizes the tools and mechanisms implemented in the context of the ODS portal and adapts them based on the ISE portal needs and particularities. The search engine that utilizes the features of ODS portal search, consists of several components, some of which are already utilized by other functionalities and mechanisms in the portal as well. Figure 8 presents the general infrastructure that supports the search mechanism and the communication between the specific individual components.



17/05/2016 v.1.0

Figure 8: Architecture of the Search Engine

Following this overview, the search engine uses Solr4 as the search engine in Drupal and Carrot2

5 as the

graphical output interface of the search query (i.e. navigational search). Indeed, Carrot2 is an open source clustering engine implemented in Java. Carrot2 can automatically organize small collections of search results into thematic categories. That is, the clustering plug-in attempts to automatically discover relationships

among the resources in the search results and forms human-readable cluster labels.

4 http://lucene.apache.org/solr/

5 http://project.carrot2.org/

http://lucene.apache.org/solr/

http://project.carrot2.org/



17/05/2016 v.1.0

Figure 9: Search Engine Components and their Communication

The components of this architecture are described as follow: Apache: the classical web server holds the connections to the clients, serves static files, hosts the

PHP environment and can proxy to other servers (such as Tomcat). Drupal: the content management system, a PHP server software, is the core server which has been

adjusted for the Open Discovery Space purposes, it performs the editing and identification of users and resources, assigns URLs and runs the code in each of the modules.

MySQL Database: this classical relational database is responsible for the complete persistence of the objects manipulated by the Drupal server.

The harvesting pipeline yields LOM XML documents of each of the resources. The updater script, which converts the LOM XML documents to Drupal educational objects nodes. The Solr web app: is the search component that stores records with fields and their values. This

java application returns as many search results as requested ordered in relevance value or otherwise. Solr is talked to over an HTTP request by the ApacheSolr Drupal module. It is not directly accessible to users.

The Tomcat server hosts the Solr web app and the Carrot2 server. The Search and ApacheSolr Drupal module interfaces Drupal to Solr. It creates the records that

are posted as XML documents to Solr and queries Solr for search result. It is complemented by several other modules which together provide, in principle, the expected service, but which we have adjusted and extended.

The odsApacheSolr module hooks into the ApacheSolr module to complement and/or replace some of the functionalities so that multilingualism is performed and specific features are provided.

The ods‐search‐solrcomp solr component provides a query expansion facility that wraps Solr‟s

query-parser. It lives inside the Solr web‐app. Using it, both expert queries can be input, as well as regular queries which get expanded in queries to different fields (in title and metadata text, exact in correct language, stemmed in correct language, or whitespace trimmed in any language) and injects context dependent weighting queries which perform the desired relevance sorting (preferring, to a lesser extent than the exactness of the matches, the resources in the user‟s language, the resources with positive comments, or the resources which have been actualized more recently).

While searching based on keywords, the users have the ability to narrow/adapt the results they get using the available facets that are presented at the left side of the results page, or using the criteria that are available



17/05/2016 v.1.0

for different sorting of the results, based on their preferences. Table 1 presents and describes the set of facets and sorting criteria that are available. For the facets, the corresponding ODS-AP element is also provided.

Name of facet Description ODS-AP element Discipline The domain area of the resource Classification Discipline

Repository

The source repository of the resource. If it is created

by an ODS user, then the repository is “ODS”. The rest

regard harvested / integrated resources

Data Provider

Language The main language of the resource General language

Contributor The person who authored the resource Author Full name

Tags Words that feature the resource Edu Tags

Granularity Type of resource: educational object, lesson plan,

educational scenario

Aggregation Level

Learning resource

type The type of resource regarding the content of it

Learning Resource Type

Educational

context The context on which the resource can be used

Educational context

Copyright The copyright policy that the author implies Rights copyright

Cost If a payment is required for the use of the resource or

not

Rights cost

Date The date of it latest contribution EO update date

Name of sorting

criteria Description

Relevance The relevance with the keyword imported to the search box

Title Alphabetical order based on title

Date From the newest to the latest resources

Most comments Based on the number of comments that the resources have from the portal users

Contributor Order based on the name of the contributor

Repository Order based on the source repository

Granularity Order based on the type of resource (educational object, lesson plan, educational

scenario)

Table 1: Facets & Sorting Criteria

Apart from these criteria, the following features also supported: (a) auto completion of the search terms, (b) spell check suggestions and (c) Stemming-powered search The ISE Portal is a multilingual platform where users, both registered and not, can choose to browse in any of the supported languages, with a user-interface in these languages and, where available, content in this

language as well. The choice of language can be done using the pop‐up depicted on the right. The choice of this is based on the browser‟s header or the user‟s profile. The following 24 languages are supported by the portal: English, Dutch, French, German, Greek, Italian, Spanish, Croatian, Bulgarian, Danish, Czech, Finnish, Irish Gaelic, Polish, Portuguese, Romanian, Russian, Slovenian, Estonian, Latvian, Lithuanian, Serbian, Hungarian and Swedish. Finally, for some users it is also of interest to have a view of the entire metadata file of selected resources. This functionality is supported by the “Metadata Viewer” that is accessible through the Summary Page of the educational resources and includes all the content of the ODS-AP conformant educational resource file and its location of the ISE Portal server as well. The presentation of the elements follows a structural mode.



17/05/2016 v.1.0

3.1.2.6 ISE Demonstrators Page

For more efficient access of the demonstrators implemented in the context of the project a new area in the ISE portal was developed, where all the demonstrators are presented and organized. This area is available at: http://portal.opendiscoveryspace.eu/ise/demonstrators and provides the possibility for each demonstrator to access either its summary page (summary link) in the portal or the preview of the actual content (preview link) in the ISE Delivery Environment. To access these pages the user is not required to be logged-in to the ISE portal. Figure 10 presents this new area that is accessible through the homepage of the portal.

Figure 10: The ISE Demonstrators Page

3.1.2.7 New Layout of the Home Page

In order to support the new implementations and especially the new areas developed in the portal for the “How to use” guide and the presentation of the project demonstrators, the homepage of the portal was also slightly changed. The marked sections in Figure 11 were designed and developed to direct the visitors to the relevant areas and content.

http://portal.opendiscoveryspace.eu/ise/demonstrators



17/05/2016 v.1.0

Figure 11: ISE Portal New Home Page Layout

3.1.2.8 ISE Portal Infrastructure

The ISE portal infrastructure utilizes the infrastructure established by the ODS project / portal and it was finalized during the final period of the ISE project implementation. The architecture is presented in Figure 12 and presents all the individual components that were used in the context of the ISE project. A detailed description is provided in the relevant deliverable of ODS project, namely “D9.5: Pilot Deployment of an Open Discovery Linked Learning Space”.



17/05/2016 v.1.0

Figure 12: ISE Portal Infrastructure

3.1.3 Usability Study

For the final period of the project the main target was to improve, enhance and upgrade the services provided by the portal following the requirements collected from the partners and users of the portal and also the feedback collected individually while using all the features of the portal during the project activities and the everyday communication with the portal users. In addition to these, the analysis of the data collected by tracking the users‟ actions in the portal was also a significant input on how the available services are utilized by the communities and which are the further needs and trends in the use of such tools and services. This work was tightly linked with the main objectives of the project and by the partners who are working in close collaboration with the teacher communities. The usability testing and evaluation of the portal towards these objectives took place taking into account the following:

Usability tests by experts (consortium & consortium network) Feedback collected by the online questionnaire that was made available from the ODS project and

utilized for ISE project too Feedback from partners working on the ISE portal Feedback from users participated in the project activities

3.1.3.1 The Online Questionnaire

The questionnaire that is available online in the ISE portal (see Figure 13), uses the structure and questions set by the ODS project and it is accessible to all the registered and logged-in users. It includes seven main questions that examine the following:

The community(ies) environment The available resources The intention of the users to use the supported tools in the classroom, and to recommend to others The effectiveness of the service



17/05/2016 v.1.0

Figure 13: Online Questionnaire of the ISE Portal

From the results collected (N=695 science teachers), it is shown that the vast majority of the portal users find the available services and environment useful enough to use it and also suggest it to other teachers / users (see Figure 14).

Figure 14: Results from the Online Questionnaire of the ISE Portal

Apart from the structured questions, the form also includes an open-text section, asking from the users to mention the strongest and weakest features of the portal. Having analysed this information the participants consider the strongest points to be: (a) the community building environment and tools and (b) variety of valuable and re-usable resources in various languages. As weakest feature, is considered the performance



17/05/2016 v.1.0

of the portal. Taking into account this, the technical team of the portal proceeded to several improvements in the infrastructure and also optimization of the code that is a work still in progress, since it is expected that a new architecture for the infrastructure will be established.

3.1.3.2 Experts’ Usability Test

The current section presents summary of the results of the usability test performed for the ISE portal. It is worth mentioning that the usability tests took place at the start of the final project period but not reported in D5.2.2 deliverable. All collected comments and feedback were addressed in the current version of ISE portal. For these tests the following aspects were implemented:

Heuristic analysis for the ISE Portal6

User testing following contextual approach7

o Five second tests, asking the users to tell us as much as possible about the website topic, what it offers, etc.

o Unstructured interview, asking the users about the structure of the page, their impressions about where they will get if they press on different links, the possibilities of the site as well as other subjective impressions.

Focus groups‟ testing on specific pages

In total 695 unique statements were found, 46.5% were negative (n=323), 41.6% were positive (n=289) and 18.7% (n=130) were neutral. Some recommendations made by the researchers are: Clarity: Make the purpose of the site clear through a short description with readable pieces of text on the “Home” page and the „About us” page. Also explain who the audience is and make sure that the entire site is suitable for both the primary and secondary target audiences. Some respondents also prefer the use of one common language within the platform, preferably English. Layout: Make sure the layout is not too inconsistent. Be sure that each page has the same layout so that the pages be experienced as positive. This can be done by creating more consistency in terms of layout and use of colour, and by making choices that are tailored to the target audience. Structure Be consistent in the structure of the pages and don‟t make it too busy. For example, make a homepage with a few blocks or with more space between the blocks, to ensure that in the future the layout would be experienced more pleasant and that the clarity goes up. By adding navigation bars and by having consistency between the different sections within the platform, the structure will be clearer.

Table 2: Summary of the Results from Focus Groups’ Testing

6 A usability inspection method for computer software which involves expert evaluators examining the interface and judging its

compliance with recognized usability principles (the "heuristics"). More information is available at: http://www.usability.gov/methods/test_refine/heuristic.html 7 User-Centred Design (UCD): An approach to design that grounds the process in information about the people who will use the

product. UCD processes focus on users through the planning, design and development of a product. More information is available at: http://www.usabilityprofessionals.org/usability_resources/about_usability/what_is_ucd.html

http://www.usability.gov/methods/test_refine/heuristic.html

http://www.usabilityprofessionals.org/usability_resources/about_usability/what_is_ucd.html



17/05/2016 v.1.0

Major/global issues: The use of font sizes and colours seems to be arbitrary. Font sizes, colours and styles as bold or

italics should be used following an information schema. The way in which colours, sizes and styles are used don‟t respond to a particular hierarchy or information organization schema and highly confuse the users.

The template used is not responsive. The page shows information in different languages: English, Greek, Czech. Users were not able to properly identify the basic aspects of the website in the five seconds tests.

Five of them were not even able to realize about the education topic. In the ten seconds test users were able to appreciate the basic aspects of the site: “It‟s related to

education”, “It‟s an European project” but not the utility. This means that when users were asked about “How could you take advantage of this site?”. Most of them were not able to reply, and those who could only provide very vague (and sometimes incorrect) replies: “To contact other teachers”, “To look for education resources”, “To participate in on-line courses”.

Subjective impressions regarding the design were disappointing. We provide some of the most frequent appreciations:

o “The design is chaotic.” / “I don‟t know were to look.” o “Too much information.” / “Too many colours.” o “I don‟t understand what this page is about.” / “I don‟t think I‟m a potential user of this site.” /

“I think this site is of not use for me.” o “It‟s a very old-fashion design.” / “The design looks quite old.” (A couple of users even asked

if we were going to re-design / re-launch an old project) o “This doesn‟t seem to be accessible.”

Both the heuristic analysis and the tests with users clearly point out that a mayor redesign is required for the landing page.

Table 3: Summary of the Results from 1st

Users’ Tests

Following, the questionnaire that was used for the usability of search page is presented in Table 4.

Name:

Teaching at (include country):

Teaching experience (years, area):

Email:

Date:

Short Questions Strongly Agree

Agree Neutral Disagree

Strongly Disagree

1. The design of the search page fits with the overall portal design.

2. The search functionalities are clear and understandable.

3. I find the search page easy to use.

4. My overall impression of the search page is positive.

Top five “Filter by” options



17/05/2016 v.1.0

Please rank the 5 most important filtering options choosing from this list: Language, Age, Context, Tags, Contributor, Date, Discipline, Granularity and Repository. If you consider any other option, please tell us about it in the next page of the questionnaire.

1- 4-

2- 5-

3-

Top three “Order by” options

Please rank the 3 most important options for ordering the results choosing from this list: Alphabetical, by Date, by Age, by Popularity. If you consider any other option, please tell us about it in the next page of the questionnaire.

1-

2-

3-

Please use this page to tell us about possible improvements and problems that you have detected while trying the Search Page

Table 4: Search Page Usability Questionnaire

3.1.4 Summative Usage Data

Table 5 presents snapshots of different data from the use of the ISE Portal at two different timestamps, namely at M24 and at M36. As it is evident from Table 5, there is a considerable increase of the data from the use of the ISE Portal from the intermediate release at M24 to M36, which is four months before the end of Pilot Phase C. More specifically, the number of connected teachers has been increased by 50% and the number of teacher communities has been increased by 116%, whereas the number of external harvested repositories has been also increased by 22%. These growth rates clearly demonstrate the continuous and increased usage of the ISE Portal by its end-users.

# Type of Data M24 (31/3/2015) M36 (31/03/2016)

1 Number of Connected Teachers 4483 6726

2 Number of Teacher Communities 97 210

3 Number of External Harvested Repositories of

Educational Resources 9 11

Table 5: Data from the Use of the ISE Portal

3.2 ISE Tools Repository


This section presents the new features and developments regarding the ISE Tool Repository since the intermediate release in M24. During this period the following developments took place aiming to enhance and improve the experience of the users:

Improve the layout and presentation of eLearning Tools metadata

Enhance the searching/browsing functionalities of the repository, so as to facilitate teachers to find

eLearning Tools that match their ICT competence profile.



17/05/2016 v.1.0


3.2.2.1 Improved Layout of eLearning Tools Metadata Presentation

The layout of the “Subject domain(s)” filter has been changed from the list of checkboxes into comma separated values field. Also, the layout of the “Age range” filter has been changed from a list of values into comma separated values as well. Further to these changes and in order to improve the efficiency of the navigation to the various e-Learning Tools of the repository the metadata values to the fields of the “Subject domain(s)” and “Age Range” have been changed to active links, where the user can navigate to a list of eLearning tools with the same metadata (see Figure 15). Additionally, active links on the “Big Ideas of Science” icons were added, so as to navigate the user eLearning tools that support the same Big Ideas of Science (see Figure 15).

Figure 15: Presentation of the eLearning Tools Metadata after Improving the Layout

3.2.2.2 Enhanced Searching/Browsing of eLearning Tools

The searching/browsing mechanism of the ISE Tools Repository has been enhanced, so as to support teachers to be presented with eLearning Tools that match their ICT Competence Profile, which they have completed during their registration to the ISE Portal. This feature has been implemented with the addition of a new button at the Search section of the ISE Tools Repository. More specifically, this button is presented only to logged-in users, who have the teacher role

Figure 16: Search eLearning Tools that Match Teacher’s ICT Competences

For teachers that are logged in and do not have their ICT competence profile completed, the button is displayed inactive and the message “Your competence seems to be incomplete. Please click here



17/05/2016 v.1.0

(hyperlinked) to complete it” is presented. The button becomes active and ready to be used in filtering the tools only after the teachers complete their ICT competence profile.


The aim of this section is to present the work that has been done with users, specifically STEM teachers, to evaluate the usability of the ISE Tools Repository. The next sections describe (a) the methodology that has been adopted in our study, namely the profile of the participants who were involved in our study and the procedure that was followed, as well as (b) the usability results collected from the participants.

3.2.3.1 Sample

The study was conducted with N=61 STEM teachers. Table 6 presents the distribution of the number of STEM teachers participated in our study in different European member states. As we can notice from Table 6, our sample includes school STEM teachers from twelve European member states.

# Country Number of Teachers % per Country

1 Romania 16 26,23%

2 Ireland 12 19,67%

3 Greece 12 19,67%

4 Croatia 7 11,48%

5 Portugal 6 9,84%

6 Italy 2 3,28%

7 Bulgaria 1 1,64%

8 Finland 1 1,64%

9 UK 1 1,64%

10 Belgium 1 1,64%

11 Germany 1 1,64%

12 Spain 1 1,64%

Total 61 100,00%

Table 6: ISE Tools Repository Usability Study - Distribution of participants in different European Member States

Moreover, we present an analysis of the profile of our sample. This analysis will help us to identify a solid professional and educational profile of the respondents in order to ensure valid answers to our study. To this end, Figure 17 to Figure 21 present the distribution of the STEM teachers participated in our study regarding their gender, years of teaching experience, level of education, ICT knowledge and ICT usage during teaching. As we can notice, there is an adequate gender balance between the participants since there are 56% female teachers and 44% male teachers. The majority of the respondents have a solid professional teaching experience since 56% of them are professional teachers with more than 5 years‟ experience. Furthermore, as illustrated in Figure 19, 62% of the participants hold a post graduate degree (mostly masters degrees, with one PhD), while 33% of them hold a Bachelor Degree. The majority of the teachers (57%) who participated in the usability evaluation process were more than highly competent in using ICT (see Figure 20). Moreover, 57% of the participants were frequently using ICT in their teaching activities (see Figure 21). Thus, we can assume that our sample has enough experience with technology-supported STEM teaching so as to provide valid answers to our usability questionnaires.



17/05/2016 v.1.0

Figure 17: Gender Distribution

Figure 18: Years of Teaching Experience

Figure 19: Level of Education

2744%

3456%

Male

Female

2744%

58%

35%

2643%

0 – 5 years

6 – 10 years

11 – 15 years

> 15 years

2033%

3761%

11%

35%

BSc (Bachelors Degree)

MSc (Master Degree)

PhD (Doctorate)

Teaching Degree



17/05/2016 v.1.0

Figure 20: Previous Knowledge on ICT

Figure 21: ICT Usage During Teaching

3.2.3.2 Procedure

In order to evaluate the usability of the ISE Tools Repository an online questionnaire was developed that is available at: https://goo.gl/fqdakc The questionnaire was developed based on the System Usability Scale (SUS). The System Usability Scale is a quick and reliable tool for measuring the usability of software products. It consists of a 10 item questionnaire with five response options for respondents, where 5 denotes strongly agree and 1 denotes strongly disagree (Brooke, 1996). SUS has been extensively used for evaluating the usability of technology-enhanced learning applications (Harrati et al., 2016). As a result, it has been considered as appropriate for our study. To this end, the questionnaire included questions presented in Table 7

# Question

1 I think that I would like to use the ISE Tools Repository frequently

2 I found the ISE Tools Repository unnecessarily complex

3 I thought the ISE Tools Repository was easy to use

4 I think that I would need the support of a technical person to be able to use the ISE

Tools Repository

5 I found the various functions in the ISE Tools Repository were well integrated

6 I thought there was too much inconsistency in the ISE Tools Repository

7 I would imagine that most people would learn to use the ISE Tools Repository very

quickly

8 I found the ISE Tools Repository very cumbersome to use

9 I felt very confident using the ISE Tools Repository

10 I needed to learn a lot of things before I could get going with the ISE Tools Repository

Table 7: ISE Tools Repository Usability Questions

The questionnaire was disseminated by the project‟s national coordinators of each pilot country via e-mail to STEM pilot teachers, who have used the ISE Tools Repository and feedback was collected during the first six months of the Pilot Phase C, namely from October 2015 until March 2016. Based on the feedback received an overall SUS score was calculated as it has been proposed Bangor et al., (2008). More specifically, SUS score has been calculated as follows:

- For questions 1,3,5,7 and 9 the average score contribution is the average scale position minus 1. - For questions 2,4,6,8 and 10, the average contribution is 5 minus the average scale position. - The sum of the scores was multiplied by 2.5 to obtain the overall score of SUS.

610%

813%

1220%

1728%

1829%

1 (Very Low)

2 (Low)

3 (Moderate)

4 (High)

5 (Very High)

610%

915%

1118%

1626%

1931%

1 (Never)

2 (Rare)

3 (Sometimes)

4 (Frequently)

5 (Always)

https://goo.gl/fqdakc



17/05/2016 v.1.0

According to the relevant literature, an acceptable SUS score falls above 70, while exceptional software products have SUS scores above 90 (Bangor et al., 2008)

3.2.3.3 Results

This section presents the usability results from the feedback received from the participants of our study. Table 8 presents the mean and the standard deviation for all usability questions. The calculated SUS score following the scoring methodology described in section 3.2.3.2 is 70,04, which means that overall usability of the ISE Tools Repository is acceptable by its end-users, namely STEM teachers.

# Question Mean SD

1 I think that I would like to use the ISE Tools Repository frequently 4,15 0,89

2 I found the ISE Tools Repository unnecessarily complex 2,39 1,33

3 I thought the ISE Tools Repository was easy to use 3,92 1,10


Tools Repository 2,10 1,29

5 I found the various functions in the ISE Tools Repository were well integrated 3,87 1,07

6 I thought there was too much inconsistency in the ISE Tools Repository 2,46 1,21

7 I would imagine that most people would learn to use the ISE Tools Repository very

quickly 4,05 0,96

8 I found the ISE Tools Repository very cumbersome to use 2,39 1,31

9 I felt very confident using the ISE Tools Repository 3,85 1,12

10 I needed to learn a lot of things before I could get going with the ISE Tools Repository 2,48 1,41

Calculated SUS Score 70,04

Table 8: ISE Tools Repository Usability Results


Table 9 presents snapshots of different data from the use of the ISE Repository at two different timestamps, namely at M24 and at M36. As it is evident from Table 9, there is a considerable increase of the number of eLearning Tools that are hosted by the ISE Tools Repository between the two timestamps. More specifically, the number of eLearning Tools added to the repository has been increased by 340%. This growth rate clearly demonstrates that the ISE Tools Repository can be a central point where eLearning Tools Providers can share and disseminate their tools towards increasing their usage. More information about the type and the characteristics of the eLearning Tools that have been added to the ISE Tools Repository will be available at deliverable D4.3 - The Inspiring Science Education Inventory of eLearning tools – New Tools offered by federations. Finally, information about the eLearning Tools providers that have offered / uploaded their eLearning Tools to the ISE Tools Repository will be available at D9.6 Final Dissemination Report, where the activities of T9.3 Partner Affiliation Plan and Programme will be reported.

# Type of Data M24 (31/3/2015) M36 (31/03/2016)

1 Number of eLearning Tools

added 30

132

Table 9: Data from the Use of the ISE Repository



17/05/2016 v.1.0

4. ISE Authoring and Delivery Environment

4.1 ISE Authoring Environment


This section presents the new features and developments regarding the ISE Authoring Environment since the intermediate release in M24. During this period the following developments took place:

Development of additional entry points to the ISE Authoring Environment from the ISE Portal Upload and visualize MS Office documents during the process of adding educational content for the

inquiry activities of a lesson/educational scenario. Author lessons/educational scenarios without the presentation of questions for assessing problem

solving competences.


4.1.2.1 Enhanced Integration with ISE Portal

As already mentioned in section 3.1.2.2, the integration of the ISE Authoring Environment with ISE Portal has been adapted, so as to enable an easier and more effective access to the ISE Authoring Environment from the ISE Portal. To this end, the Single-Sign-on (SSO) API has been changed to include additional entry points from the ISE Portal to the ISE Authoring Environment. More specifically, two entry points have been developed as follows:

- An entry point from the user profile of the ISE Portal to the profile of the user in the ISE Authoring Environment the list of all lessons/educational scenarios are presented.

- An entry point from the “New Lesson Plans/Educational Scenario” at the ISE Portal to the metadata wizard of a new lesson/educational scenario in the ISE Authoring Environment.

4.1.2.2 Upload and Visualize MS Office Documents

As already mentioned in deliverable D5.2.2, each inquiry phase of a lesson/educational scenario that can be designed with the ISE Authoring Environment includes a set of predefined inquiry learning activities. These activities can be renamed, re-ordered or deleted by the teacher who is authoring the lesson/educational scenario. Moreover, each inquiry activity can include different elements such as MS office documents that are managed through a placeholder where the teacher can upload them (see Figure 22).

Figure 22: ISE Authoring Environment – Upload MS Office Documents for an Inquiry Activity



17/05/2016 v.1.0

Instead of presenting the MS office document as active links, the MS office Document Online Viewer has been exploited. To this end, the user is able to upload MS office documents (docx, xlsx and pptx) and immediately visualize them in the presented placeholder (see Figure 23).

Figure 23: MS Office Document Viewer Integrated in the ISE Authoring Environment

4.1.2.3 Authoring Lessons/Educational Scenarios without Problem Solving Questions

The ISE Authoring Environment has been updated to enable teachers to develop Lessons/Educational Scenarios without questions for assessing the problem solving competences of their students, which were mandatory in the intermediate release on M24. This provided increased flexibility to the teachers to design lessons/educational scenarios only with subject domain knowledge questions and measuring the effectiveness of their lessons/educational scenarios in terms of acquired conceptual knowledge by their students. Figure 24 presents an example of a designed lesson without problem solving questions via the ISE Delivery Environment; the section for assessing the problem solving questions are not visible to the student. It should be mentioned that in this case, the statistics about problem solving questions results are not gathered by the ISE Delivery Environment.



17/05/2016 v.1.0

Figure 24: ISE Authoring Environment – The Preview of an Example Lesson without Problem Solving Questions


The aim of this section is to present the work that has been done with users, namely STEM teachers to evaluate the usability of the ISE Authoring Environment. The next sections describe (a) the methodology that has been adopted in our study, namely the profile of the participants that were involved in our study and the procedure that was followed, as well as (b) the usability results collected from the participants.

4.1.3.1 Sample

The study was conducted with N=324 STEM teachers. Table 10 presents the distribution of the number of STEM teachers participated in our study in different European member states. As shown in Table 10, our sample includes school STEM teachers from twelve European member states.


1 Croatia 161 49,69%

2 Romania 31 9,57%

3 Ireland 22 6,79%

4 Finland 21 6,48%

5 Portugal 18 5,56%

6 Bulgaria 15 4,63%

7 UK 15 4,63%

8 Germany 13 4,01%

9 Greece 12 3,70%

10 Italy 6 1,85%

11 Belgium 5 1,54%



17/05/2016 v.1.0

12 Spain 5 1,54%

Total 324 100,00%

Table 10: ISE Authoring Environment Usability Study - Distribution of participants in different European Member States

Moreover, we present an analysis of the profile of our sample. This analysis will help us to identify a solid professional and educational profile of the respondents in order to ensure valid answers to our study. To this end, Figure 25 to Figure 29 present the distribution of the 324 STEM teachers who participated in our study regarding their gender, years of teaching experience, level of education, ICT knowledge and ICT usage during teaching. The results illustrate that there is an adequate gender balance between the participants, since there are 56% female teachers and 44% male teachers. The vast majority of the respondents have a solid professional teaching experience: since 78% are professional teachers with more than 5 years‟ experience. Furthermore, as shown in Figure 27, 48% of the participants hold a post graduate degree (29% with a Masters Degree and 19% with a PhD), while 50% of them hold a Bachelor Degree. The majority of the teachers (69%) who participated in the usability evaluation process were more than highly competent in using ICT. Moreover, 62% of the participants were frequently using ICT in their teaching activities. Thus, we can assume that our sample has enough experience with technology-supported STEM teaching so as to provide valid answers to our usability questionnaires.

Figure 25: Gender Distribution


14344%

18156%

Male

Female

7222%

5918%

4715%

14645%

0 – 5 years

6 – 10 years

11 – 15 years

> 15 years



17/05/2016 v.1.0

Figure 27: Level of Education

Figure 28: Previous Knowledge on ICT

Figure 29: ICT Usage During Teaching

4.1.3.2 Procedure

In order to evaluate the usability of the ISE Authoring Environment an online questionnaire was developed that is available at: https://goo.gl/7fHKqQ. A similar procedure was followed as with the evaluation of the usability of the ISE Tools Repository. More specifically, the questionnaire was also developed based on the System Usability Scale (SUS) and included the questions presented in Table 11

# Question

1 I think that I would like to use the ISE Authoring Environment frequently

2 I found the ISE Authoring Environment unnecessarily complex

3 I thought the ISE Authoring Environment was easy to use


Authoring Environment

5 I found the various functions in the ISE Authoring Environment were well integrated

6 I thought there was too much inconsistency in the ISE Authoring Environment

7 I would imagine that most people would learn to use the ISE Authoring Environment

very quickly

8 I found the ISE Authoring Environment very cumbersome to use

9 I felt very confident using the ISE Authoring Environment

16150%

9629%

6119%

62%


MSc (Master Degree)

PhD (Doctorate)

Teaching Degree

93%

196%

7423%

13040%

9228%

1 (Very Low)

2 (Low)

3 (Moderate)

4 (High)

5 (very High)

134%

258%

8426%

12037%

8225% 1 (Never)

2 (Rare)

3 (Sometimes)

4 (Frequently)

5 (Always)

https://goo.gl/7fHKqQ



17/05/2016 v.1.0

10 I needed to learn a lot of things before I could get going with the ISE Authoring

Environment

Table 11: ISE Authoring Environment Usability Questions

The questionnaire was also disseminated by the project‟s national coordinators of each pilot country via e-mail to STEM pilot teachers, who have used the ISE Authoring Environment and feedback was collected during the first six months of the Pilot Phase C, namely from October 2015 until March 2016.

4.1.3.3 Results

This section presents usability results from the feedback received from the participants of our study. Table 12 presents the mean and the standard deviation for all usability questions. The calculated SUS score following the scoring methodology described in section 3.2.3.2 is 73,02, which means that overall usability of the ISE Authoring Environment is acceptable by its end-users, namely STEM teachers.

# Question Mean SD

1 I think that I would like to use the ISE Authoring Environment frequently 4,09 0,77

2 I found the ISE Authoring Environment unnecessarily complex 2,20 1,05

3 I thought the ISE Authoring Environment was easy to use 3,82 1,02


Authoring Environment 2,03 1,19

5 I found the various functions in the ISE Authoring Environment were well integrated 4,00 0,86

6 I thought there was too much inconsistency in the ISE Authoring Environment 2,05 1,04

7 I would imagine that most people would learn to use the ISE Authoring Environment

very quickly 3,77 1,05

8 I found the ISE Authoring Environment very cumbersome to use 1,90 0,97

9 I felt very confident using the ISE Authoring Environment 3,97 1,00

10 I needed to learn a lot of things before I could get going with the ISE Authoring

Environment 2,27 1,19


Table 12: ISE Authoring Environment Usability Results


Table 13 presents snapshots of different data from the use of the ISE Authoring Environment at two different timestamps, namely at M24 and at M36. As it is evident from Table 13, there is a considerable increase of the data from the use of the ISE Authoring Environment from the intermediate release at M24 to M36, which is four months before the end of Pilot Phase C. More specifically, the number of teachers used the authoring environment has increased by 459%, the number of Lessons/Educational Scenarios created by teachers has been increased by 480%. This increase applies to original lessons/educational scenarios and cloned Lessons/Educational Scenarios. These growth rates strongly demonstrate the continuous and increased usage of the ISE Authoring Environment by its end-users. More information about the detailed usage of the ISE Authoring Environment per country will be available at deliverable D6.2.2 Report on Development of the Virtual Inspiring Science Education User Community.

Table 13: Data from the Use of the ISE Authoring Environment

# Type of Data M24 (31/03/2015) M36 (31/03/2016)

1 Number of Teachers used the ISE Authoring Environment 204 1169

2 Number of Lessons/Educational Scenarios Created by the Teachers used the ISE Authoring Environment

531 3080

2.1 - Number of Original Lessons/ Educational Scenarios 106 (19,96%) 831 (26,98%)

2.2 - Number of Cloned Lessons/Educational Scenarios 425 (80, 04%) 2249 (73,02)



17/05/2016 v.1.0

4.2 ISE Delivery Environment


This section presents the new features and developments regarding the ISE Delivery Environment since the intermediate release in M24. During this period the following developments took place:

Development of enhanced class assessment dashboard for the teacher to monitor students‟ assessment results

Translation of the ISE Delivery Environment to support the 13 different languages of the countries participating to the project piloting activities.


4.2.2.1 Enhanced Class Assessment Dashboard for Teacher

The teacher dashboard for monitoring results and statistics of the students to subject domain knowledge questions was enhanced. As it is presented below, minor changes have also taken place in the other sections as well, including the renaming of all tabs of the Class Assessment Dashboard, which is visible to the teachers. The new assessment tab now includes the following sections (see Figure 30):

PS Questions Real Time Statistics: provides information regarding students‟ performance on Problem Solving Questions in real-time.

SD Real Time Statistics: This is a new tab that provides information regarding students‟ performance on Subject-Domain Knowledge Questions in real-time.

PS Question Results: provides plots/graphs that depict overall class students‟ performance regarding Problem Solving Questions.

Subject-Domain Questions Results: provides plots/graphs that depict overall class students‟ performance regarding Subject-Domain Knowledge Questions.

Class Average Time Per Phase: provides information on the amount of time each student spent to

complete each phase of the lesson/educational scenario.

Figure 30: ISE Delivery Environment Teacher View – Class Assessment Feature

In the tabs related to the problem solving questions (Tab 1, “PS Questions Real Time Statistics” and Tab 3, “PS Question Results”) minor refinements have been made to visualise better the results and include lessons/educational scenarios where some students hadn‟t responded in some questions (which caused the graphs not to be displayed correctly). The same issue also affected Tab 5,”Class Average Time per Phase” and it has been corrected. The rest of the tabs, Tab 2 and Tab 4, are related with the Subject Domain Knowledge Questions. While a first version was presented in D5.2.2, the graphs have been refined based on the input received from pilot users and they have been exhaustively tested to be functional and connected with back-end APIs (see Figure 31).



17/05/2016 v.1.0

Figure 31: ISE Delivery Environment Teacher View – Subject-Domain Knowledge Questions Statistics

A completely new feature is the tab related with the Subject Domain Real Time Statistics. Similarly, with the Problem Solving Real Time Statistics tab, from this section the teacher is provided with statistical information regarding students‟ performance on Subject Domain Knowledge Questions in real-time. As shown below, for each of the students the teacher can view the correct and false answers, the number of questions not answered by the student and the number of retries. In the top area, the teacher is provided with more generic statistics for the specific lesson i.e. the overall number of students participating in the lesson/educational scenario, as well as the ability to refresh and export (in excel format) the displayed results (similarly to the other tabs).



17/05/2016 v.1.0

Figure 32: ISE Delivery Environment Teacher View – Subject-Domain Knowledge Questions Results

4.2.2.2 Multilingualism

With the support of consortium partners, the ISE Delivery Environment was translated in 13 European languages. More specifically, all menus and the sections of the ISE Delivery Environment were translated in the following languages: (i) Bulgarian, (ii) Croatian, (iii) Dutch, (iv) English, (v) Finish, (vi) French, (vii) German, (viii) Greek, (ix) Italian, (x) Irish, (xi) Portuguese, (xii) Romanian and (xiii) Spanish. In order for the end-users, both teachers and students, to change the preferred language, they have to navigate to the Settings menu and use the dropdown menu with all available languages, as shown in the figure bellow.

Figure 33: ISE Delivery Environment Teacher View – Changing the Language of the Interface



17/05/2016 v.1.0


The aim of this section is to present the work that has been done with end-users, namely STEM teachers and Students to evaluate the usability of the ISE Delivery Environment. The next sections describe (a) the methodology that has been adopted in our study, namely the profile of the participants that were involved in our study and the procedure that was followed, as well as (b) the usability results collected from the participants.

4.2.3.1 Sample

The study was conducted with N=121 STEM teachers and N=136 students. Table 14 and Table 14: ISE Delivery Environment Usability Study - Distribution of participants (STEM Teachers) in different European Member States present the distribution of the number of STEM teachers and students participated in our study in different European member states. As we can notice from Table 14 and Table 14: ISE Delivery Environment Usability Study - Distribution of participants (STEM Teachers) in different European Member States , our sample includes school STEM teachers from twelve European member states and students from eleven European member states.


1 Croatia 39 32,23%

2 Romania 22 18,18%

3 Ireland 14 11,57%

4 Greece 12 9,92%

5 Portugal 7 5,79%

6 Bulgaria 5 4,13%

7 Finland 5 4,13%

8 Italy 5 4,13%

9 UK 4 3,31%

10 Belgium 3 2,48%

11 Germany 3 2,48%

12 Spain 2 1,65%

Total 121 100,00%

Table 14: ISE Delivery Environment Usability Study - Distribution of participants (STEM Teachers) in different European Member States

# Country Number of Students % per Country

1 Romania 36 26,47%

2 Croatia 24 17,65%

3 Ireland 14 10,29%

4 Greece 14 10,29%

5 Portugal 10 7,35%

6 Finland 8 5,88%

7 UK 8 5,88%

8 Germany 8 5,88%

9 Belgium 6 4,41%

10 Italy 4 2,94%

11 Bulgaria 4 2,94%



17/05/2016 v.1.0

Total 136 100,00%

Table 15: ISE Delivery Environment Usability Study - Distribution of participants (Students) in different European Member States

Moreover, we present an analysis of the profile of our sample for both STEM teachers and students. This analysis will help us to identify a solid professional and educational profile of the STEM teachers participated in our study, as well as a balance to the characteristics of the students participated in our study. To this end, Figure 34 to Figure 38 present the distribution of the 121 STEM teachers participated in our study regarding their gender, years of teaching experience, level of education, ICT knowledge and ICT usage during teaching, whereas Figure 39 to Figure 41 present the distribution of the students participated in our study regarding their gender, age range and experience on using computers. As we can notice from Figure 34, there is an adequate gender balance between the participants since there are 55% female teachers and 45% male teachers (54 out of 121 total respondents). The vast majority of the respondents have a solid professional teaching experience since 74% of them are professional teachers with more than 5 years‟ experience. Furthermore, as we can notice from Figure 36, 48% of the participants hold a post graduate degree (30% with a Masters degree and 18% with a PhD), while 61% of them hold a Bachelor Degree. The majority of the teachers (69%) who participated in the usability evaluation process were more than highly competent in using ICT. Moreover, 65% of the participants were frequently using ICT in their teaching activities. Thus, we can assume that our sample has enough experience with technology-supported STEM teaching so as to provide valid answers to our usability questionnaires.

Figure 34: Gender Distribution (STEM Teachers)


5445%

6755%

Male

Female

3226%

1916%

1916%

5142%

0 – 5 years

6 – 10 years

11 – 15 years

> 15 years



17/05/2016 v.1.0

Figure 36: Level of Education (STEM Teachers)

Figure 37: Previous Knowledge on ICT (STEM Teachers)

Figure 38: ICT Usage During Teaching (STEM Teachers)

As we can notice from Figure 39, there is an adequate gender balance between the 138 participated students since there are 57% female students and 43% male students. Moreover, there is a balance between primary school students and secondary school students since there are 47% of students in the age range of 9-12 years old and 53% of students in the age range of 12-18 years. Finally, 75% of the students had high or very high experience in using computers. Thus, we can assume that our sample can provide valid answers to our usability questionnaires.

6150%

3630%

2218%

22%


MSc (Master Degree)

PhD (Doctorate)

Teaching Degree

93%

196%

7423%

13040%

9228%

1 (Very Low)

2 (Low)

3 (Moderate)

4 (High)

5 (very High)

54%

97%

2823%

4235%

3731% 1 (Never)

2 (Rare)

3 (Sometimes)

4 (Frequently)

5 (Always)



17/05/2016 v.1.0

Figure 39: Gender Distribution (Students)

Figure 40: Age Range (Students)

Figure 41: Experience on Using Computers (Students)

4.2.3.2 Procedure

In order to evaluate the usability of the ISE Delivery Environment two online questionnaires were developed that is available here: https://goo.gl/D4pJO6 (for STEM Teachers) and here: https://goo.gl/9bgwdk (for Students). A similar procedure was followed as with the evaluation of the usability of the ISE Tools Repository. More specifically, these questionnaires were also developed based on the System Usability Scale (SUS) and included the questions presented in Table 16.

# Question

1 I think that I would like to use the ISE Delivery Environment frequently

2 I found the ISE Delivery Environment unnecessarily complex

3 I thought the ISE Delivery Environment was easy to use


Delivery Environment

5 I found the various functions in the ISE Delivery Environment were well integrated

6 I thought there was too much inconsistency in the ISE Delivery Environment

7 I would imagine that most people would learn to use the ISE Delivery Environment

5843%

7857%

Male

Female

6447%

3022%

4231%

9-12

12-15

15-18

64%

2821%

4634%

5641%

2 (Low)

3 (Moderate)

4 (High)

5 (Very High)

https://goo.gl/D4pJO6

https://goo.gl/9bgwdk



17/05/2016 v.1.0

very quickly

8 I found the ISE Delivery Environment very cumbersome to use

9 I felt very confident using the ISE Delivery Environment

10 I needed to learn a lot of things before I could get going with the ISE Delivery

Environment

Table 16: ISE Delivery Environment Usability Questions

The questionnaire was also disseminated by the project‟s national coordinators of each pilot country via e-mail to STEM pilot teachers and students after the execution of classroom runs and feedback was collected during the first six months of the Pilot Phase C, namely from October 2015 until March 2016.

4.2.3.3 Results

This section presents usability results from the feedback received from the participants of our study. Table 17 presents the mean and the standard deviation for all usability questions from the feedback received from STEM teachers, whereas Table 17: ISE Delivery Environment Usability Results (STEM Teachers) presents the mean and the standard deviation for all usability questions from the feedback received from students. The calculated SUS score following the scoring methodology described in section 3.2.3.2 is 71,53 (for the feedback received from STEM teachers) and 71,14 (for the feedback received from students). This means that the overall usability of the ISE Delivery Environment is acceptable by its main end-users, namely STEM teachers and students.

# Question Mean SD

1 I think that I would like to use the ISE Delivery Environment frequently 4,06 0,88

2 I found the ISE Delivery Environment unnecessarily complex 2,25 1,07

3 I thought the ISE Delivery Environment was easy to use 3,86 0,99


Delivery Environment 2,12 1,17

5 I found the various functions in the ISE Delivery Environment were well integrated 3,93 0,96

6 I thought there was too much inconsistency in the ISE Delivery Environment 2,26 1,05



8 I found the ISE Delivery Environment very cumbersome to use 2,03 1,12

9 I felt very confident using the ISE Delivery Environment 3,97 1,07




Table 17: ISE Delivery Environment Usability Results (STEM Teachers)

# Question Mean SD

1 I think that I would like to use the ISE Delivery Environment frequently 3,87 1,02

2 I found the ISE Delivery Environment unnecessarily complex 2,26 1,16

3 I thought the ISE Delivery Environment was easy to use 3,82 1,05


Delivery Environment 2,26 1,24

5 I found the various functions in the ISE Delivery Environment were well integrated 4,15 0,87

6 I thought there was too much inconsistency in the ISE Delivery Environment 2,13 1,13



8 I found the ISE Delivery Environment very cumbersome to use 2,35 1,13



17/05/2016 v.1.0

9 I felt very confident using the ISE Delivery Environment 3,94 1,07




Table 18: ISE Delivery Environment Usability Results (Students)


Table 19 presents snapshots of different data from the use of the ISE Delivery Environment at two different timestamps, namely at M24 and at M36. As it is evident from Table 19, there is a considerable increase of the data from the use of the ISE Delivery Environment from the intermediate release at M24 to M36, which is four months before the end of Pilot Phase C. More specifically: (a) the number of teachers used the delivery environment has been increased by 780%, (b) the number of students used the delivery environment has been increased by 1200%, (c) the number of classroom runs has been increased by 980%, (d) the number of Demonstrators delivered has been increased by 1280% (e) the number of schools involved has been increased by 1220% and (f) the number of EU member states involved has been increased by 200%. These growth rates strongly demonstrate the continuous and increased usage of the ISE Delivery Environment by its end-users. More information about the detailed usage of the ISE Delivery Environment per country and per school is available at the ISE Dashboard Tool developed as part of the WP8 activities and it is available at: http://ise.iasa.gr/. Moreover, meta-analysis of the data collected by the ISE Delivery Environment will be reported at D8.3 Inspiring Science Education Validation Report and recommendations.

# Type of Data M24 (31/3/2015) M36 (31/03/2016)

1 Number of Teachers entered to the Classroom Runs 27 238

2 Number of Students entered to the Classroom Runs 595 7715

3 Number of Delivered Lessons/Educational Scenarios conducted by Teachers (Classroom Runs)

37 400

4 Number of Demonstrators Delivered 6 83

5 Number of Schools Involved 22 290

6 Number of European Member States Involved 4 12

Table 19: Data from the Use of the ISE Delivery Environment

http://ise.iasa.gr/



17/05/2016 v.1.0

5. ISE Educational Tools

This chapter provides detailed information on the new features and changes (since the intermediate release in M24) of the different external eLearning Tools that have been developed in the context of “Task 5.5 Tangible Interfaces”. These tools comprise (a) an augmented reality tool, namely the Science Center To Go, (b) a set of online labs and simulations, namely the Learnit3D online labs/simulations and (c) an educational game, namely the SIMAULA educational game. Finally, it is worth mentioning that while the Science Center To Go and Learnit3D online labs/simulations are included in the ISE Tools Repository (http://portal.opendiscoveryspace.eu/repository-tool), SIMAULA can be accessed through the ISE Portal Training Academies since this tool focuses mainly on teacher training.

5.1 Augmented Reality Tool: Science Center to Go

The Science Center To Go aims to assist learners with augmented reality (AR) to understand physical effects while interacting with tangible objects. The Science Center To Go consists of a set of tangible objects that are extended through an AR system. By pointing a camera towards registered objects the position of the objects is calculated and additional information is shown on the screen. All changes influence the calculated model in real time. Besides the ready-to-use Science Center To Go Suitcase a mobile version has been developed that utilizes the capabilities of nowadays smartphones and tablets and thereby reduces the need of dedicated hardware.


Since the intermediate release of the system the integration and functionality of the Science Center To Go mobile (android) application has been enhanced. First of all, a new landing page on the webserver was implemented. This allows a more seamless transition from the ISE Delivery environment to the smartphone or tablet when using a computer. Secondly the menu structure of the application was adapted to support learners with additional information besides running the experiment. The new structure also allows browsing back to the landing page. As a third feature, a new experiment for the android version was added to the set of scenarios. The Double Slit experiment, which also is included in the Science Center To Go suitcase, demonstrates the effect and interference phenomena of particles and waves as well as photons when shot at different configurations of single or double slits.


In this section the introduced features and changes are explained individually, ranging from universal to specific implementations.

5.1.2.1 Adapted Web Server Landing Page

The ISE Delivery Environment redirects the user to the specific landing page of each scenario. The newly designed landing page offers the same basic functionality as the previous to maintain consistency, but was extended. Besides a description of the experiment as shown in Figure 42, the link allows to start the application directly if already installed. The added QR code enables users to directly browse with an Android device to the same landing page, if not using the ISE Delivery on supported Android tablet or smartphone. This speeds up the process of changing from one device to another without the need of entering the URL manually or sending it via a third channel. Therefore only scanning the QR code with a suitable Android device using a QR code scanner is required.

http://portal.opendiscoveryspace.eu/repository-tool



17/05/2016 v.1.0

Figure 42: Science Center To Go Android Version - New Landing Page

For first-time users the landing page has been extended to provide all relevant information. The three-step guideline at first provides a link to the Science Center To Go Android application needed to run the experiment. In the second step a link to the necessary markers is available. By printing and cutting the markers according to the document, the learner can prepare the scenario of use on their own. The last step provides information on how to proceed as well as a second instance of the custom protocol encoded URL to directly start the application. Once installed, the experiment can be started with no further installation needed. For each experiment a dedicated landing page exists, showing individual information per scenario of use. By providing individual custom protocol encoded URLs the users are guided directly to the interactive experiment inside the application, making further explanation on how to navigate unnecessary. All pages are implemented following the responsive design principle this guarantees good readability on tablets and smartphones as well as desktop computers.

5.1.2.2 New App Dialog

In accordance with the new landing page described in section 4.1.2.1 the application menu was adapted. As shown in Figure 43 on the left of the following page, the first item is the Introduction item followed by the different scenarios of use. The introduction was added to support first-time users on how to setup the application in general and how the object recognition works for example advising users not to cover the printed markers as they are relevant for the calculation of the 3D content. The list of scenarios of use allows a flexible extension of the application such as adding a new Double Slit scenario of use. With the newly added “Double Slit experiment” the Android version of the Science Center To Go now features now three (3) scenarios of use (for more information of the already available scenarios of use please refer to deliverable D5.2.2), namely: (a) “Double Slit Experiment”, (b) “Doppler Effect” and (c) “Bernoulli Effect”. There is an individual description page available as shown in Figure 43 on the right. Besides the description a preview of how the scenario of use will look like during runtime is presented. This facilitates students to



17/05/2016 v.1.0

identify the objects needed and which 3D models are expected to appear. In addition the description page provides a link to the scenario of use landing page on the web server. There the student can download the marker needed. The custom protocol URL as described earlier redirects to this specific landing page. With a click on the „Start‟ button the actual interactive AR view of the experiment is started. The new app dialog provides an enhanced set of information to access and use the Science Center To Go Android application more easily.

Figure 43: Science Center To Go Android App's New Dialog

5.1.2.3 Double Slit Experiment

The Double Slit experiment visualizes the different concept of particles and waves side-by-side. Students are invited to test the Double Slit with virtual cannon shooting big particles or photons at a slit. The cannon might also be replaced by a source emitting waves with a certain frequency. In particle mode a virtual cannon fires virtual little "cannon balls" at the slit screen. After numerous balls a pattern analogous to a slit appears at the projection plane. The projection plane at the box shows an interference pattern. The learners are able to observe the experiment from different angles and change the distance of the screens, slits, as well as switch between the three different modes as shown in Figure 4.



17/05/2016 v.1.0

Figure 44: Science Center To Go Android - Double Slit in Particles mode

For running on Android mobile devices the tracking needs to be adapted according to the reduced field of view of standard Android tablets. In contrast to the suitcase version where changing between the different modes is implemented using a rotary switch, the mobile version uses an on screen menu making use of the touch screen functionalities. The flexibility in changing the slits and varying their position in relation to the box remain the same. With the Double Slit experiment the third experiment was added to the set of Science Center To Go Android tools extending the capabilities for experimentation that are being offered to the students.

5.1.3 Usability and Quality Assurance

During the development the described exhibits have been evaluated through expert reviews. This involves (a) partners of the ISE project, who were planning to use Science Center To Go experiments in their lessons/educational scenarios and (b) colleagues in the field of AR research. Following the design scheme defined during the SCeTGo project, the exhibits build on the results of the evaluation from end-users (http://www.sctg.eu/materials/SCTGO_GoGP-low.pdf). While the suitcase is based on a convertible laptop with an external USB camera, the mobile version uses tablets and smartphones with integrated cameras. This implies a different way of usage caused by the combination of display and sensor in one device. Throughout the phase of adaption the different approaches have been compared side by side to ensure the same quality and user experience as in the suitcase version. During the period of testing the following issues haven been raised by the projects partners:

One of the main issues was the interconnection of the ISE System and the Science Center To Go experiments especially switching between the ISE System and the experiments. This issue has been resolved through the usage of QR Codes and custom protocol as described above.

Another issue raised was the quality of the tracking in different lighting situation. Due to technical constraints the recognition of the markers works best under constant illumination. To identify and calculate the correct relative position of the markers, the checkered fields need to be fully visible to the camera. From the feedback gathered the configuration has been adapted to match most of the cases. This increases the stability of the tracking and thereby of the animations and the whole experience.

http://www.sctg.eu/materials/SCTGO_GoGP-low.pdf



17/05/2016 v.1.0

Regarding the Double Slit experiment changes have been performed as follows. While the suitcase version features a physical control dial to switch between the different modes (waves, particles or electrons), the mobile version was adapted to provide soft buttons (see Figure 44). For selecting the different mode in the suitcase version the whole experiment needs to be in the field of view of the camera. This is usable for a camera attached to a tripod. On the other hand while using a tablet or smartphone it was experienced that users were focusing more on selected parts of the experiment. For that reason the switch was implemented differently to increase usability.

Concerning the Bernoulli Effect experiment, it was stated that displaying the current angle of attack could support learners in their understanding. Due to the mathematical background of the visualization this information is available in the calculation and now displayed as an overlay on top of the box.

Even though the on-going technological progress provides more and more powerful smartphones and tablets the Science Center To Go mobile application supports devices starting from Android version 3.0 onwards. Thereby a sufficient degree of compatibility is maintained among the devices.

The aforementioned issues have been resolved and the new versions of the Science Center To Go application tested against the expectations again. During the whole runtime of the project the feedback received from the partners was constantly reflected in the adaption and development of the application.

5.2 Online Labs and Simulations: Learnit3D


The core change to the online labs and simulations provided by Learnit3D has been a move from core or pure 3D interactive tools requiring specific client software configurations (Direct3D, Java etc.) to simulations based on standard HTML5 & JavaScript technologies. Besides the HTML5 based experiments the Primary Mechanics experiment has been added to the set of tools developed by Learnit3D to be used in different demonstrators.


The end user / client deployment has been enhanced and is now supported using HTML5 and JavaScript. The fully interactive HTML5 and JavaScript simulations are supported by frameworks and libraries such as: (a) JQuery and (b) Paper.js. The developed online labs and simulations now run on all HTML5 supported browsers without need for java or similar plugins

5.2.2.1 Light Pollution

The fully interactive Light Pollution simulation allows the student to manage the light used and emitted in a small retail area car-parking facility. The aim is for students to manage a range of parameters to minimise light pollution while keeping the car-parking facility safe for the public. These parameters include (see Figure 45): (a) The number of lighting poles, (b) The height of the lighting poles, (c) The power of the lighting bulbs, (d) The types of lighting bulbs and (e) The type of shading or light restrictions on the bulb surrounds. For each combination of parameters the student views a 3D animation of the effect of their selections, including the level of visibility of the night sky.



17/05/2016 v.1.0

Figure 45: Light Pollution Experiment - Showing Interaction View

5.2.2.2 Introduction to Telescopes

The fully interactive astronomy telescope simulation introduces students to the key components and usage of the three main telescopes used for astronomy (see Figure 46 and Figure 47): (a) Refractor, (b) Reflector and (c) Cassegrain. The students learn the differences between these main types, and about basic usage.

Figure 46: Introduction to Telescopes - Launch Page



17/05/2016 v.1.0

Figure 47: Introduction to Telescopes - Different Telescopes Side-by-Side

5.2.2.3 Primary Mechanics

Primary Mechanics is a 3D desktop simulation (see Figure 7) for the introduction of basic mechanics for primary level students. Set in or else Based on an Egyptian archaeological dig-site the students work through a number of scenarios to transfer an important artefact from the site to a lorry where it will be moved to the safety of a museum. To achieve this they must use their skills and knowledge of forces, levers, mathematics, and experimentation.

Figure 48: Primary Mechanics Screenshots


While reducing the potential to exploit the full potential of 3D immersive learning, the move from a bespoke 3D simulation engine to core HTML5 & JavaScript technologies has significantly enhanced the potential deployment, take-up and usability of the LearnIT3D modules. The standard and well understood web



17/05/2016 v.1.0

interactivity will ensure students are able to use and interact with the simulations without prior knowledge or understanding. During the development process a highly iterative approach was adopted with regular advice, testing and feedback from pedagogical experts of the ISE consortium, as well as with external partners. More specifically, colleagues from NUCLIO tested and advised on the “Light Pollution” simulation and colleagues from the Northern Ireland Space Office and Council for Curriculum, Examinations and Assessment (Responsible for the qualification, Space Science Technology, tested and advised on the “Introduction to Telescopes” simulation, which supports the unit of that qualification “Distant Light in the Universe”

8

5.2.3.1 Light Pollution

During the development and testing with pedagogical experts for the ISE Consortium, as well as with external partners, we identified and resolved a number of technical and user experience challenges, as follows:

The brief was to reflect the affects of light pollution on a local environment using 3D simulation techniques, and therefore first challenge was how to present good quality 3D simulation using an HTML based delivery method to school and learning computers of varying video quality, potentially diverse operation systems, and hardware capabilities. Our solution was to build 3D simulation scenes in Unity 3D and render them as small video excerpts, embed these within the HTML experience, and deliver these to the leaner on demand within the overall structure of the light pollution web application.

The second challenge was to maximise the range of parameters available to the student to see the affects of light pollution on the local environment while keeping the number of video excerpts to a practical number. We settled on a combination of 72 possible combinations of number of lighting poles, lighting poles height, bulb type, bulb power, and shield type.

Although the original video excerpts of the 3D simulation renders correctly reflected the affect of light pollution on the local environment – a small retail outlet and residential housing - based on changes to the variable inputs, we discovered that the affect of these inputs on the visibility of the sky was not reflected. We therefore calculated this in terms of visibility of the night sky at each sample point, added a night sky dome to our Unity scenes, and re-rendered the video excerpts.

A static view of the night sky was added to the web application following a review by space observation experts in the UK. Two static views of Ursa Major can be seen at any point by the student. One shows how the constellation would be visible with no light pollution, and the other shows how the view will reflected based on the current parameters selected. Students are able to compare and contrast these views with a single mouse click.

A final addition to the light pollution web application was a 'light pollution quiz'. This 10 question MCQ (multiple choice questions) encourages the students to consider a number of topics and areas for further investigation in their learning about light pollution.

5.2.3.2 Introduction to Telescopes

During the development and testing with pedagogical experts from the ISE Consortium, as well as with external partners, there were a number of changes requested to the overall web application, as follows:

Each discrete module of the telescope web application should open in a separate HTML browser tab. This would enable the learner to quickly and easily to performing the same actions and therefore compare and contrast the different telescope types, namely (a) Schmidt-Cassegrain, (b) Newtonian and (c) Refractor.

During each telescope eyepiece manipulation, the focus in the virtual eyepiece viewer should change proportionately.

The brightness of the celestial objects being observed was changed to reflect the magnification setting of the telescope being used.

The initial development of the individual telescope modules shared the same celestial objects to be observed – Jupiter, Mars, and the Moon. This did not reflect the actual usage of the different types of telescope in respect of Solar System and deep sky observations. The original three space objects

8 http://www.rewardinglearning.org.uk/microsites/space_science_technology/

http://www.rewardinglearning.org.uk/microsites/space_science_technology/



17/05/2016 v.1.0

were retained for the Schmidt-Cassegrain telescope, with Saturn, Andromeda and the Moon used for the Refractor telescope, and Jupiter, Andromeda and the Moon for the Newtonian.

In focus mode for the Schmidt-Cassegrain telescope the learner affects focus by dragging the primary mirror using the mouse. If the 'show paths' option was turned on the light paths did not follow the movement of the primary mirror. This was changed and the paths are now dynamically calculated and rendered when the primary mirror is moved.

The learner changes the focus of the Newtonian and Refractor telescopes by altering the position of the virtual eyepiece with the mouse. Initially, it was possible to position the eyepiece outside the practical range. This was changed after review so that, if the student drags the eyepiece outside a set min or max position it returns to the min or max position on mouse up.

The user experience of each discrete telescope module was changed after review to share a common approach. The identification of the individual telescope components, drawing of the light paths, selection of space objects to be observed, and the core modes of operation now conform to a single usage paradigm.

5.2.3.3 Primary Mechanics

During the development and testing with pedagogical experts from the ISE Consortium, as well as with external partners, a number of issues and challenges were resolved as follows:

The initial concept and delivery was too serious and dry for the target learners (primary aged 7 to 10). We researched similar STEM learning games for primary to determine tried and tested techniques to maximise learner engagement. This included adding a small degree of humour, increasing colour usage, and the addition of sound affects.

Further to the previous point we increased the level of engagement by adding more historical setting and information, especially at the beginning of the simulation. This helps to stimulate learner interest from the start. We also added an Egyptian 'guide' to explain tasks and concepts to the children.

The early development of core tasks allowed the children to attempt a task only two times before the correct potential combinations of number were revealed. In keeping with the experimental aspect of the learning, the lead instructional designer recommended that this should be replaced with the ability to try again repeatedly until a correct combination had been discovered.

In addition to the change to allow multiple attempts to solve the problem, the lead instructional designer also requested that we restricted the learner to solve a specific task before moving on to the next. Previous versions allowed a more non-linear approach, but research has shown that at primary age learners react and perform better using more structure serious games.

In all tasks, there is more than one combination of input parameters to achieve a correct outcome. To guide the learners through this particular concept performance and data feedback to the learners was increased after the review stages. When a learner does not select a correct combination of input parameters full information and guidance is now give in clear and supportive language.

Initially when the learner found a correct combination of input parameters to achieve a correct outcome they received a 'well done' message and were moved to the next task. To emphasise the concept of there being more than one combination of input parameters to achieve a correct outcome we inform them, as part of the 'well done' message, that other combinations 'might' also be correct, and encourage them to investigate these. A 'try again' button was introduced to support this new functionality.

5.3 Educational Game: SIMAULA


The SIMAULA educational game is designed and implemented for teacher training on inquiry-based teaching. The responsible partner, namely COVUNI, has already developed and presented the first scenario which attempts to help teachers to situate inquiry-based teaching into the context of teaching the topic of CO2. The game has now been enriched with two more scenarios such as the electromagnetic spectrum and the Foucault pendulum encompassing different interfaces such as level objectives, win conditions, fail conditions and students. The scenarios have been chosen in consultation with the consortium and the existing demonstrators prevalent in the ISE Portal. Further add-ons have been made with regards to enriching the dialogues with in-game experiments in order to make the game-play more interactive and immersive. In congruence to the inclusion of the scenarios, intrinsic feedback mechanisms have been integrated to steer learning and development; and are combined with the visual extrinsic feedback to support



17/05/2016 v.1.0

teacher‟s learning and performance. Furthermore, a user manual and a short presentation have been developed for helping players to get to know the basic features of the game. The current version of SIMAULA along with the user manual and the presentation is available from the training academies of the ISE Portal and it can be used so as STEM teachers to be trained in basic principles of inquiry-based teaching.


The overarching new features of the game include the electromagnetic spectrum and the Foucault pendulum scenarios highlighting nested dialogues to prompt users to learn two distinct features of the inquiry cycle, namely phase 3: hypothesis generation and design and phase 4: analysis and interpretation.

5.3.2.1 The Electromagnetic Spectrum

The Electromagnetic spectrum scenario has been designed around the following topics: Different assumptions/definitions/evidence on the kinds of radiation and familiarise with the kinds of

applications in which electromagnetic waves are used. Different ways of expressing a hypothesis for how radiation works. Testing a hypothesis of the use of radiation, for example between visible and invisible objects. Testing a hypothesis of radiation effect for explaining why it is harmful.

Then, these topics were mapped to game goals that the player would have to achieve while playing the game. The learning objectives are presented in Figure 49 and they are the following:

Ask three inquiry-based questions in a row Get all 4 students to inquisitive Get 2 students thinking about a hypothesis Help students set up, or stage your own, experiment correctly Get a student to test a hypothesis via an artefact

Figure 49: SIMAULA – Level Objectives

It uses the nested dialogue process (see Figure 50), which is comprehensively described in D5.2.2, using chat mapper, and then imported into Unity 3D via XML. The questions started by introducing the topic via a question. The player has 4 possible answers to choose from. The chosen answer influences the students‟ attention and comprehension meters, which are increasing or decreasing accordingly. The attention and



17/05/2016 v.1.0

comprehension mechanics have been updated to becoming decreased when the player fails to answer in a given time.

Figure 50: SIMAULA - Electromagnetic Spectrum Nested Dialogues Process

Questions asked by specific students have been added (see Figure 51), so as to initiate personal support and involvement emulating what is happening in a real classroom, where a continuous dialogical process may take place between a particular student and a teacher.

Figure 51: SIMAULA - Responding to a Question Raised by a Student



17/05/2016 v.1.0

A visual experiment is depicted to show how the light is refracted to different colours and then the students are asked to explain their observations based on the initial hypothesis (see Figure 52)

Figure 52: SIMAULA – The Light Experiment

5.3.2.2 The Foucault Pendulum

Similar with the electromagnetic spectrum scenario, the Foucault pendulum scenario has designed to deal with the following topics (see Figure 53):

Searching and constituting evidence on how the earth rotates Circular motion conservation of mechanical energy Physical Pendulum (actual model designed by Rich) in the class for demonstrating its basic

properties like a thin rope, a mass attached to the end of the rope (bob) and a fixed point where the pendulum hangs.

Investigate why it takes more than 24 hours to complete the round? Parameters that should be studied: Length of the pendulum, starting angle of the pendulum, mass of

the bob at the end of the pendulum, force of gravity.



17/05/2016 v.1.0

Figure 53: SIMAULA - The Foucault Pendulum

These topics were then mapped to goals that the player would have to achieve while playing the game. The learning objectives were similar to the electromagnetic spectrum:

Ask three IBQ questions in a row Get all 4 students to comprehend/to motivate Get 2 students to think about a hypothesis Help students to set up, or stage their own, experiment correctly Get a student to test a hypothesis via an artefact.

To facilitate interaction, and the process of experimentation as highlighted in the inquiry-based learning framework, the game now includes the actual Foucault experiment carried out by the students has been included based on the dialogue choices provided by the teachers (see Figure 54). It is anticipated that the inclusion of experiments in the game will show in practice the importance of actual application, experimentation and hands-on learning with a particular scientific artefact.



17/05/2016 v.1.0

Figure 54: SIMAULA - The Foucault Pendulum Experiment

5.3.2.3 Feedback Mechanics

Feedback mechanisms have been re-thought and re-developed to allow players to receive meaningful, intrinsic feedback that targets explicitly on improving learning during the inquiry cycle. Feedback appears when an answer to a question is not correct, in a text format on top of the dialogue box. Along with the icon-like extrinsic feedback, it is aimed to provide some deeper conceptualisations of what is inquiry-based learning and how it can be enacted in the context of a specific question (see Figure 55)

Figure 55: SIMAULA - Intrinsic Feedback mechanic



17/05/2016 v.1.0


The purpose of this study is to investigate the usability of the SIMAULA education game prototype in its current state. This will focus on 7 key usability features, namely:

The Interface The Use of Graphics The Use of Text The Use of Sound The Control and Interaction Systems The Feedback given to the Player The Responsiveness of the Game

Any of these features implemented poorly could result in a game that is not fit for purpose; as such identification of any potential problems during the development process is vital so they may be addressed. Each of the above listed features should be considered general enough, and as the actual game content is not being evaluated in this test, that the target audience for this project (science teachers) will not be required to take part. As no manual will be provided, and no training or tutorials will be given, the required candidates for this study will possess the following competences:

Moderate degree of computer literacy Experience of using computers to perform tasks Ability to recognize an expected output resulting from a performed action

As such it was decided that 30 students from the School of Computing, Electronics and Maths at Coventry University would be suitable candidates. We thought that a purposive sample of 30 students would be appropriate for the purpose of testing functionality and interaction with the game as opposed to testing the actual content (i.e. learning effectiveness). It is anticipated however, that in order to conduct an evaluation study for exploring the game‟s effectiveness in terms of awareness and learning increase of inquiry-based learning would require an extensively larger sample of more than 100 science teachers. Students were approached and asked to participate on a voluntary basis, they were provided with some information on the nature of the study and asked to read and answer a consent form. Only those who agreed to the following statement proceeded onto the study.

5.3.3.1 Questionnaire Design

The questionnaire was designed with a focus on closed questions with a small number of semi-structured questions to help add richness to the data. It has been purposefully designed and delivered online as to facilitate participants‟ efforts to respond to the questions at their own time and pace; as well as to structure and organise the process of retrieving data and curry out the analysis. The questionnaire is available online at: http://goo.gl/forms/U0r3GxxQS8. A screenshot of the initial page of the questionnaires is presented in Figure 56:.

http://goo.gl/forms/U0r3GxxQS8



17/05/2016 v.1.0

Figure 56: SIMAULA Usability Questionnaire

The questionnaire is divided into the following sections: Background Information: This section gives some background on the participant, such as

computer literacy, subject of study and age. Each of these are possible causes of bias in the feedback given and care should be taken when considering the usefulness of the data for those who do not meet the requirements set out in section 4.3.3.

Testing the Game: The purpose with this section was to evaluate the user reactions to various functions of the games design such as the control system and mechanisms, the objectives, and win conditions. It was decided that answers would be measured on a scale of a positive, negative or a mixed response to the system being examined. This would provide sufficient data to be collected



17/05/2016 v.1.0

without introducing too much granularity of data, though this decision can be revisited later for future implementations of the questionnaire.

Aesthetics and the GUI: Since a person‟s sense of aesthetics is more often down to personal preference, for the GUI we asked instead the participants to describe their experience of using the games interface. To do this a scale of 1-9 was introduced with a strong negative (such as frustrating) at the lower end, and a strong positive (such as satisfying) at the other end, participants were then asked to mark on the scale their experience of the games interface. This should allow us to track if there is a trend towards positive, negative or neutral on various aspects of the game. Also tested were aspects such as the legibility of the text and the layout of the interfaces windows. A section allowing additional quantitative feedback was also added to allow participants to experience opinions outside of the confines of the scale.

Learning the System: Using a similar scale to the previous section, the users reaction to learning how to use, and play, the game was gauged. A section allowing additional quantitative feedback was also added to allow participants to experience opinions outside of the confines of the scale.

Interface Capabilities: This section uses the same design as previously used. It is designed to gather information specifically on the interface and system capabilities such as response time and rate of system failures. As before a section allowing additional quantitative feedback was also added to allow participants to experience opinions outside of the confines of the scale.

Multimedia: The last section deals with the multimedia portions of the game, such as image and sound quality. Once more adopting the system to allow us to identify trends was used and a section allowing additional quantitative feedback was also added to allow participants to experience opinions outside of the confines of the scale.

5.3.3.2 Findings

The Findings showed that SIMAULA is a usable game to play with and there were no major problems experienced from the selected participants in terms of GUI, capabilities, multimedia and effort to learn playing the game. Although some of the users felt that the game could be further enriched with more in-depth GUI in 3 levels: before, during and after playing the game. In this section we present the responses of the participants to the questions they have been asked and the correlations emerged from the analysis of the data. The 30 students were from the computer science department, primarily studying computing (70%) and associated subjects such as games development (13.3%). This means that all students were computer literate and their technical expertise allowed them to evaluate the usability of the game as rigorous as possible.

Figure 57: Subject of Study

Responding to the question “how responsive was the game to actions that you performed”, 87,1% said that the game was completely responsive and the 12,9% said that it was somewhat responsive. This was mainly



17/05/2016 v.1.0

because some participants could not play the game due to out-dated hardware of the computers they were using.

Figure 58: SIMAULA Game Responsiveness

Concomitant to the high degree of responsiveness that the participants experienced, the visual aspects of the game seemed to be perceived as a key factor of engagement. 80,6% said that the game‟s visual aspects were engaging, 12,9% felt that 12,9 was somewhat engaging and 6,5 not at all. This was felt because in some instances players experienced the virtual classroom camera a bit disorienting as they would have liked to have all students vertically aligned to be viewed from the front to the back.

Figure 59: SIMAULA Engagement Levels

The core element of SIMAULA that supports the learning process of becoming aware of how inquiry-based teaching is used in the classroom is the dialogue mechanic. It is essential for all users to be able to conceptualise the value of the nested dialogue process and discern the learning value out of each dialogical process that starts from the teachers and followed-up by the student. 77,4% of the participants felt that the



17/05/2016 v.1.0

dialogues (i.e. dialogue mechanisms, dialogue boxes, box colours etc.) were completely natural and 19,4% felt they were in the borderline. This was mainly because they thought that it could be centred in the middle of the GUI rather leaning towards to the top.

Figure 60: SIMAULA Game Dialogue Mechanism

Participants felt that they were able to select a dialogue with their mouse or touch in an intuitive way without any major problems. The “respond” button was accessed easily when the user wished to answer a student‟s question after the initial dialogue response.

Figure 61: Selecting a Dialogue using the Game Controls

The level of involvement that the player may experience relates to the increased usability of the different features of the game. Participants found that the GUI, including the characters, classroom environment, classroom objects, icons, boxes, and text, would likely increase involvement. 21 (73,33%) felt that they became involved while playing the game due to the rich experience that the game offered, while around 7 (23,33%) felt mildly involved and 1 (3,33%) not involved at all.



17/05/2016 v.1.0

Figure 62: Involvement in Game Play

Adjusting the experience of playing the game to individual prior knowledge and experiences of playing games in general and playing SIMAULA in particular was essential to understand the reasoning behind the game. The entire design should be as intuitive as possible to allow players to align their own experiences with the game‟s experience. 87.1% said that it took less than a minute to adjust the game‟s experience to their personal experiences and 12.9% felt that they ware adjusting slowly – more than one minute due to their inexperience of playing educational games.

Figure 63: Adjustment of the Experience of Playing the Game

The time it took to the participants to adjust the experience of playing the game to their own experiences is directly related to the level of proficiency the players gained with playing the game. It was reasonable therefore for most of the users to perceive that they gain proficiency 77,4% with game play whereas 19,4% they did not feel proficient due not being able to adjust the game experience properly.



17/05/2016 v.1.0

Figure 64: Level of Proficiency in Interacting with the Game

Participants felt that feedback provided by the system: (a) in the form of icons, (b) in the form of textual deep description of what needs to be considered, (c) via the log where a review of what has been answered is provided for the user as to have a history, (d) what it has been responded, was clear and useful.

Figure 65: Feedback Mechanics

Focusing on questions and trying to respond based on an informed awareness of how the specific question can be applied to science teaching and learning with inquiry cycle depends on the overall usability of the game. Participants experienced that the GUI and overall efficiency of the game allowed them to become focused on answering the questions without any functional and graphical dysfunctions noticed. 77,4% felt that they were fully focused on responding to the questions while 22,6% felt that they were occasionally focused.



17/05/2016 v.1.0

Figure 66: Level of Focus in Responding to Questions

Achieving the game‟s objectives indicates that the user is getting to know how inquiry-based learning is used by enacting the science scenarios in the game. Participants felt that providing a robust interface with clear objectives helped them to reach the objectives. Most of the participants 23 (76,67%) perceived that it was very easy to achieve the objectives and 7 (23,33%) felt it was not. Sometimes they felt that a game level was tool lengthy which caused them to loose track of what it the objective of the game. It was a suggestion that each time an objective is achieved a visual notice should be provided, notifying the players that an objective has been fulfilled.

Figure 67: Game Objectives

In correlation to the easiness of understanding the game objectives, participants felt that it was relatively easy to understand the win conditions of the game 74,2% whereas 12,9% said that it was somewhat easy to understand and another 12,9 said that they did not understand the win conditions at all. This was mainly because they did not see when a winning condition had been met. A visual icon was suggested to resolve this situation.



17/05/2016 v.1.0

Figure 68: Winning Conditions

Similarly, the fail condition was easy to understand by most of the users 64,5%. The same concern was noticed in terms of encompassing a visual representation that would notify the user when a failing condition emerged.

Figure 69: Fail Conditions

The overall experience of the GUI was positive as the participants felt that it was clear and clean containing only the necessary elements for conveying the science scenarios. The colours used were not too bright or too dark and the fonts and texts were at the appropriate size for reading them without making a substantial effort.



17/05/2016 v.1.0

Figure 70: Overall Reactions to the GUI

As argued, the text on the screen was clear and to the appropriate size for the users to understand the meaning of the dialogues. Text was an important aspect of the GUI as it enabled the dialogues to be comprehensive enough for the users to understand the question.

Figure 71: Text on the Screen

Overall, the respondents were asked to provide feedback on how natural their interactions with the game feel. As SIMAULA is a “point and click” game with no complicated buttons combination, participants felt natural for them to interact with the controlling mechanisms of the game. The different interfaces were simple to read for becoming aware of various performance and progress updates emerged during the game play.



17/05/2016 v.1.0

Figure 72: Overall Experiences of Interaction with the Game Play

5.3.3.3 Conclusions

The overall usability of the game seemed to have had positive results to the 30 computer science students at Coventry University. Participants felt that the GUI was natural, intuitive and robust to discern the learning outcomes designed for inquiry-based teaching and learning of. Responsiveness and natural interactions were highly valued as positive, correlated with the dialogue mechanics, game objectives, win and fail conditions and feedback mechanics. However it was noticed as part of future recommendations that the game should include more visual representations to notify the user about when an objective has been met or a win/fail condition has occurred. Immersion was evidenced especially when focusing on responding to a particular question or making effort to improve based on the feedback received. This had to do with the simplistic interface and the easiness of the game controls (point and click mechanism). Since the game‟s main mechanics was the nested dialogues, it was not mentioned that plurality and diversity of game elements should also be considered. However, we feel that in order to make the game more engaging and motivating, future iterations should include other mechanics that would make the game more engaging and interactive by encouraging the player to do something other than responding to questions. A follow-up usability test would be helpful to see what other participants with different characteristics, (e.g. students / teachers, from different countries in different schools / universities) would experience the usage of the game. This will pave the way for further enhancements with regards to the GUI. In conjunction to this, a future study with science teachers would be helpful to elicit their experiences of gaining awareness on using inquiry-based teaching in the classroom.



17/05/2016 v.1.0

6. Conclusions

This deliverable, namely “D5.2.3 Deployment of Final Inspiring Science Education System”, presented the outcomes of the work done in WP5 for the 3rd project period. More specifically, this deliverable presented (a) the technical development and new features/functionalities of the different components of the ISE System since the intermediate release of the ISE System at M24 (described in previous deliverable D5.2.2), (b) the study that was performed for evaluating the usability of all ISE System‟s components during the first six months of the Pilot Phase C, namely from October 2015 until March 2016 and (c) the final deployment of the ISE System and summative data about its usage until the end of M36 (31/03/2016). It is evident from the results of the presented usability studies that all ISE System‟s components are of acceptable usability since the calculated SUS scores were higher than 70. Moreover, it is clear from the usage data that there is a continuous and increasing usage of all of the ISE System‟s components, and it is expected that usage data from all the ISE System‟s components will continue increasing until the end of Pilot Phase C, namely end of July 2016. It should be noted that ISE Authoring Environment and ISE Delivery Environment were received the highest growth rates (from the intermediate release at M24) in terms of usage from their end-users. These growth rates are very prominent and they demonstrate the large-scale deployment capacity of all ISE System‟s components. Finally, these results set the ground for preparing “The Inspiring Science Education Federation”, which will ensure the sustainability of the ISE ecosystem after the end of the project. More information about the ISE Federation will be reported at deliverable “D9.3 The Inspiring Science Education Federation”.



17/05/2016 v.1.0

7. References

Bangor, A., Kortum, P.T., Miller, J.T., 2008. An empirical evaluation of the System Usability Scale. International Journal of Human–Computer Interaction 24, 574– 594 Brooke, J., 1996. SUS: a „„quick and dirty” usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, A.L. (Eds.), Usability Evaluation in Industry. Taylor and Francis, London, pp. 189–194 Harrati, N., Bouchrika, I., Tari, A., & Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computers in Human Behavior, 61, 463-471.