ar-lesson, augmented reality explaining itself: a proposal ... · ar-lesson, augmented reality...

AR-Lesson, Augmented Reality Explaining Itself: a Proposal andImplementation of an Augmented Reality Lesson for Computer Science and

Technology Courses.

Hugo Rodrigues da Silva Filho, Fernando da Fonseca de Souza,Fernando A. Aires Lins, Luis G. R. Alves

Computer Science CenterFederal University of Pernambuco{hrsf,fdfd,faal2,lgra}@cin.ufpe.br

Abstract

Although the use of Augmented Reality (AR) in manytypes of application for training, education and other areasis well known many institutions that offer courses in Infor-mation Technology, Computer Science or others in the samefield still do not have significant classes exploring the po-tential of AR. This work proposes and describes the partialimplementation of an introductory class on AR using its fea-tures, i.e., superimposed text and 3-D objects over the realworld scene, fiducial marker tracking and user interactionto demonstrate concepts and practical issues found in ARsystem development. Therefore, turning the learning expe-rience much more effective than a theoretical based regularclass. The software also enables teachers to introduce ARto theirs students without the need of much expertise or timespending preparing the lesson.

1. Introduction

The technological evolution of computer peripheral de-vices such as Head Mounted Displays(HMD) and also ofthe world wide popularization of mobile devices as the newgeneration mobile phones and hand held devices are in-creasing the general interest on applications that provide tousers with new forms of interaction with machines, otherpeople and the surrounding environment.

Interactive, multimedia and adaptive approaches to les-son design are one of the most lasting effects of computerintroduction in education. Interactive lessons are those thatprovide feedbacks and information to the learner that as-sumes an active behavior, differently from regular lessonsin which the participation can be rather passive due to so-cial, environmental or behavioral characteristics of each stu-dent. Lessons can also be adaptive by considering learner

responses to evaluate levels of ability, motivation, progress,background and other related factors, and through this eval-uation, alter its course by proposing different levels of ques-tions, activities or a change in timing slowing down orspeeding things up [8] [5].

In this sense, AR-Lesson presents interactive featuressuch as the demonstration of the coordinate system used inARToolKit, where the system awaits for the user to com-plete a task in order to proceed the lesson. Further informa-tion can be found in Section 4.

One of the project’s goals is to generate a flexible soft-ware architecture to allow reuse of code and modules sinceour intent is to extend this type of interactive lesson to othercourses and contents by adding a lesson creator tool whereteachers and professors shall be able to build simple ARlessons to a very diverse range of disciplines by using textexplanations, images, audio and video.

The use of video lessons has been studied at least sincethe late 40s and its effectiveness compared to paper-basedlesson has been proved in experiments such as the one con-ducted by U.S. Army in 1947. In that experiment threegroups would receive the same training but with differ-ent versions: one was a video lesson, the other one was aself study paper-based lesson and the last one was a reg-ular class. Results show no significant difference betweenthe groups performance [5]. Many other studies have re-searched the use of media and computers in education.

As a starting point to the project we have done a basic re-search based on interviews (in which most items were sub-jective) and a short electronic survey that took place in orderto provide us more information about how the overall sce-nario of understanding and knowledge on Augmented Re-ality currently is in local Universities that offer ComputerScience and IT courses. More information is shown in Sub-section 2.1.

After checking the early results from the initial online

XII Symposium on Virtual and Augmented Reality Natal, RN, Brazil - May 2010

14

survey the AR lesson was planned as in a regular collegecontext using a few slides and images. Then, we filtered thecontent according to our best judgment of what would beconsidered essential to an introductory class and, therefore,should be included in the early demo version of the soft-ware. Content classified as too hard to explain or demon-strate, or of an advanced level was left out and shall be in-cluded in future versions.

1.1. Augmented Reality

Augmented reality is the process of superimposing areal environment scene with virtual three-dimensional ob-jects, generated by computer programs, through some tech-nological device. It allows the user to see the real worldaround added of virtual objects to it as complementary de-vices. This causes the user to have the idea that both vir-tual and real content coexist in a determined environment.Augmented Reality systems can be defined as those whichcombine real and virtual objects and real scenario, that pro-vide real time interaction and that registers the objects in3-D. [12] [3].

1.2. Related Work

Although we have no record of works or software di-rectly connected to our intents there are a number ofprojects and research works that have are worth to mentionsince they have somehow inspired and guided our researchand thus the prototype implementation.

Augmented Reality has been used with educational andtraining goals. One example to be mentioned is the useof AR to teach astronomy concepts such as earth-sun re-lationships” in an application that lets students manipulatevirtual planets while viewing their respective information[13]. Another significant work is HITLab’s interface projectknown as Augmented Tangible Molecular Models. Thisproject helps students to understand complex molecularstructures and features a Magic Book with animations [1].

A well known example of the use of Augmented Real-ity in different types of applications and environments canbe verified in the Magic Book technology: the augmentedbooks vary from children storybooks to technical manuals.The HITLab group, ATR MIC Labs, and Hirokazu Katofrom Hiroshima City University worked in several bookssuch as the one that features the fable of a Japanese princess.While reading through the book readers are introduced toeach character and can interact with the story by complet-ing tasks. The 3-D objects are displayed above the paperbook [6]. More recently, in 2008, HITLabNZ students’ re-leased a training software for English language learners thatoffers a tangible interactive interface made of 3-D cubes andan animated agent. By playing with the cubes that repre-

sent letters the student interacts with the system and getsresponses from the agent [4].

2. Project Inception

This section reports the inception phase of the work inwhich the most important goals were to develop a clear vi-sion of the project’s concept and check for its relevance.

2.1 Electronic Survey

The survey contained 9 questions and there were 12participants, all undergraduate or postgraduate attendingComputer Science courses. At the first part of the surveystudents were asked, among other questions, if their col-lege or university offers Computer Graphics, Multimedia orcourses alike, and also, to those who attended these courses,we asked to which extent they were taught.

The survey also contained the two explanations below:

• An Augmented Reality introduction: section contain-ing two examples of its use. One was a Magic Book [1]and the other was a hand held application that dis-played notes on the buildings of a city. These exampleswere enriched by two pictures;

• An introduction to our project: section describing theproject’s goals and concept. This section was enrichedby a picture one of AR-Lesson’s system screen.

These explanations were necessary since we needed toknow how the students’ that didn’t know anything aboutAugmented Reality would feel about our project.

By checking their background we realized that thosewho took Computer Graphics or equivalent classes alreadyhad notions on VR and AR but most of them still had nei-ther the practical experience nor the initiative to do anyAR system programming. As shown in Figure 1 most ofthem thought that programming an AR system was eithertoo hard or time consuming, sometimes they judge the taskas unbearable without a proper training. On the other hand,67% participants responded that their level of interest was5 on a 1-5 scale of interest( 1 being no interest and 5 be-ing the highest level of interest) on trying AR-Lesson andlearning more about the development of Augmented Real-ity Systems. The other 33% were fairly interested choosingeither 4 or 3 on the same scale.

2.2. Informal Interviews

The interviews were held in a informal fashion, verymuch like the so called coffee meetings. These meetingstook place after the electronic surveys with the goal ofcatching more subjective information such as how hard the


15

Figure 1. Level of AR project developmentcomplexity according to students that partic-ipated of the survey.

students thought it would be to develop a simple AR ap-plication or how useful such an this application type couldbe.

Results were stimulating and showed the need for an in-novative approach. We have also informally interviewed afew college professors of Programming Languages, Algo-rithms, Data Structures and Computer Graphics and theyseemed to be very open and interested in using the pro-posed AR approach in their classes to support lessons andshow relations between AR (technology and theory) andtheir courses.

It is important to mention that the data collection wasmore of an evaluation to analyze the project’s relevance thana depth study on student and professor abilities in VR andAR. Therefore, its results are not statistical data but only astarting point to a wider research.

More information will be available once we get moretests done with a larger number of students. We plan onperforming a more detailed study on the effects of the AR-Lesson within more constraints and controlled variables andsome sort of assessment. Indeed the assessment could bedone inside the AR world of AR-Lesson, or it could have apaper based test since it is simpler.

3. Tools and Methodology

This section enumerates and describes the tools and thedevelopment methodology used to build the prototype.

3.1. Methodology and Software Process

We have chosen to use a minimal instance that mixes bestpractices of incremental iterative software development, theUnified Process and the Scrum agile framework [10] [2].

The work started by focusing on the tasks consideredmore risky to ensure the project’s viability and future prob-lems that could cause the release to run late or over bud-

Figure 2. Examples of high priority UseCases.

get. This included learning and coding the systems’ basicOpenGL functions, texture mapping procedures and AR-Toolkit marker management. When the team was sure therewould be fairly little risk of facing a big programming issuethe Use Case implementations took place. A few prioritizedUse Cases are shown in Figure 2.

As for the use of an incremental and iterative process wehad full cycles of planning, requirements, design, imple-mentation, testing and evaluation.

3.2. ARToolKit

ARToolKit is a multiplatform software library that helpson the development of Augmented Reality applications byproviding means of dealing with 3-D objects registration,acquisition of camera image, marker patterns tracking, ren-dering, and interaction [11].

ARToolKit provides the video capture facility that is nec-essary to do marker tracking from camera data. The processof marker tracking demands processing time, consequently,it often decreases the application frame rate. Nevertheless,using a small number of markers at a time can guaranteethat the software will run with satisfactory frame rates evenon low configuration set ups.The performance of the libraryis directly related to the hardware used to run it; it varies ac-cording the type of video card, the amount of RAM memoryand the capacity of the processor, for example.

3.3. Displaying Graphics with OpenGL

ARToolkit doesn’t provide any sort of direct support tographics display or user interaction, although its extendablearchitecture is quite simple, as the examples that accompanythe library show and its use is straight forward to simpletasks like displaying 3-D models. Consequently any devel-opment must be done with the use of a graphics library orAPI such as Microsoft DirectX, OpenGL or Ogre 3D.


16

In this work we have chosen to use the OpenGL graphicslanguage mainly because of its performance that made of itan Industry standard [14].

3.4. Programming Languages and Develop-ment Tools

The programming languages used were C and C++ andthe programming was Microsoft Visual Studio 2008 Inte-grated Development Environment (IDE).

The user shall be positioned in front of the multi markerpattern in a distance between 85 and 105 centimeters so thevirtual texts and 3-D objects can be visualized, thus ensur-ing good interaction with the system. By aligning the user’sbody with the center of the markers set it will be provided agood position for the user to select and occlude markers pre-cisely once the system responds differently to single markerocclusion and multi marker occlusion.

The data model for the prototype is quite simple becausethere are a number of features that have not been designednor implemented yet. In the model, the Lesson entity rep-resents the main subject to be discussed. An occurrence ofa Lesson must have at least one Topic, but it can also havemultiple Topics. A Topic is basically the description of asection within a Lesson. An occurrence of Topic must haveone or more Steps. The Step entity is like a slide in presenta-tion programs. A Step might contain text, image, audio andalso a 3-D model. The prototype’s conceptual data model isshown in Figure 3.

4. AR-Lesson: Lesson Design

The lesson is being designed to present the fundamentalsof Augmented Reality in a simple and stimulating fashion.

In AR-Lesson we currently have the following configu-ration:

• Topic #1 - Welcome Message and Instructions: thispart consists of two subtopics. The first one is a wel-come message with a short introduction to the project.The second one is a brief instructional section to ex-plain how the lesson will work;

• Topic #2 - Introduction to VR, MR and AR: this partcontains an introduction and explanations on the mainconcepts of Virtual Reality, Mixed Reality and Aug-mented Reality. This section comprises definitions,images and texts of well known examples of each area;

• Topic #3 - Devices and hardware: this section presentsa variety of devices such as the types of Head mountedDisplays, haptic devices, data gloves, wearable devicesand others that can be used in VR, AR and MR to en-hance user experience;

Figure 3. Summarized conceptual datamodel.

• Topic #4 - Developing AR with ARToolkit: an intro-duction to ARToolKit, pattern recognition and overallstructure of an AR application;

• Topic #5 - Basic program structure step by step: a stepby step guide into a basic program;

• Topic #6 - Review and Conclusions.

We are designing a proper hardware support mechanismto better present AR-Lesson. Basically it shall consist of alarge base that will hold a adjustable metal or plastic tube toadapt the platform to user position and height. The markerset is already attached to a solid box and the support mecha-nism will allow sliding to the sides and back and forth. Theconcept is shown in Figure 4.

A important feature is the Virtual Board (Figure 5). Itconsists of a quad structure in which the explanations aredrawn. The Virtual Board position may vary during the les-son, it may also be hidden or enlarged.

Figure 6 shows a user in the process of advancing in alesson by occluding two markers. The yellow cube meansthat the marker at that position has been detected as in oc-clusion state consequently the system is checking if there isa second marker been hidden to give appropriate feedbackto the user by presenting the next step of the lesson.

The introduction on VR, MR and AR includes, for exam-ple, references the definition of Augmented Reality systems


17

Figure 4. Designed support mechanism withattached multiple marker set.

Figure 5. System introducing the VirtualBoard.

Figure 6. User occluding one pattern andthen a second pattern to proceed in the les-son.

as it is in Azuma’s work Survey of Augmented Reality” [3].Figure 7 shows a student reading the definition for charac-teristics that must be present in AR systems.

After the definition steps the system presents examples,one of them is the Magic Book [1]. At this time the sys-tem presents the example software and displays images, asshown in Figure 8. Then it invites the user to turn the cam-era toward the picture to take a better look at it, as shown inFigure 9.

We have also rewritten and adapted excerpts from HIT-Lab’s tutorials since we found them valuable resources.First, these tutorials have been written by ARToolKit staffthus it is official material. Second we found them very help-ful, simple and very well assembled.

Figure 10 shows an adapted example taken from HIT-Lab’s Tutorial 2: Camera and Marker Relationships” [7].This is a interactive part of the lesson in which the usermust follow instructions to perform tasks like reposition-ing the marker set to the left, to the right, pulling it closer orpushing it away. The goal is to explain how the coordinateschange and how this can be useful. In the example the useris positioning the set to the left until it reaches position -100in the x axis(in camera perspective), this value belongs tothe objects transformation matrix to the camera CoordinateSystem.

Once the user takes the marker set to the requested posi-tion, a icon is displayed to notify task’s completion and thesystem presents the next step of the lesson. In this case thecoming step is the presentation of the code behind the task,as shown in Figure 11.

The system is also capable of presenting 3-D models fea-


18

Figure 7. Azuma’s definition for AR systemsdisplayed onto the Virtual Board.

Figure 8. System introducing the Magic Bookexample.

Figure 9. User visualizing the Magic Book ex-ample by rotating the camera and positioningit closer to the marker set.

turing automatic rotation. It then presents the explanationon how to load Wavefront 3-D Models (in the .obj file for-mat) and manipulate them 12.

5. Conclusions and Future Work

Results showed positive responses to the project fromundergraduate and postgraduate students. Consequently,the main goal of validating the hypothesis that the devel-opment of an Augmented Reality system to teach aboutthe subject is worth it can be considered as been achieved.Those who had access to the under development prototypehave shown great interest and enthusiasm with the chosenapproach.

Future works include improvements on the 2-D inter-face, revising and improving texts and definitions, additionof videos and creating better and more attractive 3-D ob-jects. However, the most valuable improvement pointed bythe testers would be enhancing learner interaction with thesystem. This can be accomplished by adding a third markerthat shall hold a virtual pointer that activates system func-tions through corner detection in the same fashion as it is incommon 2-D user interfaces implementations of events likebutton clicking. Another way of accomplishing this wouldbe using haptic devices that would allow users to touch andmanipulate. Also, there is the possibility of using the unin-strumented hand approach where the interaction is done byhand postures, as described in Kolaric’s work [15]. Finally,a more complex approach could be chosen: the use of awearable platform like miva [9].

Another serious consideration is to design and develop


19

Figure 10. Sequence of user completing atask. Initial position of the marker set (a);User repositioning the marker set(b); Userfinishing activity and receiving system feed-back(c).

/*Check if the ’A’ marker set is atposition -100(Camera CS)*/if (config->trans[0][3]<=-100){/*Draw the TASK COMPLETED icon*/draw(config->trans,

config->marker[0].trans,"taskOK.bmp",2);

}

Figure 11. Sample code behind interactivetask.

Figure 12. User visualizes a 3-D Model andexplanation.

a much flexible Lesson Editor in which non-programmerssuch as teachers and parents would be able to build sim-ple lessons. For now, we will redesign the system architec-ture and improve its implementation so, if this new directionproves to be valuable there will not be much rework.

We look forward to the acquisition of a Head MountedDisplay and some study is being done to assure cost effec-tiveness and hardware compatibility. Furthermore, whenit comes to system development the best deal would be avideo see through with built in camera since it would avoidextra work of designing and attaching a camera.

Finally, although the initial results have been impressiveand confirmed our hypothesis of the need for a system ad-dressing the teaching and popularization of Augmented Re-ality, the real impact can only be measure once the first ver-sion of AR-Lesson gets released. Further study will be heldand more precise and new conclusions will be drawn.

References

[1] (2008) HITLab Projects Homepage. Available:http://www.hitl.washington.edu.


20

[2] S. Alliance. Computer games educa-tion overview, 2009. Available:http://www.scrumalliance.org/resources.

[3] R. T. Azuma. A survey of augmented reality. Teleoperatorsand Virtual Environments, 6, 1997.

[4] M. Baird, B. Reeves, and P. Buchanan. Arinteractive tutorial, 2008. Available:http://www.hitlabnz.org/wiki/COSC426.

[5] R. C. Clark and R. E. Mayer. E-learning and the science ofinstruction: proven guidelines for consumers and Designersog Multimedia Learning. Pfeiffer, 2007.

[6] R. Harrill. Readers become part of the action through high-tech mixture of traditional storytelling and virtual realityin uw’s ’magic book’, September 2000. Available:http://uwnews.washington.edu/ni.

[7] HITLab. Tutorial 2: Camera andmarker relationships. Available:http://www.hitl.washington.edu/artoolkit.

[8] D. H. Jonassen. Interactive Video. Educational TechnologyPublications, Englewood Cliffs, New Jersey, USA, 1989.

[9] a. D. d. S. Joo Marcelo Teixeira, G. Moura, L. H. Costa, anda. J. K. Veronica Teichrieb. miva: Contructing a wereableplatform prototype. Proceedings of the IX Symposium onVirtual and Augmented Reality, 2007.

[10] C. Larman. Applying UML and Patterns: An Introduction toObject-Oriented Analysis and Design and Iterative Devel-opment. Bookman, 2 edition, 2004.

[11] A. H. P. on HITLab, December 2009. Available:http://www.hitl.washington.edu/artoolkit.

[12] e. a. Paul Milgram. Augmented reality: A class of dis-plays on the reality-virtuality continuum. telemanipulatorand telepresence technologies. SPIE, 2351, 1994.

[13] B. E. Shelton and N. R. Hedley. Explor-ing a cognitive basis for learning spatial rela-tionships with augmented reality. Available:http://www.newhorizons.org/strategies/technology.

[14] D. Shreiner, M. Woo, J. Neider, and T. Davis. Openglprogramming guide: The official guide to learningopengl, version 1.1, September 2009. Available:http://www.glprogramming.com/red.

[15] M. G. Sinia Kolari, Alberto Raposo. Direct 3d manip-ulation using vision-based recognition of uninstrumentedhands. Proceedings of the IX Symposium on Virtual andAugmented Reality, 2008.


21

ar-lesson, augmented reality explaining itself: a proposal ... · ar-lesson, augmented reality...

Documents