Biology Learning using Augmented Reality and

Gaming Techniques

Dalia Marzouk, Gehad Attia, Nashwa Abdelbaki Center of Informatics Sience Nile University, Cairo, Egypt

Email: dalia.marzouk@nileu.edu.eg, gehad.attia@nileu.edu.eg, nabdelbaki@nileuniversity.edu.eg

Abstract: Learning process in education systems is one of the most important issues that affect all societies. Nowadays it becomes one of the main challenges facing learner and teachers as well. This is because of more than one reason; the gap between the student's needs and the way of teaching, the number of students per class in some countries, ignoring different skills and capabilities of different users, etc. There is an increasing need to make this process go smooth, interesting and attractive. More important it has to be interactive, collaborative and multiplayer- based. Augmented Reality (AR) means mixing reality and computer aided models to enhance our understanding to some concepts. Applying this technology will enhance dramatically students' understanding. Gaming Techniques (GT) and Augmented Reality in education will convert learning process from boring activity to an attractive one and enjoyable to play and understand. Moreover, they will help the students to gain educational benefits instead of losing time in playing useless. In this paper we introduce our novel idea to apply the concepts of Augmented Reality and Gaming Techniques to make learning process more interesting and effective especially in biology learning. We apply these concepts to design and construct a biology lab to deliver biology lessons related to body anatomy and related concepts in an interactive, collaborative and pleasant attractive way. Keywords: Education; Learning; Augmented Reality (AR); Gaming Techniques (GT); Kinect

I. INTRODUCTION

Nowadays, education is facing great challenges like the dependency on boring activities and limited number of communication methods like reading, writing and listening. This is in addition to neglecting the use of other teaching and attractive methods like using pictures, videos, sounds, etc. The number of students per class in some countries affects the efficiency of education systems. Accordingly there became an increasing need to find an innovative way that cares about and deals with all of these challenges. Here arise some important questions. What would happen if 3-dimensional object is not just viewable on screen? What if you can touch, rotate or scale it? How can we effectively enhance the learning experience, increase student's engagement and meaningful interaction with the world? What will happen if we use surface computing and tablets in learning process? Augmented Reality (AR) and Gaming Techniques could perfectly answer these questions. AR is the variation of Virtual Environments (VE), or Virtual Reality as it is more commonly called. VE technologies completely immerse a user inside a synthetic environment. While being immersed, the user DOI: 02.WCMCS.2013.1.13 © Association of Computer Electronics and Electrical Engineers, 2013

Proc. of World Cong. on Multimedia and Computer Science

80

cannot see the real world around him. In contrast, Augmented Reality allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. Therefore, Augmented Reality supplements reality, rather than completely replacing it [1]. “Figure 1” shows the meaning of Augmented Reality and how it differs from Virtual Reality.

Figure 1. Migram's Reality-Virtuality Continuum

Today, scientists, architects, medical students, engineers, and countless other professionals use Augmented Reality for training, education, research exploration, and the refinement and testing of designs. Augmented Reality has vast potential implications and numerous benefits for the augmentation of teaching and learning environment [2], [3]:

Helping students to explore class materials from different angles;

Enhancing collaboration between students and instructors and among students,

Enhancing student creativity and imagination,

Helping students take control of their learning at their own pace and on their own path,

Learning with various styles of learning.

Chen, Y tried to investigate how students could interact with Augmented Reality and physical models to evaluate their perceptions regarding representations in learning about amino acids. In this experiment, using markers, students enjoy rotating objects to see them in different orientations [4]. [5], [6], and [7] demonstrate 3D molecules structure and simulation of molecular behavior and interactions. In astronomy, the Augmented Reality technology is applied as a method for better students’ understanding of sun-earth system concepts of rotation/revolution, solstice/equinox, and seasonal variation of light and temperature [8]. In the biology area, a learning system on interior human body is produced to present the human organs in details [9]. Construct3D specifically was designed for mathematics and geometry education with three-dimensional geometric construction models [10]. It can help teachers and students to explore properties of interesting curves, surfaces, and other geometric shapes. Physics is another area where Augmented Reality can be used. Duarte, Cardoso, and Lamounier evaluated Augmented Reality to dynamically present an object that varies in time, such as velocity and acceleration [11]. Video Game could be another way of interactive education such that it assists students in easily grasp class concepts. Augmented Reality games give the educators the opportunity to utilize a new highly visual and highly interactive form of learning to show relationships and connections between objects. Malone [12] used these concepts to outline several guidelines that should be included in any education program. These guidelines include:

having clear and meaningful goals,

giving students feedback on their progress through multiple goals structuring and scoring,

having different difficulty levels. The decision of each used level must depend on the learner skills, and

surprising the player with random elements.

Making use of these guidelines will bring a new area of education methodologies and techniques. In this paper we introduce our Augmented Biology Lab (ABL) that mixes Augmented Reality concepts and Gaming Techniques to improve learning process. In our ABL we target to satisfy more than one need. It is trying to decrease the gap between biology knowledge and understanding. This gap exists due to the difficulty to imagine the positions of body organs, their shapes inside the body, how they interact, and the diseases related to each organ, etc. Augmented Reality and visualization techniques present a promising

Augmented Augmented

Reality (AR) Virtuality (AV)

Real Environment

Virtual Environment

81

solution for this need. Using Gaming Techniques improve the student engagement in the education process. This leads to enabling the students of self-managing the learning process in an interesting way. The rest of this paper is organized into six more sections. In section II, we explain our biology lab in details, its targets, and how to implement these targets in a game. The methodology we followed to provide our concept in Augmented Biology Lab (ABL) is discussed in sections III to VI. Finally the conclusion and future work are introduced in section VII.

II. AUGMENTED BIOLOGY LAB

Biology is a natural science concerned with the study of of life and organisms. Human structure is from the main approaches of biology science. In this field Students had difficulty with:

a) visualization of what they have learned in lectures; 2D materials are not easy to reconstruct in 3D world, b) the relationship of each organ to its surrounding structures, c) 3D zooming in interactive software to explore deeper layers such as vessels and/or nerve structure, d) visualizing and applying knowledge practically.

Human body structure requires students to memorize a great deal of information and contextualize this within the range of body functions. It involves teaching through pictures, diagrams and real models for easier understanding, and the application of virtual reality and Augmented Reality to such activity has great potential. Through 3D visualization, students can understand human structure in a less complex way when compared to other traditional teaching methods. In our Augmented Biology Lab we aim to mix studying and entertainment in the same educational platform to avoid boring from studying and consuming time in other activities. Our lab emphasizes in this phase on teaching biology specifically. The direct objective of the ABL is to target mainly the elementary school students. Our ABL makes students able to learn the shape of their bodies' organs, their positions and their functions. In that way the students will gain a gradually increased amount of information about their organs, and their biological systems. It emphasizes moreover, on the functionality of the biological systems. As an application for this concept we propose a multi-player game that contains a combination of organs and their names. At the beginning of every playing session a combination of organs in column and their names will appear in another column. The player will have to select the organ and its right name. After selecting the organ and its right name the system will position the organ in its right place on the image of the player's image. “Figure 2” shows a sample of the playing session.

Figure 2. The proposed game

The game could be a multi-level game starting from beginners’ level and gradually gets complicated to reach the most advanced level. The beginner level will introduce the pupil to the different body organs and their names. Its objective is to help the pupil in his first contact with body organs to recognize each different organ separately. The second level introduces the student to a specific biological system. Its target is to make him differentiate between different systems and their functionalities.

82

The third level contains have the organs and the systems that the organs belong to and vice versa. This level will make the student understand the function of every organ and how it affects the functionality of the organs. The fourth level includes parts of the organs like the right ventricle for the heart. The fifth level contains the systems and diseases related to them. The sixth level contains pairs of systems and the interaction between them. Each level is associated with a bonus and some challenges to achieve which helps the learner to enjoy the challenge and keep attracted to the learning process. In that way we offer the pupil/student to learn more than one concept at a time. The biology game makes him feel that he could learn biology concepts on his body in an interactive way driven by the desire to win the game. To develop our concept we use Microsoft Kinect and Kinect SDK. This combination enables us to use three outputs which are RGB frame, depth frame and the skeleton frame. Using the depth frame, the objects are displayed in gray image related to their distances from the Kinect sensor. The skeleton frame tracks twenty joint for up to six players. The Kinect technology offers audio input [13]. “Figure 3” shows a diagram that describes the sequence of states that the player passes during using the game. The diagram contains five zones or sessions. The first session is the logging session. The player could play with his old account or create a new account. This session facilitates tracking user's behavior and makes the game adaptive to the player. These days the research communities give increasing interest for serious educational games and how to make it personalized learning experience [14]. The second session is the place where the player chooses if he can start the game or go to the tutorial which is the third session where the game instructions are illustrated. The fourth session is the actual playing session. Here the actual steps the player showed follow to play. First the player chooses the style of playing. The game provides more than one way to check the answer using Kinect capabilities like making a certain movement as a sort of funny activity. Or the student could make some sound like clapping or say some funny word. After the player chooses the style of playing, he can begin with selecting either the organ's shape or the organ's name and then select the other one. The system gives the player the ability to undo his selection and choose again. Then the student asks the system to check his selection if right or wrong. If the selection is right the organ will appear on his right position on the body. If the answer is right the shape of the organ will appear on its right place on the player image. But in case of the wrong answer, the system will give the player a hint about the right answer. In both cases the knowledge of the player will increase. After that the system would update the right selections and decide if the game will proceed or the playing session has been finished. The next session is the ending session where the player could see his score and will be asked if he want to play another session or not. By that way the student would gain increasing level of knowledge about the organs and body's systems although he is playing.

III. THE SEQUENCE OF IMPLEMENTATION

To implement our proposed game we have certain stages and missions we should accomplish with the help of Kinect. As shown in “Figure 4”, we classified our sequence of implementation into five stages:

1) Video Capturing; where the Kinect captures both video frames and image frames in order to process them and apply Augmented Reality concepts on them,

2) human segmentation in order to extract the image of the player from the background. This stage also includes skeleton detection of the human body which will help us in determining the probable positions for the organs,

3) scaling and positioning the organs textures to be suitable to the student’s body, 4) game logic which is described in the previous section. In this phase of our work, we concentrate only on

hand gesture which is used as a tool for selection and make the system realize that if the player is clapping or not which is considered as a metaphor by which the system is enforced to check the answer. 5) The final implementation stage is viewing the Augmented Reality output of the game on the monitor. In this phase of our work we focus on the most challenging tasks, stages 2 and 3, the “Human Body Extraction” and “Scaling and Positioning” tasks including hand gesture as well. Stages 1, 4, and 5 are reserved for future publications.

83

Figure 3. Sequence of states diagram

Figure 4. Sequence of implementation

Video Capturing

Human Body Extraction

Scaling and Positing

Game Logic

Viewing

84

IV. HUMAN BODY EXTRACTION

In this task we extract the human body from the background In order to show his body on the right side of the screen. Kinect SDK enables us to analyze the depth frame and detect frame pixels that belong to tracked player; there is a unique index for each player which is stored in the first three bits of the depth pixel data as shown in “Figure 5”. A bit mask of 8 (0000 0111) get the player index from the depth value.

Figure 5. Pixel data

In that way, we can determine for each pixel in the depth image if it belongs to a valid player index or not. The next step is to get the corresponding color pixel for every depth pixel belongs to a player. This method may not be suitable in some cases. For example, if the player wears wide cloth or there is an object near the player. All these situations will make wrong segmentation. Also, if another player appears in the area of the sensor detection, the new player could be tracked instead of the main player. To solve this problem we apply constraints on the playing environment and we define and track player features to identify the main player.

V. SCALING AND POSITIONING To make the game generic and suitable for any player with any width and height we design the positioning and scaling of the organs texture in a general form that is flexible to change with any variance in the player size. We use the twenty joints of the Kinect skeleton frame to accomplish this task.

Figure 6. a. Rectangle for each organ. b. The rotation angle for limbs

“Figure 6.a” shows the organs rectangles drawn with the use of skeleton joints in one figure. For example for the rectangle where the lung texture should be drawn could be obtained with the following equations:

푟푒푐푡푎푛푔푙푒.푋푐표푟푑 = 푠ℎ표푢푙푑푒푟퐿푒푓푡푃표푖푛푡.푋푐표푟푑 + 16

× (푠ℎ표푢푙푑푒푟푅푖푔ℎ푡푃표푖푛푡.푋푐표푟푑− 푠ℎ표푢푙푑푒푟퐿푒푓푡푃표푖푛푡.푋푐표푟푑)

푟푒푐푡푎푛푔푙푒.푌푐표푟푑 = 푠ℎ표푢푙푑푒푟퐿푒푓푡푃표푖푛푡.푌푐표푟푑 푟푒푐푡푎푛푔푙푒.푊푖푑푡ℎ = 푠ℎ표푢푙푑푒푟푅푖푔ℎ푡푃표푖푛푡. 푋푐표푟푑

− 푆ℎ표푢푙푑푒푟퐿푒푓푡푃표푖푛푡.푋푐표푟푑

푟푒푐푡푎푛푔푙푒.퐻푒푖푔ℎ푡 = 23

× (푠푝푖푛푒푃표푖푛푡.푌푐표표푟푑

− 푠ℎ표푢푙푑푒푟퐶푒푛푡푒푟. 푌푐표표푟푑)

85

This method suits more organs in the body truck. To the area of the head we need the segmented user accurately to know the upper point in the head to be y-coordinate for the brain area also to know the limitations of the x-coordinate and brain width. The pervious method is needed for the limbs. Also we need to know the rotation angle of the limb. From figure.6.b assume that the angle between the vertical axis and the bone is called ɸ as shown in “Figure 6.b”. For example for the right arm ɸ will be:

VI. GESTURE

The basic idea in this task is the Hand Gesture which is used in: 1) the selection of organ’s name and organ’s shape image and then check if his selection is right or not.

We use here the position of left and right hands with the hovering buttons, 2) zooming in and zooming out. The system allows the student to zoom in to make the image that he

selected bigger. He can also zoom out to make the image smaller. Also this will be used in the scrolling feature. We use here the X-coordinate of the right hand and the left hand and also the X-coordinate of the Spine. The final usage of hand gesture is:

3) to make the student return back to the game after switching to the navigation view. We use here the Y-coordinate of the left and the right hands and also Y-coordinate of the head.

VII. CONCLUSION AND FUTURE WORK

Learning process in education systems is one of the most important issues that affect all societies. Nowadays it becomes one of the main challenges facing learner and teachers as well. New technologies like Augmented and Virtual Reality have become very important learning and teaching aids in education today. Integration of these technologies will increase students’ interest and retention of information. We have presented our novel Augmented Biology Lab (ABL) that aims to support learning biology using Augmented Reality and Gaming Techniques together. We have mix entertainment and studying in the same educational platform which manages the learning process in a positive way. The direct objective of the ABL is to make students able to learn the shape of their bodies' organs, their positions and their functions. In that way the students/learners gain a gradually increased amount of information about organs and biology systems. As an application for this concept we propose a multi-player game that contains a combination of organs and their names. The student/learner needs to mix and match each of the organ shape and its name then check its correctness that will then visualize this organ on the student‘s body. In the future, there are many ideas that can enhance our ABL. Visualizing the organs in 3D instead of 2D is on top of our currently planned stages. This will make the player/learner feel that he could imagine the concepts of biology in a more effective way. The other feature we are planning to add on our ABL is the concept of adaptive games. This feature aims to make gaming and teaching method specialized to every student/learner. This could obtained by sensing the student/learner behavior and style of learning and make this an input to the game so that the scenario of the game could adapt the player/learner. The area of sensing and modeling the player behavior is a wide area and there are more than one way to sense the player behavior like facial expressions; the facial changes in response to a person’s internal emotional states, intentions, his movement, his tone.

REFERENCE

[1] Yuen, S., Yaoyuneyong, G., Johnson, “Augmented reality: An overview and five directions for AR in education, ” Journal of Educational Technology Development and Exchange, 4(1), 119-140, 2011.

ɸ= 2 × 휋

− 푡푎푛퐸푙푏표푤푅푖푔ℎ푡푃표푖푛푡. 푌푐표푟푑 − 푠ℎ표푢푙푑푒푟푅푖푔ℎ푡푃표푖푛푡.푌푐표푟푑퐸푙푏표푤푅푖푔ℎ푡푃표푖푛푡. 푋푐표푟푑 − 푠ℎ표푢푙푑푒푟푅푖푔ℎ푡푃표푖푛푡.푋푐표푟푑

86

[2] Yuen, S.; Yaoyuneyong, G.;& Johnson, E. (2011). Augmented reality: An overview and five directions for AR in education. Journal of Educational Technology Development and Exchange, 4(1), 119-140.

[3] Azuma, R.T, “Survey of Augmented Reality. Presence: Teleoperators and virtual environments,” vol. 6, pp. 355-385, August 1997.

[4] Chen, Y., “A study of comparing the use of augmented reality and physical models in chemistry education,” ACM International Conference on Virtual Reality Continuum and its Application, Hong Kong, China, June 14-June 17, pp. 369--372, 2006.

[5] Patrick and Gudrun, “Augmented Chemical Reactions Small molecules with a large impact.” [6] Samarth, Sameer, Praroop, “ Augmented Chemistry: Interactive Education System,” International Journal of

Computer Applications (0975 – 8887) Volume 49– No.15, July 2012. [7] Patrick ,Marcus, “Dynamics in Tangible Chemical Reactions,” International Conference on Chemical Engineering

(ICCE 2009), Amsterdam, Netherlands, Sep 23 - 25, 2009. [8] Shelton, B., Hedley, N., “Using Augmented Reality for Teaching Earth-Sun Relationships to Undergraduate

Geography Students,” First IEEE International Augmented Reality Toolkit Workshop, Darmstadt, Germany, 2002. [9] Juan, C., Beatrice, F., Cano, J., “An Augmented Reality System for Learning the Interior of the Human Body.

Advanced Learning Technologies,” ICALT '08. Eighth IEEE International Conference, Santander, Cantabri, pp. 186—188, 2008.

[10] A. Dünser, H. Kaufmann, K. Steinbügl, J. Glück, “Virtual and augmented reality as spatial ability training tools,” Proceedings of the 7th ACM SIGCHI New Zealand chapter’s international conference onComputer-human interaction: design centered HCI ,125-132. Christchurch, New Zealand, 2006.

[11] Duarte, M., Cardoso, A., Lamounier Jr., “Using augmented reality for teaching physics,” WRA’2005 – II Workshop on Augmented Reality. 2005.

[12] Malone, T.W, “What makes things fun to learn? A study of intrinsically motivating computer games,” (Report CIS-7). Palo Altao, CA: Xerox Palo Alto Research Center, 1980.

[13] Zhengyou Zhang, “Microsoft Kinect Sensor and Its Effect,” 2012. [14] Francesco Bellotti, Riccardo Berta, Alessandro De Gloria, and Ludovica Primavera, “Adaptive Experience Engine

for Serious Games,” IEEE TRANSACTIONS ON COMPUTATIONAL INTELLIGENCE AND AI IN GAMES, VOL. 1, NO. 4, DECEMBER 2009.

[15] http://download.cnet.com/DigenoGame-Digestive-System-Games/3000-2129_4-75439403.html#ixzz2TojjYGrd [Accessed: April 2013].

[16] http://raychambers.wordpress.com/2011/10/16/digestive-systemkinect-and-science/ [Accessed: April 2013]. [17] http://raychambers.wordpress.com/2012/04/04/kinect-mix-and-match/ [Accessed: March 2013].