An AR Edutainment System Supporting Bone Anatomy LearningPhilipp Stefan∗†‡,§ Patrick Wucherer†‡§ Yuji Oyamada†¶ Meng Ma‡ Alexander Schoch‡

Motoko Kanegae‖ Naoki Shimizu‖ Tatsuya Kodera‖ Sebastien Callier‖ Matthias Weigl∗∗

Maki Sugimoto‖ Pascal Fallavollita‡ Hideo Saito‖ Nassir Navab‡

ABSTRACT

We present a medical Augmented Reality (AR) edutainment systemfor bone anatomy learning. This learning environment, called ARbone puzzle, is a metaphor for bone anatomy learning with AR vi-sualization and intuitive interaction. AR bone puzzle uses its user’sbody as a puzzle frame and computer generated virtual bones aspuzzle pieces. Users learn bone anatomy by assembling the virtualbone pieces on their body. Key features of this system are 3D ARvisualization and intuitive gesture based user interaction.

Index Terms: H.5.1 [Multimedia Information Systems]: Artifi-cial, augmented, and virtual realities; I.2.1 [Applications and Ex-pert Systems]: Medicine and science

1 INTRODUCTION

Anatomy learning is one of the most important and challengingtasks for undergraduate medical students. Anatomy atlases and ca-daveric dissection are used in traditional learning approaches. Theatlas is used to study the basic knowledge of human anatomy. Incadaveric dissection cadavers are used to teach.The atlas is used tolearn the theory while cadaveric dissection tends more to practicallearning.

Over the recent years, anatomy teaching has been significantlyreduced regarding the content delivered and time allocated to thissubject [5, 7].Furthermore, the shortness of staff in anatomy that isqualified to teach is another serious problem. Thus, there has beenan increasing need to develop anatomical teaching and training aidsboth for the educators and the students [1].

The concept of Augmented Reality (AR) has been introduced foranatomy learning. The VR 3D jig-saw puzzle [8] is a VR educationsystem that is based on the metaphor of a jig-saw puzzle. Their tar-get objects are composed muscles, ligaments, and bones of a foot.Their users learn the structure by solving a virtual 3D puzzle gamewith a haptic device so that they can learn the spatial relation and theappearance of the anatomy. Both of the magic mirror [2, 6] and theSAR atlas [4] are AR education systems that visualize virtual ob-jects on their user’s body by gesture based interaction. The magicmirror overlays virtual internal organs on user’s body shown on alarge display like a mirror. The SAR atlas directly overlays the vir-tual objects, which are internal organs, some bones, and muscle, onuser’s body by using a projector. Furthermore, the virtual contentsare dynamic objects, e.g., beating heart and breathing lung. The

∗Corresponding author email: stefanp@in.tum.de†Three authors equally contributed to this paper.‡Chair for Computer Aided Medical Procedure (CAMP), Technische

Universitat Munchen§Chirurgische Klinik und Poliklinik, Ludwig-Maximilians-Universitat

Munchen¶School of Fundamental Science and Engineering, Waseda University‖The Graduate School for Science and Technology, Keio University∗∗Institut und Poliklinik fur Arbeits-, Sozial- und Umweltmedizin,

Ludwig-Maximilians-Universitat Munchen

(a) System setup (b) Mirror like AR visualization

Figure 1: The proposed AR bone puzzle: (a) Hardware setup. (b)Mirror-like AR visualization: (Left) AR window and (Right) refer-ence window.

AR bone puzzle follows these works where visualization is shownon the user’s body within a mirror-like display.

2 AUGMENTED REALITY BONE PUZZLE

This paper aims to support medical students and young children tolearn human bone anatomy. We propose an AR edutainment sys-tem, called AR bone puzzle, for achieving this purpose. The ARbone puzzle is a metaphor for bone anatomy learning with AR vi-sualization and intuitive interaction. The AR bone puzzle uses itsuser’s body as a puzzle frame and virtual bones, which are gener-ated by Computer Graphics, as puzzle pieces. In the game, wholeskeleton is decomposed into 16 sub groups and each sub group isgiven a name, e.g., thorax and humerus. A user assembles a com-plete skeleton by putting the virtual bone pieces on his/her ownbody. The user learns bone anatomy through the puzzle game.

A sketch of our hardware set-up is shown on Fig.1(a). A largevideo display, which is put in front of a user, is used for visu-alization. A calibrated color-depth camera pair, which is locatedabove/below the display, is used for both visualization and interac-tion between the system and the user. In our proposed system, aKinect is used as the calibrated color and depth camera pair.

Figure 1(b) shows the visualization shown on the video displayduring the game. The video display separates its window into twoparts, AR window on the left side and reference window on the rightside. The AR window is the main working place where the userplays the puzzle game. The AR window shows video stream cap-tured by the color camera like a mirror and overlays virtual boneson the user’s body for the puzzle game. On the other hand, the refer-ence window shows learning materials and visual hints for support-ing the puzzle game. More detail of the visualization is describedin the latter part of this section.

2.1 Main puzzle game

The AR bone puzzle starts with system initialization and then runs amain puzzle game. The initialization asks the user to show a Kinectcalibration pose to measure his/her body size. The virtual bonepieces are resized with respect to the measured body size so thatoverlaid virtual bones are well-fitted to the user’s body. The mainpuzzle game takes three steps for each bone and the user iterates thethree steps until the entire skeleton is completed.

113

IEEE Virtual Reality 201429 March - 2 April, Minneapolis, Minnesota, USA978-1-4799-2871-2/14/$31.00 ©2014 IEEE

(a) Select a new bone (b) Grasp the selected bone (c) Fit the grasped bone

Figure 2: The three steps for fitting a bone.

Figure 2 shows a sketch of each step. The first step randomlyselects a new bone. The reference window shows a zoomed-up ofthe selected bone and rotates it ± 20 degrees about the vertical axisto provide 3D perception [3]. At same time, the AR window showsthe name of the bone and the remaining area in the AR window iscovered by a black veil to highlight the name. In this way, clut-tered background will not disturb the user’s attention as shown inFig.2(a). The second step is to grasp the bone piece as shown inFig.2(b). Voice guidance asks the user to raise his/her hand whenhe/she is ready to start solving a new bone and the selected bone isstuck to the raised hand. Once the bone is stuck to the user’s hand,the black veil in the AR window is removed to start bone fitting andthe reference window changes the content to showcase the entireskeleton. Since the entire skeleton is shown in the reference win-dow, the user can refer to the skeleton to find out where the graspedbone is to be fitted. The final step is to fit the grasped bone to thecorresponding body part as shown in Fig.2(c). This “positioning”gesture is regarded as a trigger for judging the bone fitting. Whenthe positioning gesture is recognized, the system measures the dis-tance between the hand position and the center of the correspond-ing body part in 3D space. If the measured distance is below somethreshold, the bone is then affixed correctly on the correspondingbody part.

2.2 AR visualization

AR visualization in the AR bone puzzle has two features: mirror-like display and visual hints. As a way of visualization, AR bonepuzzle adopts a mirror-like display, which is proposed by the magicmirror system [2, 6]. Via the mirror-like display, users can easilyunderstand the geometric relation between the virtual bones over-laid on their body and real bones. The other feature is that the ARbone puzzle gives the users two different visual hints for solvingthe puzzle game during step 3. The first hint is displaying the entireskeleton in the reference window. The bone of interest in the entireskeleton rotates in the reference window so that the user can dis-tinguish where the bone should be placed. The other hint is shownin the AR window. The system synchronizes the virtual bone withthe motion of its corresponding body part, e.g., “Columna Verte-bralis” is synchronized with corresponding body trunk. Therefore,the user can find out the corresponding body part by moving his/herbody part individually.

2.3 Gesture based interaction

Interaction in the AR bone puzzle includes grasping a virtual bone,moving the grasped bone, and fitting the bone to the right position.Most intuitive gestures representing grasping/releasing the virtualbones might be closing/opening a hand. We have tested all gesturesthat are implemented in OpenNI, and our experience tells us thatonly hand waving and hand pushing gestures are accurately recog-nized and that hand position is accurately tracked. Thus, we havedecided to use a raising hand gesture for grasping bones and a handpushing gesture for fitting bones.

3 FEEDBACK FROM USER STUDY

We have conducted a user study with non-medical students, med-ical students, and an anatomy professor. Here, we mention onlyfeedbacks from the participants due to lack of space.

An anatomy professor suggested us to make the level of detailcontrollable. The proposed system decomposes the entire skeletoninto 16 parts as mentioned above. This decomposition is fine forjunior students but not enough for senior students. Controllablelevel of detail lets users adjust the level of difficulty with respect totheir knowledge.

4 CONCLUSION

This paper proposes an AR edutainment system, so called AR bonepuzzle, for supporting people in learning human bone anatomy.This is the first system aimed for bone anatomy learning to the bestof our knowledge. The AR bone puzzle has two key features: ARvisualization and intuitive gesture based interaction. The ability todirectly augment the user’s body with virtual anatomy and the intu-itive interaction with it make this approach particularly appealing.We hope to continue improving our system and together with clin-icians and anatomy educators, bring its usability in medical class-rooms one day.

ACKNOWLEDGEMENTS

This research was partially supported by the Strategic Young Re-searcher Overseas Visits Program for Accelerating Brain Circula-tion of Japan Society for the Promotion of Science, G2308.

REFERENCES

[1] K. W. S. Ashwell and P. Halasz. An acrobat-based program for grossanatomy revision. Medical Education, 38(11):1185–1186, 2004.

[2] T. Blum, V. Kleeberger, C. Bichlmeier, and N. Navab. mirracle: AnAugmented Reality Magic Mirror System for Anatomy Education. InIEEE Virtual Reality, 2012.

[3] A. X. Garg, G. R. Norman, K. W. Eva, L. Spero, and S. Sharan. Is thereany real virtue of virtual reality?: the minor role of multiple orientationsin learning anatomy from computers. Academic medicine: journal ofthe Association of American Medical Colleges, 77(10):S97–S99, 2002.

[4] A. S. Johnson and Y. Sun. Exploration of spatial augmented reality onperson. In IEEE Virtual Reality, 2013.

[5] J. Kerby, Z. N. Shukur, and J. Shalhoub. The relationships betweenlearning outcomes and methods of teaching anatomy as perceived bymedical students. Clinical Anatomy, 24(4):489–497, 2011.

[6] M. Ma, P. Fallavollita, T. Blum, U. Eck, C. Sandor, S. Weidert,J. Waschke, and N. Navab. Kinect for interactive ar anatomy learning.In International Symposium on Mixed and Augmented Reality (ISMAR),2013.

[7] K. Patel and B. Moxham. The relationships between learning out-comes and methods of teaching anatomy as perceived by professionalanatomists. Clinical Anatomy, 21(2):182–189, 2008.

[8] B. Preim, F. Ritter, and O. Deussen. A 3d puzzle for learninganatomy. In International Conference on Medical Image Computingand Computer-Assisted Intervention (MICCAI), pages 750–757, 1999.

114