Head Tracking for 3D Games:Technology Evaluation using CryENGINE2 and faceAPI

Andrei Sherstyuk∗

Sunflower Research Labs

Anton Treskunov†

Samsung

ABSTRACT

Mouse, joystick and keyboard controls in 3D games have longsince become a second nature for generations of gamers. Recentadvances in webcam-based tracking technologies made it easy tobring natural human motions and gestures into play. However,video tracking is a CPU-intensive process, which may have a nega-tive impact on game performance.

We measured this impact for different types of 3D content andfound it to be minimal, on multi-core platforms. We provide detailson implementation and evaluation of our system. Also, we suggestseveral examples how natural motion can be used in 3D games.

1 INTRODUCTION

Bringing natural human motion into a 3D interactive environmentrequires balancing hardware costs, fidelity of tracking and compu-tational resources available. Recently introduced webcam-basedtracking solutions are capable of delivering high-quality motiondata on consumer hardware, practically on every desktop. That, inturn, allows to integrate natural motion into 3D games easily. How-ever, most high-end game engines are computationally intensive,often pushing the host systems to their limits.

In this technology evaluation, we investigate whether video-based head tracking is practical and computationally affordable for3D games. We approach this problem by building and testing agame prototype, integrating a high-end photorealistic game plat-form CryENGINE2 from Crytek [2] with a state-of-the-art faceAPIhead tracking software from Seeing Machines [6]. Both systemsare briefly described below.

CryENGINE2 made its debut with Crysis game, which set newstandards of visual quality for first person shooters. CryENGINE2was used by creators of Blue Mars world [1], featuring highly re-alistic avatars, with dynamically simulated hair and layers of cloth.The power of CryENGINE2 is behind the success of Entropia Uni-verse adventure game, which holds the record of hosting the mostexpensive piece of virtual estate ever purchased with real money.

FaceAPI tracking engine [6] is rapidly gaining recognition inboth academic and gaming communities. It was used in studieson estimating gaze direction from eye appearance, with free userhead rotations [3]. Marks et al investigated optimal operating con-ditions for faceAPI [4]. FaceAPI was used in a shared virtualtraining environment, emphasizing non-verbal communications be-tween users [5]. There are reports on integration of faceAPI withUnity3D and Valve engines [7]. However, as of this writing, thereis no published work on evaluating faceAPI with “heavy-weight”gaming platforms, such as CryENGINE2, Unreal or Unigine. Thus,the question whether faceAPI is an affordable addition to high-quality games remains open.

In this work, we provide experimental evidence that the answerto this question is positive.

∗e-mail: andrei.sherstyuk@gmail.com†e-mail:anton.t@sisa.samsung.com

2 THE SYSTEM IMPLEMENTATION

We used a non-commercial version of faceAPI, which tracks posi-tion and rotation of user head at 30 Hz. For gaming environment,we used Blue Mars Sandbox Editor from Blue Mars SDK, builtupon CryENGINE2 and used with permission from Avatar-RealityInc, the makers of Blue Mars Online world [1].

The head motion data exchange between the Editor and faceAPImodule was implemented and tested in two configurations: (1)faceAPI running as a server and communicating with the Editorvia datagram socket; (2) faceAPI compiled into the Editor directly.The latter solution appeared more convenient and allowed faster ini-tialization of tracking: 1-2 seconds as compared to 2-3 seconds incase (1). The client/server configuration was initially intended fornetwork setup, anticipating rendering to be slow with the trackerrunning on the same computer.

The prototype was tested on two computers, running Windows 7OS. Each computer had 4 GB RAM, nVidia GeForce graphics cardand an integrated web camera. Other specifications:

Computer CPU Graphics(1) Sony Vaio CW Intel Duo 2.53 GHz GT 230M, 512 MB(2) Samsung RF510 Intel i7 1.60 GHz GT 330M, 1 GB

As indicated, both systems have relatively modest hardware,which made them sensitive to variations in computational load.This turned out to be an advantage, allowing for detecting and mea-suring changes in game engine performance due to tracking.

3 IMPACT OF TRACKING ON GAME ENGINE PERFORMANCE

The preliminary tests were conducted on scenes with mixed con-tent, that included both static objects and dynamically deformedobjects, such as user avatars. One of those scenes is shown in Fig-ure 1. The results showed that the impact of tracking on renderingspeed was negligible. In simple scenes that were rendered fasterthan 30 FPS, frame rate dropped 10% each time when faceAPI wasstarted, and remained low until the user face was acquired by theengine (1-2 seconds). Then, FPS recovered to its initial value. Incomplex scenes (over 1.5 M triangles, FPS< 30), starting and stop-ping the tracking engine had no noticeable effect on FPS.

Figure 1: Test scene with mixed content: 3rd person view on useravatar at Welcome Area Dance Floor in Blue Mars 3D social world.Triangle count 1.51 M, update rate 30 FPS, on both systems. Theavatar head is tilted back and sideways, controlled by user head ro-tation. Tracking has no impact on FPS for this scene.

67

IEEE Virtual Reality 201316 - 20 March, Orlando, FL, USA978-1-4673-4796-9/13/$31.00 ©2013 IEEE

Figure 2: Dynamic test scene with 20 animated avatars.

To investigate further, we measured the system performance inscenes with separate types of content: static meshes that use littleof CPU time and dynamic objects that require processing on CPU.For that purpose, we used the standard avatar model from BlueMars SDK, with cloth and hair simulated by mass-spring models.In static tests scenes, simulation was turned off and avatars wererendered as static meshes. In dynamic tests, the avatars were run-ning the “standing-idle” animation, with cloth and hair deformedon CPU at every frame, as illustrated in Figure 2. For static and dy-namic modes, the scene size was varied from 1 to 20 avatar objects.Each scene was rendered twice, with and without head tracking.The test results are plotted in Figure 3 and listed in Table 1.

The test results allow us to draw two conclusions.

• For static 3D content with GPU-bound rendering, tracking haszero impact on performance. As Table 1 shows, frame rateslowly decays from 76 to 66 FPS (81 to 68, for system 2), asthe number of objects increases from 1 to 20. These valuesremain unchanged, when tracking is turned on and off.

• When the content is predominantly dynamic, performance islimited by CPU. In this case, tracking has measurable effecton rendering speed, with mean slowdown of 2.6 and 3.7 FPS,for systems 1 and 2, respectively. On both systems, the costof tracking does not seem to depend on the scene size.

4 DISCUSSION AND CONCLUSIONS

To the best of our knowledge, this project presents the first study onintegrating faceAPI with CryENGINE2, and the results are very en-couraging. In our experimental settings with multi-core platforms,the impact of motion tracking varied from non-existing to low, in-cluding the worst case scenario with all-dynamic scene content.

We can conclude that motion tracking is not only practical, butalso enabling technology for 3D games. For instance, it is easyto imagine how head movements can enhance players interactionin social 3D worlds, when their avatars are in close proximity toeach other. Nodding, head shaking and more subtle uses of bodylanguage, such as gaze averting or seeking eye contact, becomeavailable. Figure 4 shows one example. Head motions can also berecorded and played back with a single keystroke, for convenience.These new interfaces constitute rich ground for further research.

REFERENCES

[1] Blue Mars Online. www.bluemars.com.[2] CryTek. www.cryteck.com.[3] Y. S. Feng Lu, Takahiro Okabe and Y. Sato. A head pose-free approach

for appearance-based gaze estimation. In Proceedings of the BritishMachine Vision Conference, pages 126.1–126.11. BMVA Press, 2011.

[4] S. Marks, J. A. Windsor, and B. Wunsche. Optimisation and compari-son framework for monocular camera-based face tracking. In Proceed-ings of IVCNZ’09 24th International Conference, 2009.

[5] S. Marks, J. A. Windsor, and B. Wunsche. Head tracking based avatarcontrol for virtual environment teamwork training. In In Proceedingsof GRAPP 2011, 2011.

[6] Seeing Machines. http://www.seeingmachines.com/product/faceapi/.version 3.2.6, September 2010.

[7] T. Sko and H. J. Gardner. Head tracking in first-person games: Inter-action using a web-camera. In International Conference on Human-Computer Interaction, pages 342–355, Berlin, Heidelberg, 2009.

Table 1: System performance, with and without tracking, for sceneswith static and deformable objects. FPS values for static scenes arenot affected by tracking on both systems.

objects : tris performance, FPSsystem 1 system 2

static dynamic static dynamictracking off / on tracking off / on

1 : 0.13 M 76 73 / 73 81 76 / 762 : 0.25 M 73 71 / 71 79 72 / 723 : 0.37 M 72 56 / 55 77 71 / 704 : 0.49 M 71 48 / 44 76 69 / 655 : 0.61 M 71 40 / 36 75 60 / 556 : 0.73 M 70 33 / 32 74 50 / 457 : 0.85 M 70 30 / 27 73 44 / 398 : 0.97 M 69 27 / 24 72 40 / 369 : 1.09 M 69 25 / 19 72 35 / 30

10 : 1.21 M 69 20 / 14 71 31 / 2811 : 1.33 M 68 16 / 12 70 29 / 2412 : 1.45 M 68 13 / 9 70 25 / 2013 : 1.57 M 68 9 / 7 69 23 / 1814 : 1.69 M 67 8 / 6 69 20 / 1515 : 1.81 M 67 8 / 5 69 16 / 1216 : 1.93 M 67 7 / 4 69 15 / 1117 : 2.05 M 66 6 / 4 68 14 / 1118 : 2.17 M 66 5 / 3 68 14 / 1019 : 2.29 M 66 5 / 3 68 13 / 920 : 2.40 M 66 4 / 3 68 11 / 8

5 10 15 20

020

4060

80

5 10 15 20

020

4060

80

System 1 System 2

Number of avatars

Fra

mes

per

sec

ond

tracking offtracking on

Figure 3: Impact of tracking on rendering dynamic objects, with allmeshes rebuilt for every frame on CPU (see Figure 2). Tracking startsto become noticeable in scenes with more than 2 avatars, at averagecosts of 2.6 and 3.7 FPS.

Figure 4: Adding personality in close interactions, with free-formhead rotation and fixed gaze direction.

68