a development of virtual reality game utilizing kinect ... · pdf filea development of virtual...

International Journal of Applied Engineering Research ISSN 0973-4562 Volume 11, Number 2 (2016) pp 829-833

© Research India Publications. http://www.ripublication.com

829

A Development of Virtual Reality Game utilizing Kinect, Oculus Rift

and Smartphone

DongIk Lee, KiYeol Baek, JiHyun Lee

Student, Department of Multimedia Engineering,

Andong National University, Gyeongdong-ro, Andong-si, Gyeongsangbuk-do, South Korea.

E-mail: [email protected], [email protected], [email protected]

Hankyu Lim(corresponding author), Professor, Department of Multimedia Engineering,

Andong National University, Gyeongdong-ro, Andong-si, Gyeongsangbuk-do, South Korea. E-mail: [email protected]

Abstract-

Video games such as arcade games, computer games, and

smart device games have attracted the attention of many

people, regardless of age, gender, and social status. Video

games are controlled via various types of input devices that

have emerged over time. In recent years, virtual reality (VR)

games that require VR-specific devices like a head-mounted display (HMD) have become popular. The Oculus Rift is one

of the most widely used VR HMDs. This paper describes the

design and implementation of a VR game that is operated

using three different types of device. It uses a Kinect, an

Oculus Rift, and a smartphone as input device in order to

increase user convenience and cost efficiency. The developed

VR game provides a greater degree of user control, thus

giving a competitive edge over existing games employing

only one or two input devices.

Keywords: Head-mounted Display, Oculus Rift, Kinect,

Smartphone, Virtual Reality Game

Introduction Game providers and gamers are watching with great interest

the next-generation input devices. For example, the Oculus

Rift is an upcoming HMD for virtual reality games. The

growing popularity of VR has brought about a steady increase

in VR content along with continuous innovations in VR

devices. In particular, HMD technology has attracted

significant interest from industry. Virtual Reality games that require VR-specific devices like a head-mounted display have

become popular. One of the most widely known HMDs is the

Oculus Rift, a relatively cheap and easy-to-use VR HMD

developed by Oculus VR. It is expected that VR content

combined with an HMD like the Oculus Rift will be used in a

wide variety of industrial sectors, such as video gaming, film

and media, education, sightseeing, the healthcare and medical

field, sports, and advertising. However, the majority of VR

games that have been in development over the past few years

rely on traditional controllers like a keyboard, a mouse, and a

gamepad. The expectations of fully immersive VR cannot be met by such devices, and a natural user interface using

gestures and spoken commands is required to enable users to

control and interact with objects in the virtual world.

Nevertheless, there are still some types of inputs that need to

be explicitly given via an input device, which makes the use

of wired and wireless input devices inevitable.

The KOS, a VR game developed in this work, is controlled

with a combination of a Kinect (a motion tracking input

device) and an Oculus Rift (a representative HMD product). It

also uses a smartphone to offer more diverse input methods.

The KOS was implemented using C++ and DirectX 11. The

KOS manipulates the gun aiming based on the player ’ s

movements tracked by the Kinect and utilizes smartphone's touch, swipe and drag features to play the game. The

smartphone was chosen as a supplementary input device

because of its high penetration rates. For dedicated input

devices, additional costs are necessary to purchase third party

hardware, but the majority of people nowadays already own a

smartphone. Today's smartphone incorporates various sensors

and advanced features that can be exploited to feed useful

inputs to the VR content including video games [1-5]. The

function spec and flow-chart of KOS and server are

introduced. Also the map design is showed. The VR game is

implemented by using three different types of input device, which allows players to make diverse kinds of motions to

operate the game.

Related Research 1) Kinect The KOS game was implemented using Kinect v1. The Kinect

SDK provides skeletal tracking, the capability to extract and

track the skeleton image of one or two people moving within

Kinect's field of view. A Kinect skeleton is made up of 20

body joints (head, hands, feet, hip center, etc.). Among the Kinect skeleton data (20 joints coordinates in 3D space), the

KOS game uses those corresponding to the player's right hand

in order to manipulate the gun aiming. Microsoft DirectX 11,

a collection of application programming interfaces (APIs)

used to implement the KOS, is based on a left-handed

coordinate system, whereas the Kinect uses a right-handed

coordinate system. To obtain the correct joint coordinates for

DirectX 11, the joint location information from the Kinect is

modified by multiplying -1 to the z-coordinate.



830

Figure 1: Coordinates System [2]

2) Oculus Rift

The KOS was developed using the Oculus Rift DK2, the latest

development kit for the Oculus Rift. The Oculus Rift requires

graphic rendering for two different viewpoints of the left and

right eyes. In addition, fish eye rendering is made for the left

and right screens at a significant computational cost. The use

of Oculus Rift thus increases the time of graphic rendering,

which is a big obstacle in improving graphic quality and gaming performance. This is why the implemented KOS

performs only lighting calculations with regard to the diffuse

texture and bump mapping, without introducing shadow and

blur effects. Like the Kinect, the Oculus Rift is based on a

right-handed coordinate system. Hence, Oculus Rift's head

tracking data passed to DirectX 11 that uses a left-handed

coordinate system need to be corrected to compensate the

differences [6].

Figure 2: Oculus Rift Coordinate System[3]

3) Input Devices The KOS game presented in this paper uses the smartphone as

input device. The smartphone has a number of built-in sensors, such as gyroscopes, accelerometers, proximity

sensors, ambient light sensors, and Hall sensors, that are

useful in capturing player gestures and movements. The use of

smartphone also gives an advantage that an application that

communicates with a PC can be built using tap, swipe, and

drag gestures. That is, it is possible to implement an input

device with various features for VR games without the need

for manufacturing or purchasing dedicated input devices. This

paper develops a game application that communicates with a

PC using the tap, swipe, drag, and vibration features of the

smartphone. There exist some other commercial devices for

VR games, including the Razer Hydra (a motion and

orientation detection game controller), the Myo armband (a

wearable gesture control and motion control device), Virtuix

Omni (an omnidirectional treadmill), the Emotiv Epoc (a

neuro-signal acquisition and processing headset), and Oculus Touch (a hand controller system that understands hand

gestures) [5].

Function and Implementation 1) Function Analysis

The developed KOS game can be divided into three parts: the

KOS client implementing the VR game, the smartphone as an

input device, and the server responsible for the

communication between the KOS client and the smartphone. The communication between the server and the client is

performed using Input/Output Completion Ports (IOCP). In

the smartphone, the Channel Selector is used to process

asynchronous sockets. It is possible that the KOS is

implemented using the Transmission Control Protocol/Internet

Protocol (TCP/IP) instead of IOCP. The KOS program

includes features such as head tracking and display

capabilities relying on the Oculus Rift, player motion sensing

capabilities relying on the Kinect, gun aiming manipulations,

and conversion of the smartphone packets sent by the server

to an input data format. Figure 3 shows the data flow diagram

of the KOS game.

Figure 3: DFD Structure Chart of KOS

2) Flow Chart

Figure 4 below is a flow chart of the KOS and the Server. The

Server was implemented as an IOCP Server and was designed

to wait for clients’ access when the program has started and process Packets on receipt of them according to packet

sources. Packets coming from the PC are processed by the

Server only and Packets coming from Smartphones are also

processed directly by the Server sometimes. Since the Packets

are for game manipulation in most cases, they are to be

transmitted to the KOS. When started, the KOS initializes the

System, attempts access to Clients, and processes Packets

coming from the Server to manipulate the game. The KOS is



831

configured to make the game progress in accordance with

determined logics and store the scores and time in the DB

when the game is over.

Figure 4: Flow Chart of KOS and Server

3) Map Design

The map was designed by arranging the objects in 3Ds Max as

shown in Figure 5. However, to utilize DirectX11 Instancing,

designed objects should be directly arranged in games. As

shown in Figure 6, information on the Position (XYZ) and

Rotation(Pitch(x), Yaw(y), Roll(z)) of objects was recorded

by object. Then, the objects were rearranged in the GameTool

as shown in Figure 7, extracted in the form of scenes, and

used in the implementation of InGame.

Figure 5: Map Design on 3Ds Max

Figure 6: Coordinate and Rotation Value by Object

Figure 7: Object Scene in Game Tool

4) Aiming Function utilizing Kinect

The KOS moves the gun aiming point along the x and y axes

of the screen based on the x and y coordinates of the player's

hand detected by the Kinect. To shoot, the KOS computes the

3D ray direction of the current x and y coordinates of the hand

on the screen using the camera's view and projection matrix.

The computed ray direction becomes the ballistic trajectory,

and the effects of its encounters with virtual objects

(monsters) are checked to harm the monsters [7].

5) View of Left and Right eyes of Oculus Rift, Projection

Matrix Calculation

The KOS uses Oculus Rift's head tracking information and the

camera's view and projection matrix in order to compute the

view and projection matrix for the left and right eyes [8].

Figure 8: Rendering of KOS with Oculus Rift



832

6) UI Event Processing

In the developed VR game, the Kinect and Oculus Rift

devices enable players to control and interact with objects in a

virtual world, but it is not possible to use mouses or keyboards

to play the game. The developed game has user interfaces

(UIs) on a three-dimensional (3D) plane in the 3D game world and renders a 3D scene to a texture. Button UI events are

related to harming colliding objects (monsters). When the

smartphone is tapped over the Button UI, ScreenRay is

created to determine whether any monster is hit by the

gunshot. Figure 9 and Figure 10 presents a screenshot of the

Ranking and Start Button UIs of the developed VR game.

Note that there are colliding objects shown in the screenshot.

To make them easily recognizable, the colliding objects are

marked with a red line. Figure 11 is another screenshot of the

Ranking and Start Button UIs that are viewed at a different

angle.

Figure 9: 3D UI Object With Collision OBB

Figure 20: 3D UI Object With Collision OBB

(Different angle)

Figure 31: Play KOS with Kinect, Oculus Rift, Smartphone

Conclusion The implemented VR game uses three different types of input

device, which allows players to make diverse kinds of

motions to operate the game. It also allows more various input

methods and additional game controls, maximizing the

utilization of the features provided by each device. The Kinect

and Oculus Rift devices enable players to control and interact

with objects in a virtual world, but it is not possible to use mouses or keyboards to play the VR game. The Kinect

provides fairly accurate motion sensing, but it may be difficult

to use it in a limited space due to the need for a practical

tracking area. With the Oculus Rift, the wearer's movement

range can be further restricted, which makes the use of Kinect

less effective. Apart from the Kinect v1 and the smartphone,

other types of input devices that are suitable for use in VR

content are currently under development and continue to grow

in number and diversity. One can use all of the three input

devices at the same time but it is also possible to use them

selectivelyi.e., only one or two devices best suited to the characteristics of a game are used. The developed game has

user interfaces on a three-dimensional plane in the 3D game world and renders a 3D scene to a texture. The developed VR

game provides a greater degree of user control, thus giving a

competitive edge over existing games employing only one or

two input devices.

Acknowledgement This work was supported by a grant from 2015 Seoul Accord

project(R0613-15-1148) of MISP(Ministry of Science, ICT

and Future Planning) and IITP(Institute for Information and Communication Technology Promotion). .

References

[1] Junghyun Kim, Hankyu Lim, “Game Implementation

based on Event utilizing gameUnity 3D”, 2015

Proceedings of KIPS Fall Conference, pp. 977-979,

2015.

[2] Dongik Lee, Hankyu Lim, “Virtual Reality Contents using the OculusLift and Kinect”, Proceedings of the

MCSI, pp. 102-105, 2015.

[3] Hangchan Jung, Hankyu Lim, “Kids Game Story-

telling”, 2015 Proceeding of KMMS Fall Conference,



833

pp. 416-418, 2015.

[4] Daegeun Kwon, Hankyu Lim, “Action Simulation

Game Design using the Leap Motion Controller”,

Proceeding of ICMEM 2014, pp. 381-391, 2014.

[5] Dongik Lee, Hankyu Lim, “Game Development

utilizing Several Input Devices”, Advanced Science and Technology Letters, vol. 113, pp. 134-139, 2015.

[6] Coordinate Systems (Direct3D 9), https://msdn.

microsoft.com/ko-

[7] kr/library/windows/desktop/bb204853(v=vs.85).aspx

[8] Oculus Rift Coordinate System, https://developer.

oculus.com/documentation/

a development of virtual reality game utilizing kinect ... · pdf filea development of virtual...

Documents