1

An Interactive Real World Virtual Puzzle Game using OpenCV

Fatin Nurliyana Muhamad Izhar1 and Mohd Shahrizal Sunar2

1,2Faculty of Computing, Universiti Teknologi Malaysia (UTM), 81310 Johor Bahru, Johor, Malaysia

{fatinnurliyana8@gmail.com1, shahizalsunar@gmail.com2}

Abstract. This document presents the development of an interactive puzzle mobile game application that uses physical objects to play the game. The idea to start this project is based on the current trend where children prefer to play games in smartphones and tablet. This situation leads to them having less interaction with physical objects and socialize with other people. Thus, the purpose of the project is to combine both gameplays as Real Stick, a colour-pattern matching mobile game application that using six-coloured Popsicle sticks as the physical objects. In this project, the physical sticks will be integrated with the mobile game application by implementing Color-Based Object Detection and Rotated Bounding Box techniques using OpenCV library. The process of the physical sticks’ colour detection will be done using the front camera of the device. A reflector will be placed at the front camera to allow the game to see the gameplay area in front of the device and stand to set the device to be in place. The development of the project will be done fully using Unity 3D integration with OpenCV for Unity Asset Plugin. Based on the approach to integrate the mobile game application with OpenCV library, the objective of the project, which is to study, design and develop a mobile game application that combine both game application and physical objects could be achieved.

Keywords: Interactive Mobile Game Application, Physical Objects, Color-Based Object Detection, Rotated Bounding Box, OpenCV

1. Introduction

The importance of physical props especially toys changes with the evolution of game in mobile devices. Toys do not only help with exposing children to their future, but also social and cultural harmony. Therefore, the playful interaction with the objects is necessary (Gephart, n.d). In addition, both of the play environments have their own purposes, advantages and disadvantages. Cutting off one of them equals to cutting off one of the source of learning experience. Thus, the project is proposed. Instead of choosing only one, this project combines both physical game and digital game into one.

UTM Computing ProceedingsInnovations in Computing Technology and Applications

Volume 2 | Year: 2017 | ISBN: 978-967-0194-95-0

2

The project aims to develop an interactive colour-pattern mobile game application for mobile device named as Real Stick that combines both physical objects and game application in one game. The difference between this game with other existing mobile game applications is that it requires physical interaction between the player and physical objects to play the game while the existing mobile game applications only require the player to tap the screen to play the game.

In order to allow the player to interact with the mobile game application using physical objects, the game is integrated with Color-Based Object Detection and Rotated Bounding Box techniques. This integration allows the game to identify the colour of the physical object used for the game using front camera of the device. As for this project, the physical objects used are Popsicle sticks. There are six different colours used; yellow orange, red, purple, blue and green. As shown in Figure 1, there are two additional play kits needed for the gameplay; reflector and tablet stand. The reflector is used during the gameplay to focus on gameplay area where it will recognize the colour of the Popsicle sticks. It is placed it in front of the front camera of the device. The tablet stand holds the tablet in place during the gameplay.

2. LiteratureReviewIn computer vision, object detection is meant to detect specific location and size of a

particular object in an image or a video scene. It is works by using image processing and image analysis with a number of algorithms and methods to achieve the required classification and measurement. In the following section, the related areas focus on object detection will be discussed. Referring to the article, A brief introduction to OpenCV, Culjak et al (2012) explained that Open source computer vision or more known as OpenCV is an open source library that widely been used for image and video analysis. It is designed for computational efficiency and with a strong focus on real-time application. In spite of that, it is released under BSD license, hence it is free for both academic and commercial use.

2.1 Object Detection Techniques using OpenCV OpenCV library provides a number of methods and algorithm that can be used as the techniques to detect objects and it varies based on the objects’ colors, shapes and edges. In the following sections, the techniques that related to the project to detect multiple color objects in real time is discussed.

Multi-Object Traking Method 2.1.1.

Multi-object tracking method is one of the object detection’s technique that can be used to detect an object. A study is conducted by Moon et al (2017) entitle A Comparative Study On Multi-Object Tracking Methods for Sport Events is aimed to compare and identify the best algorithm for multi-object tracking.

Figure 7: Overview of the mobile game application with physical objects

3

Kernalized Correction Filter (KCF) object tracking technique is recognized as a state-of-the-art in many fields. To be compared with Median Flow, Multi Instance Learning (MIL) and Boosting, this technique is using the object color characteristic to work on object tracking which using correlation filter without learning or model update. As the result, the KCF technique shows high success rate in even though it requires less resources that other techniques that using learning-based approach. The performance of the multi-object tracking algorithm is compared in the table below.

Kernel based method 2.1.2.

Garofalaki et al (2015) presented a development about the color object detection and object tracking by a robot car named The Chassis that using an Android device to track the motion of the object. Implementing the element of Computer Vision with OpenCV library functions, the application in the device captured the image of the object and the image will be manipulate using OpenCV library functions.

As the result, a group of white pixels that form a coherent space, is analysed to determine the largest contour which is assumed to be the desired object. The largest contour is shown with white rectangle that bound around the desired object whereas the other object of the same color is ignored.

CAMShift algorithm 2.1.3.

CAMShift algorithm or called as Continuously Adaptive Mean Shift is one of the algorithm

Figure 8: Tracking failures in tested object tracking techniques (Source: Moon et al, 2017)

Figure 9: Image processing steps: (a) Frame of the captured image; (b) Binary image after color filtering; (c) Binary image after pixel noise removal; (d) Object tracking after largest contour evaluation (Source: Garofalaki et al, 2015)

4

in provided OpenCV to detect an object in real time. It is based on mean shift algorithm which used robust method on finding mode in density distribution of data set. This method has multi-functionality in manipulating the density of the object, not only for color distribution but also for the texture and motion of the object (Veerapur and Bhat, 2013).

3. Methodology

In order to produce an interactive game application that uses physical objects as part of the game interaction in the gameplay, the game application itself needs to be integrated with object detection technique to detect the presence of the physical objects’ colors. Figure 4 shows the methodology used for the project divided to object detection process and gameplay

4. Implementation

4.1 Object Detection

The implementation of the Color-Based Object Detection and Rotated Bounding Box techniques used in the mobile game application. Both of the technique implementations will be done using the live video input from the camera of the mobile device.

4.1.1. Color-Based Object Detection technique

Figure 5: Object detection implementation testing using physical sticks’ colors through webcam

Figure 5 shows the result from the implementation of color-based object detection technique on physical sticks where the physical sticks’ is labelled correctly according to its color name with its current position. Plus, by the success of this implementation, the next implementation to create rotated bounding box around the sticks’ also can be done.

4.1.2. Rotated Bounding Box technique

Based from the result from the object detection process earlier, the next implementation is to bound a rotated bounding box around the physical sticks. The purpose of this technique is to extract the of the angle bounding box as to know the physical sticks’ angle. The rotated bounding box technique is using physical sticks’ contour as its input. The process is

5

done by applying RotatedRect function onto the contour to bound the contour with a bounding box. Then the angle of the bounding box can be access through RotatedRect’s angle properties.

4.2 Game Concept and Gameplay

Game concept and gameplay describe the implementation of OSMO gameplay with color-pattern matching game concept in the mobile game application.

4.2.1 Mobile Game Application and Physical Objects

Figure 6: The framework of the mobile game application integrated with color-based object

detection The project will use OSMO gameplay concept with 6-color Popsicle sticks to play the color-pattern matching game. Figure 6 shows the framework of the mobile game application integrated with Color-Based Object Detection technique along the overall gameplay environment.

Figure 7: Mini Detachable Magnetic Periscope Lens

This project uses a mini detachable magnetic periscope lens as the reflector. The lens contains a mirror that will reflect the environment in front of the device in 90o degree. As for the tablet stand, a multi-angle foldable stand is used to hold the tablet in place when the player plays the game. The multi-angle stand has 10 different viewing angles which it can be adjusted from 0o to 100o degree. The size of the stand is 107mm x 120mm (4.2 x 4.7inch).

6

Figure 8: Multi-Angle Foldable Stand

4.2.2 Color and Pattern Matching Game

The implementation of the concept of colour and pattern matching game in the game where it can be seen through the game modes and levels; classic and time attack mode which can be refers in section 4.3.

4.2.3 Demonstrating Interactive Mobile Game Application

Figure 9 : Gameplay real environment

Figure 9 above is the overall view of the gameplay of the game. It shows where the reflector is placed which is in front of the front-facing camera, how the stand is used to hold the device, and the gameplay area of the sticks.

4.2.4 Main Menu Interface

Figure 10 : Main Menu Interface

7

The main menu interface of the game will be shown to the player once the game application started. As shown in the Figure 10, the game tittle and interactable buttons; play, tutorial and quit button will be displayed. From here the player can choose either to start playing the game, watch the animated tutorial or quit the game.

4.2.5 Game Tutorial Interface

Player can view this animated tutorial from the main menu screen using the tutorial button. In tutorial panel, there will be 3-step animated tutorial played for each step. The steps provided will guide the player on how to setup the gameplay environment using the reflector and tablet stand. Plus, the tutorial also will show the player how to play the game using the physical sticks. It is visualized as the real gameplay environment with hand gestures to help the user understand the gameplay better.

Figure 11: 3-Step tutorial animation (a) Step 1 (b) Step 2 (c) Step 3

4.2.6 Game Level Interface

Figure 12 : Gameplay Interfaces (a) Classic Mode (b) Time Attack Mode

Figure 12 above is the gameplay interface of the game modes; Classic and Time Attack mode of the mobile game application. When either of these game mode is selected by the player, the process of the object detection will be initiated upon the gameplay started. The following figure shows the different states of the stick during the object detection process using physical stick’s color. The figure shown is taken based on the Classic Mode Level 1. The same output will be shown for the other physical sticks’ color in Time Attack Mode and the other levels.

8

Figure 13 : Initial state of the yellow stick in the pattern

Through the levels, different sticks’ pattern with different sticks’ color will be shown. The initial state of each stick upon the level start is as shown in Figure 13 where the solid stick would have an outline around the stick. When this stick is shown during the gameplay, its indicate that the game could not detect any stick’s with yellow color within the gameplay area.

Figure 14: Hint stick in the pattern

As been mentioned before, there will be a hint button at the top right of the interface. This function can be used by the player to enable a hint stick to guide the player to match the same angle of the physical sticks with the sticks’ pattern. As shown in the Figure 14, when the hint feature is enabled, then the solid sand hint stick will be shown and its angle is representing the angle of the same color physical stick placed in front of the device. While, if the hint feature is disable then only the solid stick is enable indicates the existence of physical stick of same color as in the stick pattern found in front of the device. Through the gameplay, when the player be able to match the physical stick with the stick pattern’s angle of same stick’s color, then the glow stick will be enable to show that the win condition is met.

Figure 15: Glow stick in the stick pattern

9

4.2.7 Physical Stick Tracking Interface

Figure 16: Physical sticks tracking

This interface will be shown in the game when the player succeeds to finish a level. The purpose of this interface is to make sure that the player clears the space by remove all the physical sticks exist within the gameplay area before proceed to the next level. In order to tell the player to clear the space, a hand gesture animation will be play and stop when the space is cleared. Then, the game will directly display the pattern in the next level.

5. Testing and Results

The evaluation and result from the testing done on the mobile game application will cover the Performance Testing, User Acceptance Testing as well as Usability Testing.

5.1 Performance Testing

In order to obtain the optimum level of specifications to run this mobile game application on mobile devices, it is tested on several devices with different hardware and software specifications. Table 1 shows the specifications of the devices used to run this project.

Table 1: Testing devices and it specifications

Properties Device 1 Device 2 Device 3 Name Asus Zenpad 7.0 Lenovo Tab3 7 Lenovo A859 Platform Mobile Mobile Mobile Android Versions 5.0 6.0 4.2

Processor Intel® Atom™ x3-C3230 Quad-Core,

64bit

MediaTek™ 1.3 GHz Quad-Core,

64-bit

Mediatek™ Quad-core 1.3 GHz

Cortex-A7

Camera (MP) Front: 2 MP Rear: 8 MP

Front: 2 MP Rear: 5 MP

Front: 1.6 MP Rear: 8 MP

Screen Resolution (pixels) 1280 × 800 1024 × 600 1280 × 720

The performance of each device is measured by comparing the frame rates to finish the defined tasks.

10

Figure 17 : Result of the devices performance testing

In order to obtain optimum level of specifications to run this mobile game application on mobile devices, few devices are tested based on their different specifications. The trend of the performance result of the three devices depict the same pattern for every defined task. The device performance is measure based on the frame rate to load a scene. Most of the tasks that load normal scene behaviour took around 60fps with 15ms. While when the devices load level scene during the gameplay, the devices show the decline in the frame rate values. It may cause by the physical stick detection process occurred during the gameplay as it need much power to process the live video input to process the input.

5.2 User Acceptance Testing

User acceptance testing (UAT) is a testing done to evaluate the mobile game application requirements; functional and non-functional requirements for user experience. It is tested based on the requirements defined during the design phase. The test is conducted using Black Box Testing to identify errors and irrelevant function at through the interface in the mobile game application. Based on the results, there are no bugs found during the testing. All of the events show result as the expected result.

5.3 Usability Testing

Usability testing is the another test conducted for the project. The purpose of the test is to identify usability problems and determine the user’s satisfaction about the mobile game application. Another reasons are to observe the problems encounter and experience confusion by the user when playing the mobile game application.

The testing is done by giving questionnaires to the selected 10 users; 5 novice users that are also the target user of the game which is children between 2 to 6 years old and 5 non-novice users who have basic knowledge background related to the computer science field. As for the novice users, the questionnaire is filled by the kindergarten teachers during their observation on the children performance’. The teachers’ guide them to play the game through the evaluation session. Briefly before the testing session, the teachers are briefed about the gameplay and the scope of the testing.

In general, the users also commented on the concept of the game that they think the concept is fun, interesting and quite addictive. One of the novice user state that the game would be more fun and exciting if the game can be play in the bigger screen such as PC or television.

11

Finally, most of the users also suggest that the game could be improve by enhance the accuracy of the physical sticks detection during the gameplay.

6. Conclusion

In conclusion, the aim of the project to develop an interactive color-pattern matching mobile game application using physical objects is accomplished. It is done by using color-based detection technique of the physical sticks’ color and rotated bounding box technique to get the angle of the phyical sticks rotated by the player. Besides that, the project is tested on its performance, user acceptance, usability testing. Based on the results, it is said that the performance of the device during the object detection process is depends on the device’s specifications. All of the requirements; functionality and non-functionality of the project result as the expected outcome. The response from users during the usability testing also stated that they are satisfied with the mobile game application and suggest that the game would be more better by enhancing the accuracy of the object detection process.

References

1. Culjak, I., Abram, D., Pribanic, T., Dzapo, H. and Cifrek, M., 2012. A brief introduction to OpenCV. MIPRO 2012, pp.1725-1730.

2. Garofalaki, Z., Amorginos, J. and Ellinas, J., 2015. Object Motion Tracking Based On Color Detection for Android Devices. International Journal of Computer, Electrical, Automation, Control and Information Engineering, 9(4), pp.962-965.

3. Gephart, S.M., n.d, Traditional Toys in the Age of Digital Technology: Applying Symbolic Interactionism to Children at Play.

4. Radesky, J.S., Schumacher, J. and Zuckerman, B., 2015. Mobile and interactive media use by young children: the good, the bad, and the unknown. Pediatrics, 135(1), pp.1-3.

5. S. Moon, J. Lee, D. Nam, H. Kim and W. Kim, 2017. A comparative study on multi-object tracking methods for sports events. 19th International Conference on Advanced Communication Technology (ICACT), Bongpyeong, 2017, pp. 883-885.

6. Veerapur, K. and Bhat, G., 2013. Colour Object Tracking On Embedded Platform Using Open CV. International Journal of Recent Technology and Engineering (IJRTE), 2(3)