Gesture phonesKrishna Kumar.S, Sai Venkata Vinay .C

1st year- ECE,Sona College of Technology, Salem, Tamil Nadu.

Abstract: There are thousands of cell phone models available in the market, all with varying features in recent years, cell phones have also seen considerable progress in terms of user interaction, evolving all the way from QWERTY keypad to touch screens, and the next transition in user interface is gesture recognitions on Smartphone, such phones are termed as gesture phones. This document comprises of gesture control technology which explains how a Smartphone understands gestures. The scenarios where smart phones are operated by gesture recognition on the day to day basis are also explained. They are adding a whole new dimension to multimedia via touch free technology. Mobile operating like android and windows 8 and so on use gesture recognition with their own features. With that, this new experience can be offered to broader audiences on a wider range of devices. Gesture phones are enforcing many innovations on because of its advantages to

mankind. But it’s not so simple for it has many several challenges which remain a mystery for gesture recognition. However, these problems can be overcome with different tracking and new technologies with good reasoning. This paper precisely includes the innovations that can be implemented for more efficient use of gesture phones.

1. How a SmartphoneUnderstands gestures

Gesture recognition is done through a few types, among them a few are optical tracking, inertial tracking, and calibration. The device understands by computing the position of the gesture, its orientation, acceleration and also its angular speed. The information collected is processed and finally the gesture is recognized and the necessary response is given. Fig. 1 explains clearly the model of recognition of gestures

Fig.1: Prototype of gesture phones[1]

on Smart phones.

1.1 Optical Tracking

The gestures are tracked by analyzing the variations in the color of each and every pixel recorded through one or more cameras. The different movement is recognized by the camera and compares it with saved gestures while the phone is in a steady position and it gives vibration. This buzz indicates that the movement has been recognized. The gesture is saved and implemented to carry out the corresponding task(s) you have attached to its profile.[3]

There are three common technologies that can acquire optical tracking that is 3-d images, each with its own advantages. Stereoscopic vision, structured light pattern and time of flight (TOF) among which stereoscopic vision is ideal for Smartphone, tablets and other consumer devices. The analysis of these 3-d images brings 3-d technology into reality.[4]

1.1.1 Stereoscopic vision

It uses two cameras to obtain a left and right stereo image. As the computer compares the two different images, it develops a disparity image that computes the displacement of the objects in the images.

The views from each camera are different; it becomes possible to determine the distance to scene points by triangulation. Essentially, if we can identify a pair of pixels that image the same 3D point, one in the left camera image and the other in the right image, then the 3D coordinates of the point can be

calculated as the intersection of light rays from that point to each of the pixels.

Fig.2: a rock on the surface of Mars, as imaged by a stereo pair on one of the Mars Exploration Rovers. Both the original (typically left camera) intensity image and a false-color “elevation map” image are shown.[1]

1.1.2 Pros and cons of stereoscopic vision technology

Stereoscopic vision technology requires amount of software complexity for highly precise 3D depth data that can be processed and analyzed in real time by digital signal processors (DSPs).[1] Errors in camera alignment as well as insufficient lighting leads to significant errors in the calculation of depth values. Precise camera synchronization is required if the object is in motion.

Advantages of the sensors are the use of well established hardware (2D-cameras) with a wide range of capabilities. Cost effective and fit in a small form factor, making them a suitable for Smartphone, tablets and other consumer devices. It

cannot deliver high accuracy and response time.

1.1.3 Triangulation

The coordinates and distance to a point can be found by calculating the length of one side of a triangle, given measurements of angles and sides of the triangles formed by that point and two other known reference points.

Fig.3: two cameras focusing on the same point

1.1.3.1 Calculation:

l= dtanα

+ dtanβ

Therefore,

1d=1l( 1tanα

+ 1tanβ

)

Using the trigonometric identities,

tanα= sin αcos α

and

sin (α+β )=sinα cos β+sin βcos α

This is equivalent to:

1d=

sin (α+β )l sinα sin β

From this way, it is easy to determine the distance of the unknown point from observation point, its north or south and east or west offsets from the observation point, finally its full coordinates.

1.2 Inertial Tracking

The motion of Smartphone and similar devices is tracked with high precision, without depending on external structure or prior knowledge of the environment of usage.

The camera was assumed to have a global shutter, where all the pixels in an image are captured at the same time. But cameras for inertial tracking use CMOS sensors with rolling shutters, capturing each row of pixels at slightly different time instant. By this way the variation in the position of the pixels in comparison with the image with normal shutters, the Smartphone understands its own motions.[2]

These motions are recognized as inputs for which the response is to perform some task.

Fig.4: Example of an image taken using rolling shutters.[2]

Fig.5: These motions are recognized as inputs for which the response is to perform some task. Different movements are saved and each of them can perform functions.

2. Cases of Gesture Recognition Technology

Gesture recognition technology would be widely spread and used in future on mobile phones as a common feature in them. A few examples of cases, which are enhancing the user experience compared to what is available today, are listed below as follows:

1. We can answer or unanswered an incoming call with a wave of the hand while driving and can make outgoing calls similarly. This feature is termed as call control. This comes under optical tracking

2. We can skip tracks on your media player while listening songs using simple hand motions either optically or inertial or two dimensionally.

3. We can even operate the other devices without using their remote control but by using gesture phones.

4. We can turn down or increase the volume phone or switch off the mobile through it.

5. The most interesting feature at future on mobile phones would be browsing WebPages with the hand movements without operating a phone

6. We can scroll web pages, or within an eBook with simple left and right hand gestures.

7. We can switch off the mobile phone, mute it or can change to vibration mode through gestures.

8. Another interesting use case is when using the Smartphone as a media hub; a user can dock the device to the TV and watch content from the device- while controlling the content in a touch-free manner from a far position.

Amongst the above mentioned cases, a few are available in the market as a feature in phones that come under the labels like iPhone, Samsung, micromax, Kinect, Lava, QUALCOMM, etc. These mentioned cases are ideal when touching the device is a barrier such as when hands are wet, with gloves, dirty etc. (we all are familiar with

the annoying smudges on the screen from touching). Moreover, these are done either by two dimension gesture recognition or optically, in an inertial way.

3. Applications of gesture technology on android and windows phones

The few above mentioned cases are inbuilt features in mobile phones but a few are not inbuilt in them. In order to overcome this problems and give more emphasis on gesture tool. Software developers have developed and coined some apps which can be operated on mobile OS’s. The apps which are brilliant in use and usage are listed below but for them to operate we need front view camera at most one or sensors in them. Of course, all most all smart phones have them. Point Grab’s [5] solution is ideal for use with tablets and smartphones in two primary scenarios: when operating the device closely without the possibility of physically touching the screen (i.e. hands are not clean), or when the user wishes to watch the device content on a large screen that is connected to the device. In both cases, Point Grab’s solution brings next generation hand gesture technology with gestural interfaces to a diverse range of applications. Air Swiper is one of the most feature rich apps in this set. This application allows you to work more effectively and faster with your phone. It has a number of features including unlocking and locking the device, controlling SMS and sound control. Hovering Controls is one other such app which costs $1.34 and you can set it up to launch predetermined apps by hovering over

the sensor, swiping once, or swiping twice. Some of the things you can do with this app include silencing an alarm, controlling media playback like videos or photos with just gestures. [9]

3.1 Modes of two dimensional app

The app has two modes. The Target mode which opens a pre-determined app with a set gesture and the carousel mode, which allows you to select from a set list of apps. Scribble to open any app-Google Gesture Search is an app that has been around for some time and even though it requires you to actually touch the screen it is a really useful app. The app allows users to quickly access contacts, applications, settings, music and bookmarks on your Android device by just drawing letters or numbers on the screen. The app is self- learning and becomes more useful the more you use it.

4. Benefits of gesture technology

As gesture technology is a high level processing vision algorithm. There are various benefits available for this gesture architecture and these benefits helped the researchers to coin many innovations and develop applications. These benefits are:

1. This gesture technology enables a flexible programming environment to the user.

2. It is very user friendly as it provides multiple tasking, multiple programming, multiple sharing, multiple instructions, and multiple data transmission.

3. These gestures help in paralysing the data and task with throughput which includes fast prototyping and optimisation.

4. Moreover, this gesture phone would be highly portable than any other device.

As there are benefits, cons would also be there running back of it.

5. Challenges of gesture technology

1. The entire gesture technology depends on compiler’s performance if he is slow, then efficiency in fast prototyping would decrease and may lead to misconception in programming for what we program is what we get. 2. Optical tracking consumes battery highly due to high clock cycle so it affects the fair performance. 3. If the inter processor is of low standard then memory access would be inefficient. 4. Optical tracking requires a camera and with good camera resolution. More over this may require good consistent lighting. Items in the background or distinct features of the users may make recognition more difficult. 5. In periods of prolonged use of hand movements, may lead to disease called gorilla arm[6] that is users’ arms began to feel fatigue and/or discomfort.

These cons can be overcome but it requires ultrasonic waves, which is called as ultrasonic tracking. This feature in mobile phones would be the future innovation.

6. Innovations:

Ultrasonic waves[7][8] can be used for gesture recognition where it contains sensor and speakers which could produce these waves and sensors would sense the change in amplitude and follow the command based on the profile given to it .Since battery consumption is low and can’t be affected by light. It can widely be used like operating a machine far away from us.

Infrared waves are also used for gesture recognition but only for smaller applications which can be achieved by phone because infrared can’t travel through obstacles where as there is no such problem arises in ultrasonic waves but it produces intense heat when passed through obstacle but can be overcome.

The Smartphone can pause sending sound signals where we close the cell with hand while someone else is talking. This prevents the person on the other side from entering into speakers personal issues.

Sometimes people really don’t want to reveal their background during video chats. We can use the depth estimation techniques to recognize the background and only send a blurred or blackened or different background.

The Smartphone can understand stealing gesture and can actually alarm you about the burglar. This way it keeps itself in safe hands.

7. Conclusions:

Gesture phones can recognise gestures by two methods, optical and inertial tracking. Optical tracking uses triangulation method for the estimation of details including the depth of the object. Inertial tracking is based on the idea of rolling shutters. The most widely accepted innovations on gesture phones are also mentioned. Their applications in android and windows phones along with different modes of app are explained. Challenges the gesture phones will face are described. Considering different areas where this technology can be implemented for efficient use of Smartphone, few innovations are stated.

References:

1. BongWhan Choe, Jun-Ki Min, and Sung-Bae Cho; Online Gesture Recognition for User Interface.

2. Mingyang Li, Byung Hyung Kim and Anastasios I. Mourikis; Real-time Motion Tracking on a Cellphone using Inertial Sensing and a Rolling-Shutter Camera.

3. Mingyu Chen, Ghassan AlRegib, Senior Member, IEEE, and Biing-Hwang Juang, Fellow, IEEE; Feature Processing and Modeling for 6D Motion Gesture Recognition.

4. Dong-ik ko, Lead engineer, gesture recognition and depth –sensing gaurav agarwal, Manager, Gesture recognition and depth-sensing; Gesture recognition: Enabling natural interactions with electronics.

5. Innovations on gesture technology, http://www.pointgrab.com/276/gesture-mobile/

6. Diseases related to gesture recognition on phones.http://www.incosecc.org/http:/www.incosecc.org/images/2012_01_18_Gesture-Recognition.pdf

7. Ultrasonic waves usage in gesture phones http://phys.org/news/2013-11-spinoff-ultrasonic-gesture-recognition-small.html

8. Ultrasonic waves usage in gesture phones http://www.washington.edu/news/2014/02/27/battery-free-technology-brings-gesture-recognition-to-all-devices/

9. Li yin Kong 2009127643; supervisor: dr. Kenneth Wong; fyp12026 gesture recognition on Smartphone