capstone project multitouch

<Capstone Involving Multi-Touch Technology>

Senior Capstone ProjectDaniel A. Taylor

April 2013

Table of Contents

1. INTRODUCTION 1.1. THE TOUCH SCREEN

1.1.1. TEST THEORY1.1.2. OTHER TECHNOLOGIES1.1.3. BACKGROUND

2. MULTI TOUCH 2.1. FTIR

3. DESIGN3.1. PROJECT STATEMENT3.2. HARDWARE 3.3. SOFTWARE 3.4. WORKING 3.5.ADVANTAGES 3.6.APPLICATIONS

4. CONCLUSION5. APPENDICES

5.5.APPENDIX I 6. 6. REFERENCES

Capstone Involving Multi-Touch TechnologySenior Capstone Project for Daniel Taylor

ABSTRACT

Multi-touch technology is an advanced human-computer interaction technique that recognizes multiple touch points and also includes the hardware devices that implement it, which allow users to compute without conventional input devices. Multi-touch consists of a touch screen (screen, table, wall, etc.) or touchpad, as well as software that recognize multiple simultaneous touch points, as opposed to the standard touchscreen which recognizes only one touch point at a time. My Multi touch table using Frustrated Total Internal Reflection is a simple, inexpensive, and scalable technique for enabling high-resolution multi- touch sensing on rear-projected interactive surface. Different applications for multi-touch interfaces both exist and are being developed. The use of multi-touch technology is expected to rapidly become common place. Throughout the developmental process, I have exploring both the hardware and software side of Multi-touch computing. On the hardware end, various materials (camera, projection screen, etc.) were needed to construct the table. On the software end, both low level finger tracking or hand jester and high level software development were used in the Open-Source domain as a part of the Multi-touch device technology.

- 1 -


1. INTRODUCTION AN BACKGROUND

1.1 THE TOUCH SCREEN

A touch screen (Figure 1) is a display which can detect the presence and location of a touch within the display area. The term generally refers to touch or contact to the display of the device by a finger or hand. Touch screens can also sense other passive objects, such as a stylus. However, if the object sensed is active, as with a light pen, the term touch screen is generally not applicable. The thumb rule is: if you can interact with the display using your finger, it is likely a touch screen - even if you are using a stylus or some other object. Up until recently, most touch screens could only sense one point of contact at a time, and few have had the capability to sense how hard one is touching. This is starting to change with the emergence of multi-touch technology - a technology that was first seen in the early 1980s, but which is now appearing in commercially available systems. The touch screen has two main attributes. First, it enables you to interact with what is displayed directly on the screen, where it is displayed, rather than indirectly with a mouse or a touchpad. Secondly, it lets one do so without requiring any intermediate device, again, such as a stylus that needs to be held in the hand. Such displays can be attached to computers or, as terminals, to networks. They also play a prominent role in the design of digital appliances such as the personal digital assistant, satellite navigation devices and mobile phones.

Figure 1

1.1.1 TEST THEORY

There are several different ways to make a multi-touch surface, but I will focus only on one

method: the FTIR screen. An FTIR (short for Frustrated Total Internal Reflection) setup involves

three vital components: a sheet of transparent acrylic, a chain of infrared LEDs, and a camera with

an IR filter. The LEDs are arranged around the outside of the sheet of acrylic so that they shine

directly into the thin side surfaces.

- 2 -


Once the IR light is inside the acrylic, it strikes the top and bottom surfaces of the acrylic at a

near-parallel angle, and is subject to the effect known as total internal reflection. This causes it to

be wholly maintained in the acrylic. This is a little tough to describe in words, so I made a simple

diagram (Figure 2):

Figure 2

Infrared Explained

An IR touchscreen panel employs one of two very different methods. One method uses thermal induced changes of the surface resistance. This method is sometimes slow and requires warm hands. Another method is an array of vertical and horizontal IR sensors that detect the interruption of a modulated light beam near the surface of the screen. Optical imaging A relatively-modern development in touchscreen technology, two or more image sensors are placed around the edges (mostly the corners) of the screen. Infrared backlights are placed in the camera's field of view on the other sides of the screen. A touch shows up as a shadow and each pair of cameras can then be triangulated to locate the touch. This technology is growing in popularity, due to its scalability, versatility, and affordability, especially for larger units.

1.1.2 OTHER TECHNOLOGIES

Resistive

A resistive touch screen panel is composed of several layers. The most important are two thin metallic electrically conductive and resistive layers separated by thin space. When some object

- 3 -


touches this kind of touch panel, the layers are connected at certain point; the panel then electrically acts similar to two voltage dividers with connected outputs. This causes a change in the electrical current which is registered as a touch event and sent to the controller for processing. Surface acoustic wave SAW technology uses ultrasonic waves that pass over the touchscreen panel. When the panel is touched, a portion of the wave is absorbed. This change in the ultrasonic waves registers the position of the touch event and sends this information to the controller for processing. Surface wave touch screen panels can be damaged by outside elements. Contaminants on the surface can also interfere with the functionality of the touch screen.

Capacitive

A capacitive touch screen panel is coated with a material, typically indium tin oxide that conducts a continuous electrical current across the sensor. The sensor therefore exhibits a precisely controlled field of stored electrons in both the horizontal and vertical axes - it achieves capacitance. The human body is also an electrical device which has stored electrons and therefore also exhibits capacitance. When the sensor's 'normal' capacitance field (its reference state) is altered by another capacitance field, i.e., someone's finger, electronic circuits located at each corner of the panel measure the resultant 'distortion' in the sine wave characteristics of the reference field and send the information about the event to the controller for mathematical processing. Capacitive sensors can either be touched with a bare finger or with a conductive device being held by a bare hand. Capacitive touchscreens are not affected by outside elements and have high clarity. The Apple iPhone is an example of a product that uses capacitance touchscreen technology.

Dispersive Signal Technology

Introduced in 2002, this system uses sensors to detect the mechanical energy in the glass that occur due to a touch. Complex algorithms then interpret this information and provide the actual location of the touch. The technology claims to be unaffected by dust and other outside elements, including scratches. Since there is no need for additional elements on screen, it also claims to provide excellent optical clarity. Also, since mechanical vibrations are used to detect a touch event, any object can be used to generate these events, including fingers and stylus. A downside is that after the initial touch the system cannot detect a motionless finger.

1.1.3 BACKGROUND

Multi-touch technology dates back to 1982, when the University of Toronto developed the first finger pressure multi-touch display. The same year, Bell Labs at Murray Hill published what is believed to be the first paper discussing touch-screen based interfaces.

Bell Labs

In 1984 Bell Labs engineered a multi-touch screen that could manipulate images with more than one hand. The group at the University of Toronto stopped working on hardware and moved on to software and interfaces, expecting that they would have access to the Bell Labs work. A breakthrough occurred in 1991, when Pierre Wellner published a paper on his multi-touch Digital Desk, which supported multi-finger and pinching motions.

- 4 -


Finger works

In 1998, FW, a Newark-based company run by University of Delaware academics John Elias and Wayne Westerman, produced a line of multi-touch products including the iGesture Pad and the TouchStream keyboard. Westerman published a dissertation in 1999 on the subject. In 2005, after years of maintaining a niche line of keyboards and touch pads, Finger works was acquired by Apple Computer.

Perceptive Pixel and the rise of FTIR

PP is a company founded by New York University consulting research scientist Jefferson Y. Han that creates wall displays and tables that can accommodate up to 20 fingers. Han introduced the FTIR technique to multi touch screens. The displays use light emitting diodes along with infrared light to determine the point of contact. Han envisions large collaborative spaces that will allow multiple users to work and interact.

Apple iPhone, iPod touch, MacBook Air, and MacBook Pro In 2005, Apple acquired Fingerworks. In 2007 they introduced the iPhone, marking the first time multi-touch technology was used on a phone. The iPhone includes such components as a web browser, music player, video player, and a cell phone without the use of a hard keypad or stylus. Following the release of the iPhone, Apple also expanded its use of multi-touch computing with the new iPod Touch, as well as the new MacBook Air. Multi-touch was later added to the 2008 MacBook Pro line in the form of a track pad.

2. MULTI TOUCH

Multi-touch is a human-computer interaction technique and the hardware devices that implement it, which allow users to compute without conventional input devices e.g., mouse, keyboard. Multi-touch consists of a touch screen like screen, table, wall or touchpad, as well as software that recognizes multiple simultaneous touch points, as opposed to the standard touchscreen i.e. computer touchpad, ATM, which recognizes only one touch point. This effect is achieved through a variety of means, including but not limited to: heat, finger pressure, high capture rate cameras, infrared light, optic capture, tuned electromagnetic induction and shadow capture.

2.1FRUSTRATED TOTAL INTERNAL REFLECTION

Total internal reflection is an optical phenomenon that occurs when a ray of light strikes a medium boundary at an angle larger than the critical angle with respect to the normal surface. If the refractive index is lower on the other side of the boundary no light can pass through, so effectively all of the light is reflected. The critical angle is the angle of incidence above which the total internal reflection occurs. When light crosses a boundary between materials with different refractive indices, the light beam will be partially refracted at the boundary surface, and partially

- 5 -


reflected. If the ray is closer to being parallel to the boundarythen the light will stop crossing the boundary altogether and be totally reflected back internally. This can only occur where light travels from a medium with a higher refractive index to one with a lower refractive index. For example, it will occur when passing from glass to air, but not when passing from air to glass. The critical angle is the angle of incidence above which total internal reflection occurs. The angle of incidence is measured with respect to the normal at the refractive boundary. The critical angle is given by: where n

is the refractive index of the less dense medium, and n

is the refractive index of the denser medium

An important side effect of total internal reflection is the propagation of an evanescent wave across the boundary surface. Essentially, even though the entire incident wave is reflected back into the originating medium, there is some penetration into the second medium at the boundary. Additionally, the evanescent wave appears to travel along the boundary between the two materials. This wave can lead to a phenomenon known as frustrated total internal reflection. Under "ordinary conditions" it is true that the creation of an evanescent wave does not affect the conservation of energy, i.e. the evanescent wave transmits zero net energy. However, if a third medium with a higher refractive index than the second medium is placed within less than several wavelengths distance from the interface between the first medium and the second medium, the evanescent wave will be different from the one under "ordinary conditions" and it will pass energy across the second into the third medium.

3. DESIGN

3.1 PROJECT STATEMENT

My Multi touch is designed using the FTIR technology. FTIR describes the internal reflection of light. It is force-sensitive, and provides unprecedented resolution and scalability. The Touch screen had to be large enough to accommodate both hands and multiple users. This phenomenon is also used in fingerprint and robot sensors. My mission plan was to design a Multi-touch environment that allows an individual to work in a computer aid environment. A display table designed for Multi-touch interaction was the point of Capstone collaboration. Users can directly manipulate what is projected on screen and a framework within Windows 7 was programmed to make use of the table’s Multi-Input (Touch) ability.

- 6 -


3.2 HARDWARE

The basic design has a hardware and software part. Hardware requires basically IR LEDs, acrylic, camera, projector and a computer. Infra-red light has a higher wavelength than that of visible light. Therefore it has more intensity and will be felt everywhere inside the denser medium. Acrylic is the denser medium .It is a synthetic fiber having half the density of glass. An infra-red camera or a webcam is used to catch IR light. An IR block filter of the camera has to be removed since it blocks IR light. It also consists of a projector and a computer. The object has to be projected on top of the acrylic from a computer. Thus acrylic is a virtual display. The IR LEDs about 10-20 are arranged on both sides of the acrylic along its edges. The camera below the acrylic and the projector located under the acrylic.

3.3 SOFTWARE

Computer applications are necessary to communicate between a multi touch display and the computer. These applications can be developed within several languages / programming environments. For example: Processing (P5), Flash, C, C++, Java and others. Because of the usability of certain APIs and the relatively simple visualization possibilities, P5 or Flash 11, in combination with Action script 4.0 will make a great combination.

3.3.1 Calibration

- 7 -

http://danoataylor.files.wordpress.com/2013/01/multitouch_box11.jpg

http://danoataylor.files.wordpress.com/2013/04/2013-04-08-09-50-08.jpg


I have researched multiple open source calibration programs. The central software that I have chosen to power my table is Touchlib, an open source library which takes the visual data received by the camera and parses it into touch events, which can be used by other programs to provide multi-touch control. Some programs implement this library directly, allowing for standalone multi-touch apps, while others, such as those written in Action Script, require an extra software layer to allow the program to receive touch input. In this section, we’ll explain how we got both up and running.

3.4 WORKING

FTIR describes the internal reflection of light, inside a certain material. In our case, it will be infrared light, that internally reflects inside is a piece of acrylic, also known as Plexiglas. This way, infrared light is beamed inside the acrylic and reflects internally. In a simple way, you can say that, IR-light bounces inside the acrylic, from one side to another. As soon as a finger touches the acrylic surface, the internal reflection of the IR-light, is interrupted. The infrared light scatters on the finger tips. Infrared light is invisible to the human eye, but by placing an infrared camera behind the acrylic your fingertips will be visible on the infrared camera. The images that are generated by the camera contain white blobs (caused by the fingertips). These blobs will be analyzed by software. Every blob corresponds to certain coordinates. Software can by analyzing these coordinates perform certain tasks, for example move, resize or rotate objects. Multiple points are obtained on the camera .Each point is a pixel position. Either a single pixel or a group of pixels. Each point locations are identified and all operations are performed. Suppose if I wanted to zoom a picture using two fingers and move it in or out to perform zoom in and zoom out respectively. Two coordinates will be located on the camera. The difference is found which is put as the offset and it is either added or subtracted with the locations to zoom out and zoom in respectively.

3.5 ADVANTAGES

Multi touch based on FTIR is a simple and inexpensive technique .It constructs a multi touch display with the available and less costly materials.

Scalable technique that enables high-resolution graphics .It provides support to any resolution possible as all multiple points could be generated on a camera

- 8 -

http://danoataylor.files.wordpress.com/2013/03/multitouch_software2.png

http://danoataylor.files.wordpress.com/2013/03/multitouch_software3.png


It acquires true touch image information at high spatial and temporal resolutions. The actual finger print of the touch is obtained. This could be used to determine the force sensitivity on displays; either too hard or soft touches can be analyzed.

It is scalable to large installations. Any kind of applications can be made to suit multi touch using FTIR. Allows us to create sophisticated multi-point widgets for applications

Larger shared-display systems i.e. it is well suited for use with rear- projection like wall screens, table tops .All this lead to high resolution graphics.

3.6 APPLICATION

A myriad of different application for multi-touch interfaces both exists and is being developed. Some uses are individualistic e.g., iPhone, iPod touch, MacBook Pro, MacBook Air, HTC Diamond. However, multi-touch technology is mainly used to incorporate collaboration into the computing experience.

A multi touch display can be used in

Personal Computers, Laptops, Tabletops, Graphics Tablets.

It supports both LCD and CRT monitors.

Telephones, Watches, PDAs, Mobile phones.

Advanced multi touch Gaming with high graphics support

Governmental, office and business purposes

An enhanced multimedia experience including audio, video and photo sharing

Enhanced dining experience

Applications for a multi touch display are never ending.

4. CONCLUSION

Touch screens are the interface for the 21st century. Touch screens address the conflicting demands for smaller portable electronics with larger displays, by eliminating traditional buttons without sacrificing screen size. The recent release of the iPhone has created a buzz around touch screen interfaces and its multi-touch acrobatics have caught the eye of many industry leaders. There are many ways to make a multi-touch screen. Some of the early designs measured the change in electrical resistance or capacitance on a surface when fingers touched it. But these devices have limited resolution, are relatively complex, and don't easily and inexpensively scale up to large dimensions. Multi-touch technologies have a long history. This technique using FTIR is simple and easy to implement. It provides any resolution displays supported with high graphics .The applications being both made and proposed are plenty in number. A drawback of the approach is that, being camera-based, it requires a significant amount of space behind the interaction surface, though we primarily expect application scenarios where rear-projection would have been employed anyway (e.g. interactive walls, tables). Also, as an optical system, it remains susceptible to harsh lighting environments.

- 9 -


The use of multi-touch technology is expected to rapidly become common place. For example, touch screen telephones are expected to increase from 200,000 shipped in 2006, to 21 million in2012. Developers of the technology have suggested a variety of ways that multi-touch can be used including:

Enhanced dining experience

Concierge service

Governmental use

Concept mapping

Collaboration and instruction on Interactive Whiteboards

- 10 -


APPENDICES

Action script 4 (Flash)

[1] Adobe Flash, Wikipedia entry. http://en.wikipedia.org/wiki/Adobe_Flash

[2] “Flash for surface computing” by Manvesh Vyas http://www.adobe.com/ devnet/edu/articles/manvesh-vyas.html

[3] flosc: Flash Open Sound Control http://www.benchun.net/flosc/

[4] Migrating from ActionScript 2.0 to ActionScript 4.0: Key concepts and changes by Dan Carr http://www.adobe.com/devnet/flash/articles/first_as3_ap- plication.html

C++

[1] C++ Programming Language. http://isocpp.org/

Multi-Touch

[1] Han, Jerfferson Y. “Low Cost Multi-Touch Sensing through Frustrated To- tal Internal Reflection.” Symposium on User Interface Software and Technology: Proceedings of the 18th annual ACM symposium on User interface software and technology. Seattle,WA, USA, 2005. 115-118.

- 11 -


6. REFERENCES Low-Cost Multi-Touch Sensing through FTIR by Jefferson Y. Han Buxton, W., Hill, R., and Rowley, P. 1985. Issues and Techniques in Buxton, Bill. 2008. Multi-Touch Systems that I Have Known and Loved. http://www.billbuxton.com/multitouchOverview.html How to build a multi touch by Harry Vaan Der Opensource,MultitouchDisplay,http://www.technologyreview.com/Infotec

- 12 -

capstone project multitouch

Documents