3d multi-touch system by using coded optical barrier on embedded photo-sensors
DESCRIPTION
a design for 3D modelTRANSCRIPT
Hsuan He Fang
3D Multi-Touch System by Using
Coded Optical Barrier on Embedded Photo-Sensors
Hsuan-He Fang* Institute of Electro-Optical Engineering, National Chiao Tung University, Hsinchu 30010, Taiwan
Guo-Zhen Wang Department of Electronics Engineering & Institute of Electronics,
National Chiao Tung University, Hsinchu 30010, Taiwan
Chia-Wei Chang Institute of Electro-Optical Engineering, National Chiao Tung University, Hsinchu 30010, Taiwan
Yi-Pai Huang Display Institute, National Chiao Tung University, Hsinchu 30010, Taiwan
Responding Author*-Tel : +886-3-5712121 ext. 59210, E-mail: [email protected]
Abstract
Limited by construction complexity, bare finger touch systems are not ready to be implemented on
current mobile devices. Hence, we proposed a system using coded optical barrier with less hardware
and software complexity; based on the construction, touch algorithm is programed to obtained 3D
location (x,y,z) of input(s). Finally, our concept was implemented on a 4-inch panel. The system was
able to sense up to 3 touch inputs simultaneously within 35 mm working range with multi-touch
applicable.
Keywords-3D touch, near-distance touch, in-cell photo sensor, depth sensing, bare finger
(a) Presentation Style: Oral preference
(b) Topical Section: Touch and Interactive Displays / Novel Touch Configurations and Applications
(c) The first author (Hsuan-He Fang) who will be the presenter is currently a master student.
Hsuan-He Fang
3D Multi-Touch System by Using Coded Optical Barrier on Embedded Photo-Sensors
Hsuan-He Fang1, Guo-Zhen Wang2, Chia-Wei Chang1 and Yi-Pai Huang3
1Institute of Electro-Optical Engineering, National Chiao Tung University, Hsinchu 30010, Taiwan
2 Department of Electronics Engineering & Institute of Electronics,
National Chiao Tung University, Hsinchu 30010, Taiwan 3Display Institute, National Chiao Tung University, Hsinchu 30010, Taiwan
Abstract Limited by construction complexity, bare finger touch systems
are not ready to be implemented on current mobile devices.
Hence, we proposed a system using coded optical barrier with
less hardware and software complexity; based on the
construction, touch algorithm is programed to obtained 3D
location (x,y,z) of input(s). Finally, our concept was
implemented on a 4-inch panel. The system was able to sense up
to 3 touch inputs simultaneously within 35 mm working range
with multi-touch applicable.
1. Background 3D touch is a platform that user performs natural gestures to
manipulate virtual 3D images. In general, 3D touch systems can
be separated into two classes by their working range. The first
class, which is based on machine [1] and camera constructions
[2], works in far-distance environment; it is already well-applied
on TV game consoles. However, their construction limitations,
such as additional device and blind range issues, restrict the
feasibility to be implemented on mobile devices. Therefore, the
second class, 3D touch in near-distance should be established.
The most promising candidate is in-cell photo sensor
touchscreen [3]. However, it is not ready for 3D touch yet owing
to low sensitivity. To conquer the issue, extra optical designs
should be constructed. They can be diversified into two types.
First, lighting mode [4] system, where user holds patterned light
pen to interact. However, because of the extra light source,
systems of lighting mode are considered inconvenient. Second,
reflecting mode system, where ideally light source can be
integrated with display so that user can perform nature gestures;
it’s hence more user-friendly. Nevertheless, no one is perfect by
far; new construction with new algorithm should be established.
2. Prior Approaches and Objectives In a reflecting mode system, ideally, user can naturally
manipulate computer with bare hand; so it’s also named bare
finger touch. Listed below are three prior systems but still
needed to be improved. The limitation and comparison is
described in Figure 1. First, 3D finger touch with sequential
illuminator [5] was proposed by M.C. Ma et al. It used
sequential light at different tilt angles to spot the touch point.
However, in the construction, slow frame rate was induced due
to capturing sequence. In addition, complex lateral light source
made it difficult to be realized on products. Second, a system
LCD with integrated 3-dimensional input device [6] was
proposed by C. Brown et al. Their key contribution is to propose
a construction of directional sensor. It analyzed disparity in the
sensor to obtain depth value. However, owing to the sensor
design, working range was too restricted for practical use. In
addition, constrained by the construction, the aforementioned
two systems were unable to support multi-touch. Third, an LCD
display system with depth-sensing capability based on coded
aperture imaging [7] was proposed by S. Suh et al. The system
uses dynamic lensless imaging system to enlarge working range.
Nevertheless, in the construction, switches between display
mode and touch mode would reduce frame rate and cause image
flickering. Moreover, processing numbers of images accounts
for huge computation.
Briefly sum up, bare finger touch system using in-cell photo
sensor is still needed to be optimized, which should support
sufficient working range, reduced construction complexity and
less computation loading. Therefore, a new construction which
uses coded optical barrier on photo-sensors and new depth
sensing algorithm are present.
Figure 1. Comparison chart among bare finger touch systems (*proposed system)
Hsuan He Fang
3. System Architecture The present paper designed depth sensors to achieve near-
distance touch. Based on traditional LC display, there are three
main components. First, IR light is integrated with display
backlight system, acting as touch light source. With IR light
source, display image would not be affected. Second, IR photo
sensor is embedded on the TFT substrate as common in-cell sensor. Third and the most distinct portion of the whole system
is optical barrier. Optical barrier is located above and patterned
for each pixel of photo sensor. With optical barrier on top,
optical sensor becomes depth-sensitive and it will be detailed in
next paragraph. Besides, depth sensors are at black matrix
region, so aperture ratio would not be affected much and sensor
would not be influenced by backlight system.
In depth sensor, we designed an aperture above each photo
sensor, as depicted in Figure 2. Aside from fixed pitch of LC
cell ( ) in the panel, all we design is displacement between the
sensor and the aperture ( ). More to our concept, we classified
depth sensor into two types: xy-sensor and multiple z-sensors. In
xy-sensor, the aperture is directly located above the photo sensor
(i.e. ). That is, if an input is above the panel, image
captured by xy-sensor is for the system to locate 2D coordinate
( ). On the other hand in z-sensor, aperture is slightly shifted
with displacement ( ) so that the sensor is able to capture
directional light input. Moreover, we further devise the
displacement ( ) and design spacing between xy and z-sensor
( ) to be proportional (i.e. ⁄ ⁄ ). Hence the z-sensor is
able to sense the input at specific depth ( ) directly above the
corresponding xy-sensor. Nonetheless, multiple z-sensors are
around the xy-sensor, and the displacement ( ) of each z-sensor
is different in order to sense the input at different depth ( ).
Moreover, as an example depicted in Figure 2, -sensor can
capture strong signal while -sensor capture weak signal.
Generally, captured intensity of each z-sensor is highly related
to the depth (z) of input; hence a data base, which records
captured intensity and known depth, is constructed for depth-
sensing. Finally according to the database, a continuous working
range is constructed.
Figure 2. Depth-sensing principle and system architecture
4. Algorithm Touch algorithm is designed to analyze captured images to yield
3D position (x,y,z) of each touch points; it is mainly composed
of image capturing, 2D coordinate location and depth sensing, as
illustrated in Figure 3. (A) In capturing procedure. IR touch
backlight is emitted, and sensors capture distributed light
reflected by inputs; hence, one xy-image and multiple z-images
are obtained. Following, noise suppression is conducted on all
the images to reduce background noises, Gaussian noises and
panel defects, etc. (B) Followed by the 2D coordinate location
procedure. The xy-image is input and transformed into black and
white image. Location (x and y) of white pixel are regarded as
features. ISODATA [8] clustering cluster these features by
minimum distance principle. That is physically, white pixels in
adjacent positions can be grouped together as touch point(s); it
eventually separates and renders the geometrical center(s) (x,y)
of touch point(s). (C) After, 2D coordinates (x,y) are fed into the
depth-sensing engine. For each touch point, z-sensors at
corresponding location are referenced and intensity values are
extracted. Following, the intensity values ( ) of every -
sensors are input to a maximum likelihood trained model to
render depth value (z).
Figure 3. Touch algorithm
In addition, the trained model is constructed from the intensity-
depth data base which records intensity ( ) captured by -
sensors at known step of depth discretely. Following in touch
mode, we calculate the difference between and , then
retrieve probability ( ) of each -sensor at step in
Gaussian model, equation (1). Following, the probability values
are multiplied among -sensors to render final probability
values ( ) of the input at each step of depth, equation (2);
then they are normalized to be the features ( ) for training,
equation (3). Finally, a trained model with maximum likelihood
[9] weighting ( ) is then executed to render depth value ( ),
equation (4).
: number of z-sensors : number of depth steps
: standard deviation of intensity captured by -sensor in
trained model
( )
(
( )
) (1)
(2)
(3)
∑ (4)
Hsuan-He Fang
5. Experiments To test the feasibility of depth sensing, a prototype was bulit on
a 4-inch panel with in-cell photo sensor. Unfortunitly, there was
no optical barrier in the panel, so we had to attach home-made
mask on the cover glass, as shown in Figure 4. Consequently,
the spacing between sensor layer and barrier (d) was increased,
and resolution of the depth sensor was decreased. Moreover, in
the prototype, we designed one xy-sensor surrounded by four z-
sensor to construct a 50 mm working range, which was
sufficient for mobile devices. Futhermore about the prototye, the
backlight system was not implemented, we used flash light with
diffuser in place of bare finger. In addtion, limited by driver IC,
we could only obtained 1 captured image with xy- and z-images
included rather than separate images.
Figure 4. Experimental setup
5.1. Z-sensor Response First of all, we built the intensity-depth database. The input
object was moved from 0 to 50 mm above the panel with step of
5 mm, and we recorded intensity values of every z-sensors at
each step. The database can be represented as an I-D curve, as
shown in Figure 5. In the figure, ( ) stood for the captured
intensity of -sensor where the object was at step n of depth.
Figure 5. Z-sensor response (intensity-depth curve)
In the curve, we observed that the z-sensor response followed
our concept within 20 mm; four z-sensors in turn started to
capture signal while flashlight was at 5, 10, 15, 20 mm
separately. However, only -sensor obtained weakened signal
when input over 35 mm, the other z-sensors still stayed
saturated. We believed the reason was the home-made mask
where we could only attached on instead of embedded in the
panel. Such situation ( ) caused design difficulties in
alignment ( ) and issues of incompatible size between sensor
and aperture.
5.2. Depth Accuracy The ability to estimate depth value of input was also testified.
An example is shown in Figure 6 for clear understanding. In the
processed image inset, 2D coordinate (x,y) squared in red was
rendered by ISODATA clustering; meanwhile corresponding z-
sensors were squared in different color. Following, intensity
values of z-sensors ( ) were extracted; hence by using
equation (1), the intensity values were transformed into
probability density functions ( ), also shown in the inset.
Followed by the normalization, features ( ), representing
where the object would likely be, were obtained through using
equation (2)-(3). Finally, maximum likelihood model calculated
the features and retrieved depth values (z). In this case, the error
was 3 mm.
Figure 6. An example with processed data and pdf
More to the depth accuracy, we further estimated the errors at
different steps within the working range. At each step, we
measured the inputs 15 times; the depth response is illustrated in
Figure 7. The maximum error was 5 mm within 35 mm working
range, which was acceptable for near-distance touch; but it went
too large to interact beyond 40 mm. Moreover, for every step,
calculated depth values floated in an error range; the issue was
believed to be resulted from z-sensor response already discussed
in 5.1. However, if we had a panel with embedded depth-sensor
( ), we could ensure error would be less than 1 mm.
Figure 7. Depth error measurement
Hsuan He Fang
5.3. Multi-touch We also clarified the feasibility of multi-touch function. An
example of 2-touch points is shown in Figure 8, two touch
points were put at 0, 10 mm separately. For clear understanding,
the procedure flow is narrated as followed. In the figure, the
arrow 1 indicated the procedure of 2D coordinate location; xy-
image was extracted from the captured image, then ISODATA
clustering rendered the 2D position (x’,y’) of each touch points.
Following, arrow 2, the locations were mapped into the original
coordinate system (x,y) in captured image. In the final
procedures 3 in the figure, the intensity values of z-sensors were
referenced. Moreover, remind of the system architecture in
Figure 2, the spacing between xy- and z-sensors is fixed ( ). We
only regarded and extracted ( ) at correct position, while all
other valued pixels were neglected. Furthermore to the example,
the detail processed data were listed in Table 1, which included
information of z-sensor reference in red dot-line square.
Figure 8. A example of 2 touch points and separation
Table 1. Processed data of the example in Figure 8.
Moreover, we also did experiments for more inputs. However,
whether the case was success depended on extent to how much
overlapping was. In the experiments, we could resolve utmost
three inputs. Last but not least, overlapping is an issue existing
in all interactive systems, yet by how much a system outcomes.
6. Impact We proposed a novel 3D touch system that is applicable on
mobile devices. The system’s working range in near-field and it
is possible to operate with bare fingers. The key contributions of
our works were to propose a simple depth sensing structure
which consists of coded optical barrier above embedded photo
sensors. By controlling the gap and displacement between
barrier aperture and photo sensors, the depth(z) position of
finger tip can be captured easily. Besides, along with touch
algorithm, the system was able to separate multiple touch points
and render 2D coordinate (x,y) by using ISODATA clustering;
plus, depth values (z) were retrieved through a maximum
likelihood model. Finally, we built a prototype on a 4-inch panel
to test the feasibility. The system was composed of 1 xy-sensor
and 4 z-sensors where the working range in depth was 0 to 35
mm with maximum error less than 5 mm. In addition, the system
supported upmost 3 inputs simultaneously.
7. Acknowledgement We’d like to express our appreciation to National Science
Council in Taiwan for financial support under contrast
Academic Projects No. NSC101-2221-E-009-120-MY3.
Meanwhile, we appreciate AU Optronics for useful advices and
panel support.
8. References [1] S. Feiner, B. MacIntyre, T. Hollerer and A. Webster,“ A
touring machine: Prototyping 3D mobile augmented reality
systems for exploring the urban environment”, in First
International Symposium on Wearable Computers Digest of
Papers, pp.74-81, 1997.
[2] A. Olwal, S. DiVerdi, N. Candussi, I. Rakkolainen and T.
Hollerer,“ in Proceedings of the IEEE conference on
Virtual Reality, pp. 279-280, 2006.
[3] A. Abileah, W.d. Boer, T. Larsson, T. Baker, S. Robinson,
R. Siegel and N. Fickenscher,“ Integrated Optical Touch
Panel in a 14.1” AMLCD,” Planar Systems Inc., 2004.
[4] G.Z. Wang, M.C. Ma, H.Y. Tung, Y.P. Huang, H.W.
Tseng, J.C. Lo and C.H. Kuo,“ 50.4: A Virtual Touched 3D
Interactive Display with Embedded Optical Sensor Array
for 5-axis Detection (x, y, z, θ, φ),” SID Symposium Digest
of Technical Papers, vol. 42, pp.737-740, 2011.
[5] M.C. Ma, G.Z. Wang, and Y.P. Huang, “P-199: 3D Finger
Touch with Sequential Illuminator,” SID Symposium Digest
of Technical Papers, vol. 42, no. 1, pp. 1848-1851, 2011
[6] C. Brown, D. Montgomery, J. L. Castagner, H. Kato, and
Y. Kanbayashi, “ 31.3: A system LCD with integrated 3-
dimensional input device,” SID Symposium Digest of
Technical Papers, vol. 41, pp. 453-456, 2010.
[7] S. Suh, C. Choi, K. Yi, D. Park and C. Kim,“An LCD
Display System with Depth-Sensing Capability Based on
Coded Aperture Imaging,” SID Symposium Digest of
Technical Papers, vol. 43, pp. 1574-1577, 2012.
[8] Ball, H. Geoffrey, Hall and J. David,” Isodata: a method of
data analysis and pattern classification,” Stanford Research
Institute, Menlo Park,United States. Office of Naval
Research. Information Sciences Branch, 1965.
[9] C.M. Bishop, Pattern Recognition and Machine Learning,
2nd edition. Springer 2007. Chapter. 2, 3.