Doctoral Dissertation

Hybrid Framework for

Haptic Texture Modeling and Rendering

Sunghwan Shin (신 성 환)

Department of Computer Science and Engineering

Pohang University of Science and Technology

2019

햅틱 질감의 모델링 및 렌더링을 위한

혼용 체계

Hybrid Framework for

Haptic Texture Modeling and Rendering

Hybrid Framework for

Haptic Texture Modeling and Rendering

by

Sunghwan Shin

Department of Computer Science and Engineering

Pohang University of Science and Technology

A dissertation submitted to the faculty of the Pohang

University of Science and Technology in partial fulfillment of

the requirements for the degree of Doctor of Philosophy in the

Computer Science and Engineering

Pohang, Korea

6. 24. 2019

Approved by

Seungmoon Choi (Signature)

Academic advisor

DCSE

20101109

신 성 환. Sunghwan Shin

Hybrid Framework for Haptic Texture Modeling and Ren-

dering,

햅틱 질감의 모델링 및 렌더링을 위한 혼용 체계

Department of Computer Science and Engineering , 2019,

107p, Advisor : Seungmoon Choi. Text in English.

ABSTRACT

Haptic texture is one of the fundamental haptic properties that we can per-

ceive in our daily life. Since most of the tactile signals generated when we interact

with the environment are provided in the form of haptic textures, rendering re-

alistic virtual texture to users has been one of the aspirations that many haptics

researchers want to achieve. From an industrial perspective, the realistic haptic

texture is a technology of high importance because it can enhance the reality and

immersion of various virtual reality applications.

Existing techniques for modeling and rendering haptic textures can be roughly

divided into two categories. One is a physics-based texture model using a physical

model suitable for describing the physical properties of an object, and the other

is a data-driven texture model that synthesizes a representative texture signal

using a non-parametric model. The advantage of the physics-based model is that

I

the amount of data required is relatively small, and the physical interpretation

of the model is trivial because the model can be expressed in the form of physics

equations. However, this model is inappropriate to render very realistic haptic

textures for its complexity. In such cases, we need a very complicated model

that is hard to construct and cannot be simulated in a real-time system. On the

other hand, the data-based model can express such complex physical relationship

since the model postulate a black-box relationship between the user’s touch and

the resulting signal. Therefore, the data-driven model is able to express a more

realistic texture without complicated physical simulation. However, the amount

of the data for modeling increases exponentially for more input variables, and it

restricts the applicability of the data-driven model to various types of textures. In

particular, inhomogeneous textures with non-uniform characteristics are difficult

to be handled in a data-based manner because they exhibit different responses

depending on the contact position. To summarize, the physics-based model is

inadequate for realistic texture rendering, and the data-driven model has low ap-

plicability. The data-driven model is generally applied to homogeneous textures.

Based on these considerations, we propose a hybrid framework for modeling

and rendering haptic texture in this thesis. By combining the two models in

a complementary manner, we improve the low applicability of the data-driven

model and increase the reality of the physics-based model. To this end, we use

the physics-based model to model the position-related features of the texture and

apply the data-driven model to capture the position-invariant characteristic of the

– II –

textured material. Then, the two models are combined to render realistic virtual

surfaces. Our framework is composed of three parts. The first is a data-based

model using a linear predictive coding (LPC)-based method. In this part, we

present a data-based approach to acquire the position-invariant characteristic of

non-uniform textures. In the next part, we improve the modeling performance of

physics-based texture models using Photometric Stereo method. The photometric

stereo increases the resolution of the geometry model approximately ten times.

We also devise an improved rendering algorithm. Finally, the third part is about

a rendering algorithm to combine the above two haptic texture models together.

In order to prevent the two models from interfering with each other, rendering is

performed through a filtering process between the two signals generated by the

two models. The virtual textures presented through the completed framework

were compared with the real textures through user studies. We confirmed that

the proposed hybrid scheme can reproduce more realistic virtual textures than

the previous works.

– III –

Contents

I. Introduction 1

1.1 Goal of Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

II. Background 6

2.1 The perceptual space of haptic texture . . . . . . . . . . . . . . . . 6

2.2 Geometry-based modeling and rendering . . . . . . . . . . . . . . . 9

2.3 Vibration and Data-driven modeling and rendering . . . . . . . . . 12

2.4 Stochastic modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 16

III. Data-driven Modeling & Rendering of Homogeneous Texture 18

3.1 Neural network as a non-linear time-series predictor . . . . . . . . 18

3.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.3 Modeling of Isotropic Textures . . . . . . . . . . . . . . . . . . . . 23

3.3.1 Input Variables . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3.2 Model Topologies . . . . . . . . . . . . . . . . . . . . . . . . 25

3.3.3 Individual Neural Network Structure . . . . . . . . . . . . . 26

3.3.4 Error Metric . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3.5 Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 30

IV

3.5 Application to Inhomogeneous Texture . . . . . . . . . . . . . . . . 35

3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

IV. Geometry-based Modeling & Rendering of Inhomogeneous Texture 38

4.1 Geometry-based modeling and our approach . . . . . . . . . . . . . 38

4.2 Height Map Estimation . . . . . . . . . . . . . . . . . . . . . . . . 40

4.2.1 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.2.2 Photometric Stereo Algorithm . . . . . . . . . . . . . . . . 41

4.3 Friction and Stiffness Modeling . . . . . . . . . . . . . . . . . . . . 44

4.3.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.3.2 Dahl Model Identification . . . . . . . . . . . . . . . . . . . 45

4.3.3 Hunt-Crossley Model Identification . . . . . . . . . . . . . . 47

4.4 Texture Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.5 User Study 1: Assessing the realism of the virtual textures . . . . . 51

4.5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.6 User Study 2: Comparision btw. two modalities . . . . . . . . . . . 60

4.6.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.6.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . 63

4.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

V. Hybrid Texture Rendering 68

5.1 Hybrid Rendering Approach . . . . . . . . . . . . . . . . . . . . . . 68

5.1.1 Force-generation Algorithms . . . . . . . . . . . . . . . . . 69

V

5.1.2 Stability Analysis . . . . . . . . . . . . . . . . . . . . . . . . 72

5.1.3 Vibration Synthesis . . . . . . . . . . . . . . . . . . . . . . 74

5.1.4 Hybrid Rendering Algorithm . . . . . . . . . . . . . . . . . 75

5.2 User Study 1: Comparison of Force-Feedback Rendering Algorithms 76

5.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

5.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5.3 User Study 2: Assessing the Realism of Hybrid Haptic Texture . . 83

5.3.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

5.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

VI. Conclusions 93

Summary (in Korean) 96

References 98

VI

List of Tables

4.1 Means and standard deviations of the similarity scores. . . . . . . . 57

4.2 Correlation coefficients of the overall similarity score with different

measures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5.1 Formulas of the three force-texture rendering algorithms. nt is a

gradient of the texture height map. h(x) stands for the normalized

texture height. θt is the angle between the moving direction of the

user and nt. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.2 ANOVA Statistics of the user study 1 . . . . . . . . . . . . . . . . 90

5.3 ANOVA Statistics of the user study 2 . . . . . . . . . . . . . . . . 91

5.4 Aggregated roughness similarity scores . . . . . . . . . . . . . . . . 92

5.5 Aggregated overall similarity scores . . . . . . . . . . . . . . . . . . 92

VII

List of Figures

1.1 The schematic diagram of our hybrid framework. We combine

contact acceleration-based model, physics-based model and image-

based model to cover all the important attribute of real textures.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3.1 Texanner: 2D texture scanning device. . . . . . . . . . . . . . . . . 21

3.2 Acceleration recording of a textureless object (CD). Black: ac-

celeration from the stylus. Gray: acceleration from the moving

platform. In the Stop mode, the stylus did not move, so the signal

came from the accelerometer’s noise. In the Move mode, the stylus

was scanned on the CD surface with different velocities. . . . . . . 23

3.3 Structure of ΓDij based on a frequency-decomposed neural network. 24

3.4 Texture samples used for performance evaluation. . . . . . . . . . . 29

3.5 Acceleration signals and magnitude spectrums of measured and

synthesized acceleration data in the best case(Corduroy, ΓU, v =

6 cm/s, f = 1.40 N). Dark navy lines show original magnitude

spectrums, and pale pink lines represent smoothed spectrums using

a moving average filter. . . . . . . . . . . . . . . . . . . . . . . . . 31

VIII

3.6 Acceleration signals and magnitude spectrums of measured and

synthesized acceleration data in the worst case(Scrunched paper,

ΓD, v = 6 cm/s, f = 0.89 N). Dark navy lines show original magni-

tude spectrums, and pale pink lines represent smoothed spectrums

using a moving average filter. . . . . . . . . . . . . . . . . . . . . 32

3.7 Relative spectral rms errors Es. . . . . . . . . . . . . . . . . . . . 34

4.1 Apparatus for texture modeling. LEDs (marked by orange circles)

are installed inside the polycarbonate dome. . . . . . . . . . . . . 41

4.2 Examples of real materials and reconstructed height maps. Details

of each material are well preserved. . . . . . . . . . . . . . . . . . . 43

4.3 Ten materials used in the user study. Three materials that showed

the lowest scores in the pilot experiment are highlighted with blue-

dashed squares. Note that some materials (denim, rubber mat,

scrunched paper, towel, and sponge) have nonhomogeneous tex-

tures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.4 Experimental setup. (a) Screenshot of the program. Two white

squares represent the locations of two textures. The black sphere

indicates the position of the PHANToM stylus. (b) Physical en-

vironment. A barrier (indicated by a semi-transparent plane)

blocked the participant’s view. . . . . . . . . . . . . . . . . . . . . 53

– IX –

4.5 Average similarity scores with standard errors. Blue, green, yellow

bars represent the promising group, the unpromising group, and

the upper and lower bounds respectively. A: significantly different

from the lower limit, but not from the upper limit. B: significantly

different from the upper limit, but not from the lower limit. C:

not significantly different from the lower limit nor the upper limit. 56

4.6 Ten real materials used in the user study. . . . . . . . . . . . . . . 61

4.7 Experimental setup. (a) Physical environment. (b) Screenshot of

the program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.8 Average similarity scores for the five criteria. Error bars indicate

standard errors. Pairs grouped by asterisks were significantly dif-

ferent by Tukey’s HSD tests (∗ : 0.01 < p < 0.05, ∗∗ : p < 0.01). . . 64

5.1 Schematic diagram of our hybrid rendering process. . . . . . . . . . 69

5.2 Schematic diagrams for two popular texture rendering algorithms.

Circles represent the position of the haptic tool in the virtual space.

Thick straight lines represent the nominal surface, and dashed lines

show textured surfaces. . . . . . . . . . . . . . . . . . . . . . . . . 70

5.3 The path to calculate the energy gain of force algorithms. The red

dashed line represents the direction of the trajectory, and the blue

line in path A is the texture height map. We tested 2D trajectory

for a straightforward calculation. . . . . . . . . . . . . . . . . . . . 73

– X –

5.4 A diagram of vibration synthesizing using an LPC-based model [1].

The inverse model 1/P (z) synthesizes the contact vibration ag(l)

of the texture by taking white noise eg(l) as a moving average term

of the ARMA filter H(z). . . . . . . . . . . . . . . . . . . . . . . . 75

5.5 Texture materials used in the user studies. . . . . . . . . . . . . . . 77

5.6 Experimental setup of the user studies. The left figure is a picture

of the physical environment, and the right one is a screenshot of

the experiment program. . . . . . . . . . . . . . . . . . . . . . . . . 79

5.7 Average similarity scores of user study 1 for the three criteria.

Error bars indicate 95% confidence intervals. Pairs grouped by as-

terisks were significantly different according to Tukey’s HSD tests

(∗ : 0.01 < p < 0.05, ∗∗ : p < 0.01). . . . . . . . . . . . . . . . . . . 80

5.8 Average similarity scores of the user study 2 for the three criteria.

Error bars indicate 95% confidence intervals. Pairs grouped by

asterisks were significantly different for Tukey’s HSD tests (∗ :

0.01 < p < 0.05, ∗∗ : p < 0.01). . . . . . . . . . . . . . . . . . . . . . 84

– XI –

I. Introduction

1.1 Goal of Research

In haptics research, the haptic texture is one of the most important haptic

properties. Haptic texture generally refers to a sensation caused by the fine struc-

tural irregularities of an object when we touch it. Since haptic texture sensation

has a great impact on the user’s haptic experience, modeling and rendering hap-

tic sensation same to that of the real object have been one of the major research

topics in haptics for decades.

Since it is almost impossible to formularize the physical process of gener-

ating haptic texture sensation, haptics researchers have used different types of

approximation models such as geometry-based physical model, stochastic model,

data-driven acceleration models. Despite such focused research endeavors, we

still don’t have a ‘standard’ method of modeling and rendering realistic haptic

texture since haptic texture perception consists of multiple perceptual dimen-

sions such as roughness-smoothness, warmness-coldness, hardness-softness, and

stickiness-slipperiness. To the best of our knowledge, it still remains very chal-

lenging to model and renders all the major perceptual dimensions of texture in

a realistic manner.

Our research goal is to develop a holistic framework of accurate haptic tex-

ture modeling and rendering that covers major dimensions of haptic texture per-

– 1 –

ception. To this end, we develop a hybrid modeling and rendering. Our frame-

work is in agreement with the duplex theory of tactile texture perception-tactile

texture perception is mediated by vibrational cues for fine texture and by spatial

cues for a coarse texture. We provide vibration feedback using LPC-based data-

driven model, which generates a contact acceleration profile of similar spectral

content to the fine textural property of a real sample that is uniform over the sur-

face. Simultaneously, a high-accuracy geometry model of the texture is captured

using photometric geometry, and corresponding kinesthetic feedback expressing

local textural features is also rendered in order to handle the inhomogeneity of the

texture. By combining the two texture models, we aim to deliver more realistic

and immersive virtual haptic texture to the user.

1.2 Our Approach

We call our approach ‘hybrid’ because it integrates both kinesthetic and

vibrotactile feedback to deliver more realistic texture sensation. According to

the duplex theory of texture perception [2], haptic texture sensation is the com-

bination of two signals transferred through two neural channels. Fine textural

features (e.g., particle size ¿ 100µm) induce contact vibration when the user scans

the textured surface. The Pacinian (PC) mechanireceptive channel mediates this

contact vibration in the form of a temporal signal. Conversely, coarse texture

features are encoded in terms of their spatial layout. The user can perceive them

through the kinesthetic feedback generated when s/he scans them using a rigid

tool. This theory inspired us to develop and render two respective models for

– 2 –

Figure 1.1: The schematic diagram of our hybrid framework. We combine contact

acceleration-based model, physics-based model and image-based model to cover

all the important attribute of real textures.

more realistic haptic texture rendering.

To model the micro-geometry of a real object, we use the photometric stereo

algorithm. In photometric stereo, the surface geometry is estimated by ana-

lyzing the correspondence among different images photographed under precisely

controlled lighting conditions using a fixed camera. The photometric stereo algo-

rithm is known as one of the most precise methods for acquiring highly accurate

surface geometry. Since the criterion to discriminate coarse textural features from

fine textural features is approximately 100µm, such an accurate algorithm is re-

quired for texture modeling. A 3D profilometer, although extremely accurate, is

not a particularly attractive option for haptic texture modeling due to its high

price and slow scanning velocity in the order of 10−3 mm/s. We designed and

– 3 –

built a custom apparatus to apply photometric stereo to texture modeling. A new

force-feedback rendering algorithm for our high-resolution geometry is proposed

as well.

To model the contact vibration of texture scanning, we adopt an LPC-based

method. This method is the state of the art for modeling a homogeneous hap-

tic texture with a contact vibration signal that exhibits stationary behavior. To

extend the target of this method to an inhomogeneous texture, we filter out the

non-stationary components of the vibrotactile signal induced by the inhomogene-

ity of the texture. The non-stationarity is not required and should be removed

from the contact vibration model as that component are enclosed in the micro-

geometry of the surface. The objective of the contact-acceleration model in our

framework is to model the stationary contact acceleration induced by fine textural

features.

We then integrate the two models to recreate the virtual texture with high

resolution and realism. The performance of our hybrid framework was evaluated

through two user studies. The first solely targeted the rendering performance of

our force-feedback algorithm. Then, in the second user study, we assessed the

realism achieved by our framework.

The contributions of our work are 1) an image-based haptic texture modeling

method with very high resolution, 2) a kinesthetic texture rendering algorithm

for a high-resolution geometric structure, 3) a combination of a kinesthetic and

a vibrotactile texture model, and 4) an evaluation of our hybrid framework in

terms of realism, i.e., similarity between real and virtual textures.

– 4 –

1.3 Organization

The rest of this thesis is structured as follows. In Chapter 2, the background

for haptic texture including modeling and rendering methods and perception are

provided. Chapter 3 states the LPC-based data-driven haptic texture modeling

method we develop to capture the homogeneous characteristic of inhomogeneous

textures. Chapter 4 provides the geometry-based haptic texture modeling and

rendering algorithm using photometric stereo, which is suitable for rendering in-

homogeneous textural features. The combination of both algorithms is presented

in Chapter 5. The conclusion of the thesis is included in Chapter 6.

– 5 –

II. Background

2.1 The perceptual space of haptic texture

The understanding of the human’s texture perception is indispensable to

render very realistic haptic texture. Actually, the perceptual mechanism un-

derlying the haptic texture perception is still not identified clearly; it is still

unclear how the physical properties of haptic texture are connected to the per-

ceptual properties. To explore the perceptual dimension of haptic texture, the

researchers collect subjective data for various textured material and then ex-

tract the possible perceptual dimensions using data analysis methods such as

factor analysis (FA), principal component analysis (PCA) or multidimensional

scaling (MDS) [3, 4, 5, 6, 7]. Hollins et al. investigated the perceptual space of

haptic texture when participants passively touch 17 materials [3]. Participants

classified the materials into some groups based on their perceptual similarities.

The MDS results suggested three perceptual dimensions, and the first dimension

and the second dimensions showed very high correlation to the adjective rating

of roughness and stiffness, respectively. The third dimensions showed low cor-

relations to all of the adjective ratings. Later, the same authors reported that

while some participants use only 2-D perceptual space consists of roughness and

stiffness to discriminate textures, the other uses additional dimension that has a

high correlation with stickiness [4].

– 6 –

Picard et al. also conducted similar experiment [5]. In their experiment,

20 participants classified 24 materials on the basis of similarity. Four dimensions

were extracted by the MDS results on the similarities. The first dimension showed

high correlation with the label of ‘harsh’, ‘rough’, ‘hard’ and ‘soft’, ‘mellow’,

‘smooth’. Which means, this dimension can be a mixture of the rough - smooth

and hard - soft dimension. The second and the fourth dimensions were mainly

described by thin - thick and harsh - mellow, respectively. One interest thing

here is, that the third dimension was aligned with the relief - smooth scale.

Although the relief and the smooth scales are on the opposite side, these were

not perfectly aligned. This founding let us consider two types of roughness:

macro-roughness and fine roughness. Macro roughness generally described by

‘relief’ and fine roughness is usually explained as ‘rough’. However, it is not

easy to discriminate between these two perceptions because the other side of two

dimensions was same ‘smooth’. In 2004, the authors compared the haptic texture

perception of two groups of participants, with two groups of fabrics [6]. Although

two groups’ data were analyzed separately, the perceptual spaces generally show

quite good agreement. The three salient dimensions were rough-smooth, hard-

soft, and thin-thick. Also depending on the type of fabric, slippery-sticky and

relief-silky dimensions were also elicited.

The concept of two different roughness - macro and fine roughness was also

reported by some other researchers [8, 9, 10, 7]. In 2000, Hollins and his colleagues

proved that perception of fine textures is transferred by the vibration, while macro

textures are coded in term of their spatial features [9]. In their experiment with

– 7 –

sandpapers with different size of gratings, they showed that this shift between

macro- and micro-roughness happens when the grain size of the texture is about

100 µ m. In their continued work [8], the authors showed that participants’

perception of roughness can be enhanced when an external vibration is applied to

them. Further, adaptation to vibration impairs the perception of fine textures but

not coarse ones in their 2001 work [10]. They explained this phenomenon by ‘The

duplex theory of tactile texture perception’, which states that our tactile texture

perception is mediated by two different channels; one channel responds to high-

frequency vibration stimuli, while the other reacts to spatial change. Gescheider

et al. also demonstrated similar phenomenon [7]. They used plastic rased-dot

textures with different spacing as texture samples to classify. In their experiment,

they also added the adaptation of Pacinian channel as an experimental condition

to test the effect of Pacinian corpuscles on roughness coding. The MDS results

propose three dimensions of ‘blur’, ‘roughness’, and ‘clarity’. Among them, only

the dimension of clarity was affected by the adaptation of Pacinian corpuscle.

Therefore, we can regard that micro roughness, which can be explained by the

clarity dimension is mediated by Parcinican channel while macro-roughness is

not. One notable thing is that although the participants can discriminate between

two types of roughness, they integrated them when they should explicitly rate

the roughness of the textured samples.

In conclusion, the perceptual space of haptic texture has not been fully

unveiled yet as we see from the aforementioned studies. Depending on the type

of materials or participants, the haptic texture dimensions differ. However, one

– 8 –

agreement among studies is that the MDS results generally extract the dimension

of rough-smooth, hard-soft, and sticky-slippery. Also, some studies suggest two

types of roughness perception - macro and micro-roughness. Therefore, we can

consider those dimension as fundamental or most influencing dimensions and

should focus on delivering those perceptions clearly in our algorithm.

2.2 Geometry-based modeling and rendering

Most of the early haptic texture projects used the geometry profile of a tex-

tured surface to compute the response force of a force-feedback device. In 1995,

Minsky showed that rendering lateral force based on the spatial gradient of a

texture height model can display different sensations of texture [11]. Her setup,

named as Sandpaper system, was based on a 2D force-feedback joystick, which

could render only lateral forces. In her experiment, the rendered lateral forces

were proportional to the spatial gradient of height. The results confirmed that it

is possible to convey the perception of many textured surfaces using only lateral

forces. Which means 2D lateral force variation is effective to mimic the surface

with 3D microstructure. After that, Minsky and Lederman exhibited that the

Sandpaper system can deliver the roughness perception of textured patches [12].

Actually, the perceived roughness showed a linear relation with the maximum

lateral force rendered by the system while stroking the textured surface. This

work’s 3D extension was done by Hardwick and his colleagues using an Immer-

sion Impulse Engine 3000 [13]. Different from Sandpaper system, now the Haptic

interaction point (HIP) can above the textured surface. Therefore, their system

– 9 –

considers the penetration depth into the textured surface instead of the height of

the surface itself. By taking the gradient of the penetration depth, Hardwick’s

system could render textured surface in the 3D environment also. One year later,

a different approach has been proposed by Massie. In this thesis, he demonstrated

that haptic textures can be generated by perturbing the penetration depth of the

haptic tool into a virtual surface according to a texture model [14]. Different

from Minsky’s method, the rendered force was normal to the textured surface.

By varying the magnitude of the normal force, the user can feel convincing hap-

tic textures, although the surface feels like frictionless since the rendered force

contains no tangential component.

These two pioneering studies inspired many texture rendering algorithms

using force feedback. In 1997, Basdogan et al. applied the bump-mapping tech-

nique in computer graphics field to render haptic texture using both of lateral

force and perturbing normal force [15]. By perturbing the direction and magni-

tude of normal forces using the gradient of surface’s height map, the roughness

of the textured surface can be rendered. The resulting force field found to be

effective in rendering rough texture, but on the other hand, the instability issues

were also reported because of fast-changing texture’s reaction force. To comple-

ment that shortcoming, Ho et al. limited the magnitude of the resulting lateral

force [16]. Although their method is empirical, they could improve the stabil-

ity of the system while not degrading the rendering performance. Inspired by

Minsky’s gradient technique, Hayward et al. proposed a new algorithm for 2D

texture rendering based on the idea of directional finite differentiation [17]. The

– 10 –

proposed algorithm generates the texture force proportional to the directional

change in the height map of the surface while limiting the direction of the force

to the user’s movement. Compare to the previous gradient-based algorithm that

needs the height map to be differentiable at all the position, this algorithm could

render non-differentiable and discontinuous heightmap without any modification.

Recently, Campion developed a new 3D texture rendering algorithm based on

the perturbation of lateral friction force [18]. In this algorithm, the direction of

normal force does not change. Instead, the change of normal force’s magnitude

results in the change of frictional force’s magnitude, which introduces the rough-

ness perception of the textured surface. Meanwhile, Lederman and colleagues

proposed another approach for texture rendering using geometry data: they con-

sider the physical interaction between the probe of the tool controlled by the

user and the geometry of the textured surface [19]. Their first attempt didn’t

make very realistic texture sensation compare to the real one, but in similar work

done by Otaduy and Lin resulted in good-matching simulation results with the

measured data from the real materials [20].

While diverse type of rendering algorithms has been proposed using geometry

data, the modeling approach for geometry data has not been actively discussed

despite a first step to re-create the texture of a real surface in a virtual environ-

ment is to capture the geometry profile of the surface. Many researchers used a

profilometer (or a similar device) for that purpose. For example, Costa et al. used

an optical profilometer to scan rock surfaces [21], and Wall et al. used a linear

variable differential transformer to measure the displacement of a probe moving

– 11 –

over a surface [22]. Pai et al. developed a wireless haptic texture sensor (WHaT)

equipped with a 3D accelerometer, a 1D force sensor, and a visual marker [23].

As a different approach, Ikei et al. relied on photography to retrieve the height

map of a textured surface [24]. This image-based approach is convenient and

takes less time than the sensorized method, but it has been less popular because

of low modeling resolution and accuracy. A 3D profilometer can be an option to

capture the surface geometry with a very high resolution, however, its high price

and slow scanning speed impede it from being used in haptic texture modeling.

2.3 Vibration and Data-driven modeling and render-

ing

Vibration feedback is also an important feature in haptic texture percep-

tion. In 1995, Kontarinis et al. reported that vibration feedback significantly

improves the user task performance in a teleoperation system [25]. Inspired by

this founding, Okamura et al. presented a measured-vibration based rendering

algorithm [26]. In their method, the rendering of virtual environment was aug-

mented by the vibration data recorded during the exploration of real surfaces

such as tapping, strocking, or cutting with a rigid stylus. By considering the ex-

ploration speed and normal force to the environment as parameters, the authors

constructed a simple deterministic model that can synthesize virtual vibration.

As an example, tapping interaction was modeled by an exponentially-decaying

sinusoidal function. Analysis results also indicated that the vibration amplitude

of texture critically depends on the lateral scanning velocity and normal con-

– 12 –

tact force of the stylus. The author implemented a rendering system with this

vibration-generating model on2D force-feedback joystick. Through the system,

the users were able to discriminate different materials by feeling their virtual

counterparts. In their continued works, they extend their work to 3D force-

feedback device and added friction as an additional component to their model.

Since their modeling method focuses only on synthesizing the resulting vibration

itself, not the physical process underlying it, we can consider their work as an

pioneering work on data-driven haptic modeling and rendering.

Manual surface exploration using a handheld sensorized stylus has been the

prevalent data collection method for data-driven texture modeling. They are

inexpensive and easy to use and also allow us to scan surfaces in free-form.

For example, Pai and Rizun developed a wireless haptic texture sensor (WHaT)

equipped with a 3D accelerometer and a 1D force sensor, all packaged compactly

in a stylus [23]. Using WHaT extended with a visual marker for position tracking,

Lang and colleagues presented a series of studies for data-driven haptic model-

ing and rendering in virtual environments [27, 28, 29]. Their framework considers

shape, compliance, and texture for rendering with a force-feedback interface. The

shape of an object can be modeled using any standard methods. The compliance

is dynamically estimated from a linear relationship between user-applied force

and resulting acceleration. For texture modeling, their key idea is to estimate

a position model from measured acceleration data. Acceleration data are scaled

to remove the effect of applied force and then converted to a height profile using

Velvet integration. This height profile is registered onto the mesh model of the

– 13 –

object based on the scan path. The three models can be rendered using standard

force-feedback rendering algorithms. Using the same measurement method, an

alternate modeling method was presented in [30], where acceleration data are

modeled as they are without conversion to a height profile. Each scan data of

acceleration is automatically segmented so that each segment has one component

of decaying vibration that occurs when the stylus comes across one spatial feature

on the textured surface. Each acceleration segment is modeled using an individ-

ual infinite impulse response (IIR) filter, and then each IIR filter is registered onto

the object mesh. Vibration amplitude was determined using an empirically found

linear function of scan velocity and applied force. Both texture modeling meth-

ods have an advantage that the texture models are registered onto the surface of

an object based on the measured position of the stylus in the world coordinate

frame, which suggests their applicability to anisotropic or even inhomogeneous

textures. However, such extensions have not been explored by the authors yet.

Also, the performance validation through a user study or cross-validation was not

carried out sufficiently.

Another successful approach made by Kuchenbecker and colleagues is based

on linear predictive coding (LPC), and their method generates virtual textures

of very high level of realism in terms of the roughness of isotropic microtex-

tures [1, 31, 32, 33, 34]. In their first work, they demonstrated the feasibility of

data-driven haptic texture modeling using LPC model [1]. As a data-acquisition

apparatus, they used a hand-held tool with a three-axis accelerometer mounted

on top and a rotating drum with different material strips mounted along its ex-

– 14 –

terior for data collection. The data was taken in different tangential speeds and

upward forces. It is shown that an LPC model can be fitted over the texture data.

The model was able to synthesize contact accelerations as a linear combination

of previous acceleration samples, similar to the autoregressive (AR) model. One

advantage of using an LPC model is that it can synthesize the spectral-identical

signal to the vibration signal measured from real materials since it makes use of

the characteristic of Wiener filter [35]. Later, Romano et al. used an interactive

pen display tablet to modeling real texture samples and rendering corresponding

virtual textures [31]. The digital three-axis accelerometer equipped stylus pen

enable capturing normal force, scanning velocity, and high-frequency vibration

while stroking real texture samples. After converting to one-dimensional signals,

each time-domain sample data was mapped into a frequency-domain vibration

using LPC. A tool with two voice-coils was used to render vibrations which were

recreated based on tool’s speed and force. A user study assessing the realism

rating of virtual textures resulted in a mean of 65.4 out of 100, which is quite

reasonable performance. Culbertson used the same interactive pen tablet and

stylus setup for data collection [32]. She extended their previous work to fit an

autoregressive moving average (ARMA) model over the acceleration data. This

type of model was chosen to support the weakly stationary nature of the signals.

In her following work, she improved their modeling method so that it can accept

unconstrained natural moving in data collection for modeling [33]. Their previous

model required (almost) constant scanning velocity and normal force although

the data collection process is carried out using a hand-held device. Finally, they

– 15 –

constructed a haptic texture DB using their method and made their modeling

and rendering software as an open source software [34]. In most of the aforemen-

tioned work, the examined texture samples were general isotropic objects where

scanning directions had no influence on the texture perception. Even in the

huge open-source database of one hundred texture models created by Culbertson

[34], none of the samples have the direction-dependent property. Jeon and col-

leagues tackled this problem by adding two-dimensional moving velocity vector

into the parameters for texture model and applying interpolation among texture

models using radial-basis function (RBF) [36]. The cross-validation simulation

synthesized well-matched signals from the measured data. Further, to reduce the

required number of data to model anisotropic texture, they developed a novel

sample selection algorithm for anisotropic texture modeling [37]. By comparing

the modeling results between a dataset with and without the specific data point,

the author could compute the importance of each input data. As a result, they

could reduce the size of the input dataset to around one-third of the original

dataset, while maintaining the modeling performance.

2.4 Stochastic modeling

One another different approach has been proposed by Siira and Pai [38].

They found that polished surfaces generally shows normally distributed struc-

ture. By assuming the force profile resulted from the microstructure would have

the same mean and standard deviation with the surface’s normal distribution,

the authors generated texture sensation on the 2D surface using Gaussian dis-

– 16 –

tributed lateral force. This assumption is also supported by Green and Salisbury,

who measured and rendered different sandpaper surfaces in terms of remote sens-

ing [39]. By measuring the friction coefficient of different sandpaper surfaces, it

turned out that the histogram of the coefficients shows a Gaussian distribution,

whose standard deviation has an inverse relationship with the size of sandpaper’s

grating. On the other hand, Fritz and Barner tried to synthesis discriminable

virtual textures, not focusing on mimicking real textures [40]. To make virtual

textures, they perturbed deterministic profiles based on multivariate Gaussian

using random process. User study results also showed consistency with the pre-

vious work: virtual textures with larger variance were felt as rougher surfaces.

– 17 –

III. Data-driven Modeling & Rendering of

Homogeneous Texture

This chapter presents a new approach to data-driven modeling of isotropic

haptic textures using frequency-decomposed neural networks from the contact

acceleration data that are captured when a stylus is scanning on a textured

surface with diverse scanning velocities and normal forces. We first describe a

motorized texture scanner that has been developed for accurate and easy data

collection under a wide variety of conditions. We then propose two neural network

models with different topologies: a unified model that feeds all of acceleration

data, scanning velocity, and normal force as input variables to a single large

neural network and a decomposed model that consists of a number of smaller

neural networks each trained with the acceleration data for a pair of scanning

velocity and normal force. At last, we introduce a preceding filtering process to

use a data-driven texture model in our hybrid framework.

3.1 Neural network as a non-linear time-series pre-

dictor

We use neural networks to model haptic textures from contact acceleration

data. Neural networks have proven their excellent performance in predicting

time series data in a wide variety of applications over the years. We expect that

– 18 –

neural networks would have the following specific benefits for texture modeling.

First, real textured materials show nonlinear behaviors, and neural networks as a

nonlinear function approximator may better account for such characteristics than

the linear models used in prior research. Second, the previous methods either

scale the acceleration output or interpolate adjacent models using predefined

rules to synthesize texture signals for untrained scanning velocity or normal force.

Neural networks with the added input of scanning velocity and normal force

allow for learning such interpolation rules also from the data, which may lead to

improved synthesis performance. Third, we can also add scanning direction as

another input variable. This might enable straightforward and seamless extension

to anisotropic texture modeling.

In this chapter, we describe our current progress for two topics. First, we

developed a motorized device for automatic texture scanning for accurate and

easy data collection under different conditions of scanning velocity, normal force,

and scanning direction (Section 3.2). Second, we present two neural network

models with different topologies for isotropic texture modeling (Section 3.3). One

model has a decomposed architecture, resembling the previous methods in that

it consists of a number of neural networks each of which is trained for a pair

of scanning velocity and normal force. The other model has a unified structure

using a single neural network that also takes scanning velocity and normal force as

input. Further, individual neural networks are based on frequency-decomposed

neural networks in order to cope with the widespread spectral distribution of

texture acceleration data. The two models are comparatively evaluated using

– 19 –

real texture samples to find a better model (Section 3.4).

3.2 Data Collection

Manual data collection using a sensor-equipped handheld tool showed good

performance for isotropic texture modeling as reviewed earlier. For anisotropic

textures, however, this approach requires users to explore the textured surface

not only with various scanning velocities and normal forces, but also in different

directions. This is a very demanding task, leading to a high chance of resulting in

poor data sets that contain a large unsampled region in the input space. Machine

scanning can be a solution for this problem although it loses the flexibility of

manual scanning.

We developed a motorized texture scanning device, named Texanner, for

the automatic capture of vibrations. Texanner inherits the general advantages of

mechanical systems, such as high accuracy and repeatability. A pitfall is that the

sensor output of such systems might be tainted with the noise stemming from the

device itself. Texanner lessens this problem by assigning actuation and sensing

to different platforms with minimal mechanical coupling.

Texanner consists of a moving platform and a measurement platform (Fig. 3.1).

They are located on separate tables to prevent noise transfer from the moving

platform to the measurement platform. The moving platform builds on an XY

slider made of two perpendicular linear stages (ET-150-21, Newmark Systems).

Each linear stage uses a high-resolution step motor, and its circular motion is

turned into linear motion using a leadscrew. Each motor is controlled by a mo-

– 20 –

Figure 3.1: Texanner: 2D texture scanning device.

tion controller unit (DMC-21x2, Galil Motion Control). The XY slider has a

15 cm travel distance in each axis with a 20 cm/s maximum speed and a 7.5µm

position resolution.

The measurement platform includes a flat pad for the placement of texture

– 21 –

samples and a stylus with replaceable tips. The stylus is scanned on a texture

sample at a right angle by a rigid arm stretched from the moving platform. The

stylus and the arm are coupled with a linear bearing (LMF8UU, Samick) to reduce

the vertical noise transmitted from the moving platform. Vibrations orthogonal

to the surface account for more than 90% of acceleration information [41]. An

accelerometer (ADXL335, Analog Device) is attached to the lower end of the

stylus to measure texture vibrations. Another accelerometer fastened to the linear

bearing holder monitors the vibrations transferred from the moving platform.

Normal contact force can be varied by changing a weight on top of the stylus.

No active control is used here to preclude actuation noise in the measurements

of normal vibrations.

To evaluate the noise level that can be transmitted from the moving platform,

we collected acceleration data while Texanner scanned a textureless, low-friction

surface of a CD. In such cases, most measured accelerations would originate from

the moving platform. Results are shown in Fig. 3.2. The acceleration values

measured during scanning remained close to the accelerometer’s own noise level,

while the acceleration values of the moving platform itself were much greater.

For data collection with Texanner, a user first sets the normal force by

putting an appropriate weight on top of the stylus. Our GUI-based program

supports several kinds of scan paths composed of line segments. For each line

segment, we use a trapezoidal acceleration profile to capture vibrations occurring

in the constant velocity interval. The user can specify scan path and scanning

velocity, as well as acceleration and deceleration. The default sampling rate for

– 22 –

Figure 3.2: Acceleration recording of a textureless object (CD). Black: accel-

eration from the stylus. Gray: acceleration from the moving platform. In the

Stop mode, the stylus did not move, so the signal came from the accelerometer’s

noise. In the Move mode, the stylus was scanned on the CD surface with different

velocities.

acceleration measurement is 5 kHz.

3.3 Modeling of Isotropic Textures

In this section, we describe our modeling methods using neural networks for

isotropic textures.

– 23 –

Decimator

g = 10

Decimator

g = 5

Decimator

g = 2

Neural network 1

20 – 100 Hz

Neural network 2

100 – 200 Hz

Neural network 3

200 – 500 Hz

Filtered

acceleration

data

(20 – 1000 Hz)

Synthesized

acceleration

signal+

High-pass filter

Cut-off freq. 100 Hz

High-pass filter

Cut-off freq. 200 Hz

…Input

layer

…

Hidden

layersOutput

layer

ො𝑎[𝑛 − 𝑀𝛾]

ො𝑎[𝑛 − (𝑀 − 1)𝛾]

ො𝑎[𝑛 − 𝛾]

…

…𝑖𝑛𝑝𝑢𝑡𝑠

Summation &

Activation

* The nodes in the hidden layers are interconnected.

Interpolator

g = 10

Interpolator

g = 5

Interpolator

g = 2

Neural network 4

500 – 1000 Hz

High-pass filter

Cut-off freq. 500 Hz

𝑎[𝑛]𝑎[𝑛]

Scanning velocity and

contact force( )𝑣, 𝑓

* The blue-dotted line is activated only in case of the unified model Γ𝑈.

Figure 3.3: Structure of ΓDij based on a frequency-decomposed neural network.

3.3.1 Input Variables

We use contact acceleration data in the normal direction to the scanned

surface for texture modeling, preprocessed as follows. First, their sampling rate

is converted from 5 kHz to 2 kHz using a decimator with a factor of 5/2. Second,

the results are filtered using a low-pass filter with a cutoff frequency of 20 Hz.

This procedure results in the acceleration data resampled at 2 kHz with content

– 24 –

in 20–1000 Hz. This time-series acceleration data is denoted by a[n]. Note that

low-frequency components below 20 Hz have a high chance of being contaminated

by noise from various sources [34]. They also have a negligible role in texture

perception. High-frequency components over 1000 Hz exceed the perception band

of the PC (Pacinian) channel and cannot be rendered properly by most vibration

actuators [42].

Our modeling framework of isotropic textures uses the following three input

variables:

• Lateral scanning velocity: v

v ∈ {vi|i = 1, 2, · · · , Nv}, vi < vi+1.

• Normal contact force: f

f ∈ {fj |j = 1, 2, · · · , Nf}, fj < fj+1.

• Normal acceleration sequence: aij [n] (n = 1, 2, · · · , N) for each (vi, fj).

3.3.2 Model Topologies

Using the three input variables, we designed the following two neural network

models with different topologies:

• ΓU: Unified neural network that uses all the three input variables in one

large model.

• ΓD: Decomposed neural network that consists of many smaller neural net-

works in a matrix form of dimension Nv ×Nf . That is, ΓD =[ΓDij

], where

ΓDij is a neural network trained using only aij [n] for each (vi, fj).

– 25 –

ΓD resembles the models used in the previous research [31, 32, 33, 34]. In-

dividual neural networks are pure time-series predictors and relatively simple.

However, the computational cost for modeling is high since NvNf neural net-

works need to be trained. Synthesis requires an interpolation rule between the

outputs of adjacent models. In contrast, ΓU uses a single large neural network to

account for the texture property at once. The internal structure of ΓU is more

complex than that of the individual ΓDij of ΓD.

Which model has superior modeling performance was yet to seen. ΓD us-

ing simpler neural networks may have higher accuracy on training data, but its

cross-validation for untrained scanning velocities and normal forces relies on a

heuristic interpolation method. ΓU learns everything from the data including

the interpolation rule, but its vast input space is a disadvantage against accurate

modeling.

We prefer ΓU if the two models show comparable performance. Not only

its training cost is lower, but also its extension to anisotropic textures is very

simple; it may suffice to add scanning direction as additional input variable.

Extension of ΓD keeping the same structure requires a 3D array of individual

neural networks with the added dimension of scanning direction. This further

increases the already high training cost of ΓD.

3.3.3 Individual Neural Network Structure

For each ΓDij , we use a frequency-decomposed neural network [43] to cope with

the widespread spectral energy of texture vibrations. Accurate modeling of time-

series data using a neural network requires a selection of appropriate sampling

– 26 –

frequency and input size. Consider a time series x[n]. For neural network training,

we make a subsequence of size M by decimating x[n] with a factor of γ, such that

X = (x[n−Mγ], x[n− (M − 1)γ], · · · , x[n− γ]) ,

where x[n] is the output of a low-pass filter applied to x[n] for anti-aliasing

before downsampling (a standard process for decimation). Using this subsequence

X as input, we wish to obtain x[n] as output. Given this set of input and

output, a neural network can be trained effectively using the standard error back-

propagation algorithm [44].

An issue here is that we need to select γ and M accordingly to the spectral

energy distribution of x[n]. For example, γ must be small to capture the high-

frequency behavior in x[n]. If the same γ is used to model the low-frequency

behavior of x[n], M must be a high value for X to contain sufficient low-frequency

information. However, the input size M determines the structural complexity of

a neural network, such as the number of hidden layers and the number of nodes

in each hidden layer. Thus, M is a critical factor for the time and memory

requirements of neural network training and synthesis, and it should remain low.

These conflicts in the selection of optimal γ and M make it difficult to model

wideband time-series data using a single neural network.

A frequency-decomposed neural network solves this problem by using mul-

tiple neural networks, each of which is optimized to a different frequency band.

Fig. 3.3 shows the structure of ΓDij , which includes four internal neural networks

that are in charge of texture modeling in 20–100 Hz, 100–200 Hz, 200–500 Hz, and

500–1000 Hz, respectively. These four frequency bands were determined empiri-

– 27 –

cally. Each internal neural network is preceded by an appropriate decimator and

a high-pass filter. The output of each internal neural network is interpolated (up-

sampled) to restore the sampling rate to 2 kHz. These upsampled accelerations

are summed to make the final output a[n]. The structure of each internal neural

network is also shown in Fig. 3.3.

The unified model ΓU has essentially the same frequency-decomposed struc-

ture, but with two more input variables of scanning velocity v and normal force

f . v and f are fed to each internal neural network shown in Fig. 3.3. This

increases the number of input nodes by only two, but the functional relationship

to learn becomes much more complex.

3.3.4 Error Metric

As an error metric between two acceleration sequences, we use the relative

spectral rms error of a synthesized acceleration sequence a[n] to the measured

sequence a[n]:

Es = es(a[n]) =RMS(F(a[n])−F(a[n]))

RMS(F(a[n]), (3.1)

where F(·) is the operator for discrete Fourier transform (DFT) and RMS is

the operator for computing a root mean square in the frequency domain. This

spectral metric accounts for the perceptual consequences better than the time-

domain counterpart [1, 31, 33]. We use Es for parameter tuning during neural

network training.

Note that no consensus exists at the moment as to the error metrics that best

reflect texture perception. Other metrics, e.g., Hernandez-Andres Goodness-of-

– 28 –

Canvas Corduroy Denim

Scrunched paper Sandpaper Wooden pad

Figure 3.4: Texture samples used for performance evaluation.

Fit Criterion [45], can also be used instead of Es, as in [32].

3.3.5 Synthesis

Acceleration synthesis using ΓU is straightforward. With ΓD, we find four

neighbors models ΓDij such that vi∗−1 < v < vi∗ and fj∗−1 < f < f∗j , obtain

four acceleration values using ΓDij (i = i∗ − 1, i∗ and j = j∗ − 1, j∗), and then

interpolate them using barycentric interpolation.

– 29 –

3.4 Performance Evaluation

For performance evaluation, we collected contact acceleration data using

Texanner from six texture samples shown in Fig. 3.41. A spherical tip shown

in Fig. 3.1 was used for the data collection. Five values were used for scan-

ning velocity and normal force, respectively: v = 4, 6, 8, 10, and 12 cm/s and

f = 0.54, 0.89, 1.18, 1.40, and 1.91 N. Each acceleration data was 2.5-s long. We

trained ΓU and ΓD using the data of v = 4, 8, and 12 cm/s and f = 0.54, 1.18, and

1.91 N. The other data sets for v = 6 and 10 cm/s and f = 0.89 and 1.40 N were

used for cross-validation.

Parameter values used for neural network training were as follows: γ = 10

and M = 50 for all neural networks; 4 hidden layers with 30, 25, 15, and 5 nodes,

respectively, for ΓU and internal neural network 1 in ΓD; and 3 hidden layers

with 30, 15, and 5 nodes for the other internal neural networks in ΓD. These

parameter values were manually tuned using the error metric Es.

We synthesized the acceleration data using the trained ΓU and ΓD for all

combinations of v and f . Fig. 3.5 and Fig. 3.6 present the acceleration signals and

magnitude spectrums of the best and worst examples in terms of cross-validation

performance. Phase spectrums are not shown because haptic perception is in-

sensitive to phase differences [46]. The best-case example (a and b) shows very

similar time-domain signals and magnitude spectrums between the measured and

synthesized acceleration data (Es = 0.26). In particular, the largest peak around

1Among the six texture samples, the corduroy sample is not isotropic. We included this

sample to see the performance for patterned textures. The corduroy sample was scanned across

ridges (e.g., left to right in Fig. 3.4).

– 30 –

0 500 1000 1500 2000 2500

−2

−1

0

1

2

Time (ms)

Am

plitu

de (

m/s

2 )

(a) Best case: Measured,

Time domain

0 500 1000 1500 2000 2500

−2

−1

0

1

2

Time (ms)

Am

plitu

de (

m/s

2 )

(b) Best case: Synthesized,

Time domain

0 200 400 600 800 10000

0.

0.2

0.3

0.4

0.5

0.6

Frequency (Hz)

Am

plitu

de(m

/s2 )

(c) Best case: Measured,

Spectral domain

0 200 400 600 800 10000

0.1

0.2

0.3

0.4

0.5

0.6

Frequency (Hz)

Am

plitu

de(m

/s2 )

(d) Best case: Synthesized,

Spectral domain

Figure 3.5: Acceleration signals and magnitude spectrums of measured and

synthesized acceleration data in the best case(Corduroy, ΓU, v = 6 cm/s,

f = 1.40 N). Dark navy lines show original magnitude spectrums, and pale pink

lines represent smoothed spectrums using a moving average filter.

– 31 –

0 500 1000 1500 2000 2500

−1.5

−1

−0.5

0

0.5

1

Time (ms)

Am

plitu

de (

m/s

2 )

(a) Worst case: Measured,

Time domain

0 500 1000 1500 2000 2500

−1.5

−1.0

−0.5

0

0.5

1.0

Time (ms)

Am

plitu

de (

m/s

2 )

(b) Worse case: Synthesized,

Time domain

0 200 400 600 800 10000

0.002

0.004

0.006

0.008

0.01

0.012

0.014

Frequency (Hz)

Am

plitu

de (

G)

(c) Worst case: Measured,

Spectral domain

0 200 400 600 800 10000

0.02

0.04

0.06

0.08

0.10

0.12

Frequency (Hz)

Am

plitu

de (

m/s

2 )

(d) Worst case: Synthesized,

Spectral domain

Figure 3.6: Acceleration signals and magnitude spectrums of measured and syn-

thesized acceleration data in the worst case(Scrunched paper, ΓD, v = 6 cm/s,

f = 0.89 N). Dark navy lines show original magnitude spectrums, and pale pink

lines represent smoothed spectrums using a moving average filter.

– 32 –

550 Hz and other smaller peaks are well preserved in the synthesized spectrum.

In the worse-case example (c and d), both the time-domain signals and the mag-

nitude spectrums showed some level differences (Es = 0.71).

Further, we computed Es between the measured and synthesized accelera-

tion data for all combinations of v and f using ΓU and ΓD. Results are shown

as box plots in Fig. 3.7 for all texture samples. (a) and (b) are from the ac-

celeration data used for training. ΓD showed better performance than ΓU for

all the texture samples except wood, although their means were very similar

(Es = 0.373 and 0.353, respectively). In (c) and (d) where cross-validation re-

sults are provided, ΓU outperformed ΓD for all the samples (Es = 0.33 and 0.43,

respectively).

Comparing (a) and (c) in Fig. 3.7 suggests that the prediction performance

of ΓU for the untrained data (Es = 0.30− 0.39) was comparable to that for the

training data (Es = 0.32 − 0.42). The cross-validation performance was even

better in some samples (corduroy, scrunched paper, sandpaper, and wood pad).

For ΓD, however, the cross-validation performance was clearly lower; compare

(b) and (d) in Fig. 3.7 (Es = 0.26 − 0.38 and Es = 0.32 − 0.54, respectively).

These results indicate that barycentric interpolation was not very effective in

explaining the inter-model behaviors between the individual neural networks in

ΓD. It seems that ΓU succeeded in capturing the interpolation rules from the

data.

Lastly, we carried out a two-way ANOVA using neural network model and

texture sample as independent factors to validate their effects on Es. This was

– 33 –

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Canvas Corduroy Denim Paper Sandpaper Wood

Rel

ativ

e R

MS

err

or

(a) Training data. ΓU

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Canvas Corduroy Denim Paper Sandpaper Wood

Rel

ativ

e R

MS

err

or

(b) Training data. ΓD

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Canvas Corduroy Denim Paper Sandpaper Wood

Rel

ativ

e R

MS

err

or

(c) Cross-validation. ΓU

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Canvas Corduroy Denim Paper Sandpaper Wood

Rel

ativ

e R

MS

err

or

(d) Cross-validation. ΓD

Figure 3.7: Relative spectral rms errors Es.

– 34 –

done separately for the training data and the cross-validation data. When the

training data were used, ΓD showed significantly better performance than ΓU

(F (1, 8) = 12.75, p = 0.0073, while texture sample did not have significant effects

on Es(F (5, 40) = 1.87, p = 0.1220). In the case of cross-validation, however, ΓU

showed significantly better performance than ΓD (F (1, 3) = 17.10, p = 0.0310),

but texture sample did not have significant effects (F (5, 15) = 2.50, p = 0.0770).

The above evaluation results allow us to make the following conclusions: (1)

ΓD showed better modeling performance than ΓU when tested with the data used

for training. Note that this corresponds to the best-case performance in synthe-

sis; (2) ΓU showed better modeling performance than ΓD in cross-validation for

all the texture samples used. This cross-validation performance is close (but not

exactly) to the worst-case in synthesis; and (3) The cross-validation Es of ΓU

ranged from 0.30 to 0.39. This is similar to the error range (0.29–0.42) reported

in [1] using the same error metric and similar texture samples, which later en-

abled good perceptual realism ratings [31]. Therefore, we expect that our unified

texture model ΓU would be able to provide similarly realistic virtual textures.

3.5 Application to Inhomogeneous Texture

In this subsection we introduce a preceding filtering process to use a con-

tact acceleration based model in our hybrid framework. The contact acceleration

data collected from inhomogeneous textures contains non-stationarity compo-

nents induced by geometrical inhomogeneity. To apply a data-driven model to

the collected data, we filter out the non-stationary components of the vibrotactile

– 35 –

signal induced by the inhomogeneity of the texture. That non-stationarity is not

required in the contact vibration model as these components are enclosed in the

micro-geometry of the surface. The objective of the contact-acceleration model

in our framework is to model the stationary contact acceleration induced by fine

textural features.

To this end, we apply a high-pass filter to remove non-stationarity from

the inhomogeneity of texture. The cut-off frequency of the high-pass filter is

determined by trial-and-error. We find the smallest cut-off frequency for which

the resulting filtered signal passes the augmented Dickey-Fuller test, which is

widely used to test the stationarity of a given signal. The cut-off frequency

determinations vary based on the texture, within a 50 – 150 Hz range.

3.6 Conclusions

Data-driven modeling and rendering of surface textures using contact accel-

eration data is an effective method for enriching the haptic sensations of virtual

or augmented objects with improved realism. In this chapter, we present a new

approach based on frequency-decomposed neural networks and assessed its per-

formance using two models in unified and decomposed architectures. According

to our experiment using real texture samples, the unified model outperforms the

decomposed model, and it is comparable in modeling performance to the best

available in the literature. A motorized texture scanner we developed was instru-

mental for accurate and easy data collection with anisotropic textures. The major

contributions of this chapter are with the new data-driven modeling scheme for

– 36 –

isotropic textures using frequency-decomposed neural networks and the demon-

stration of its applicability to homogeneous textures.

– 37 –

IV. Geometry-based Modeling & Rendering

of Inhomogeneous Texture

This chapter presents an improved approach to geometry-based haptic tex-

ture modeling and rendering. We adopt photometric stereo, one of the most

accurate algorithms for 3D surface reconstruction, to increase the resolution of

captured geometry profiles. This benefit of higher texture resolution can enhance

the realism of rendered textures in terms of roughness. To this end, we have

designed and constructed a dome-shaped lighting structure for use in the mod-

eling using photometric stereo. With this apparatus and the photometric stereo

algorithms, we can achieve very high texture modeling resolution in the order

of 10µm. We also identify the stiffness and friction of real materials using the

Hunt-Crossley model and the Dahl model, respectively, for realistic texture ren-

dering. A user study measuring the perceived similarity between real and virtual

textures demonstrated that our system can achieve a reasonably high level of

realism in rendering textured objects with high compliance or low friction.

4.1 Geometry-based modeling and our approach

We use photometric stereo algorithms to model the micro-geometry of a

real object surface. This approach is free from any restrictions, such as isotropy

and homogeneity. Although there are some image-based haptic texture model-

– 38 –

ing methods (Section 2.2), they have been less popular because of low modeling

resolution and accuracy. A 3D profilometer can capture the surface geometry

with a very high resolution in the order of 10−6 mm, but they are not very ap-

propriate for haptic texture modeling because of their very expensive price and

slow scanning velocity in the order or 10−3 mm/s. Such high resolutions are also

an overkill to force-feedback rendering considering the resolution of most haptic

devices.

Highly accurate acquisition of object geometry using multiple images is still

a challenging problem in computer graphics. Among others, photometric stereo

is regarded as one of the most accurate methods [47]. In photometric stereo, the

surface geometry is estimated by correlating the correspondence among different

images that are photographed under precisely controlled lighting conditions us-

ing a fixed camera. We achieve such lighting conditions using a lighting dome

designed to have a structure appropriate for haptic texture modeling. Multiple

photographs taken using the lighting dome are fed to our haptic texture model-

ing algorithms to build the height map of the target surface. We also identify

the stiffness and friction of the texture sample using two parametric model, the

Hunt-Crossley model and the Dahl model, respectively. We then integrate the

three models to re-create virtual texture with high resolution and realism. Lastly,

we verify the performance of our modeling and rendering method by a user study.

The contributions this chapter are with 1) the image-based haptic texture

modeling method with high resolution 2) the combination of the data-driven

model and the physical parametric models of stiffness and friction, and 3) the

– 39 –

evaluation of our modeling and rendering method in terms of realism, i.e., simi-

larity between real and virtual textures.

4.2 Height Map Estimation

General photometric stereo algorithms in computer graphics require dedi-

cated and expensive apparatus to cover a large workspace and support objects of

arbitrary shapes. For haptic textures, however, we can assume relatively small

and almost flat texture samples. We also use an ordinary digital camera for better

reproducibility [48].

4.2.1 Apparatus

To accurately locate lighting instruments, we designed and built a dome-

shaped lighting structure (Fig. 4.1). Placing multiple light sources at the fixed

locations is effective for the precise control of lighting conditions, which is the

most important requirement of photometric stereo. To this end, lighting-dome

structures are commonly used for photometric stereo algorithms. Unlike other

lighting domes, ours is much smaller, and the camera is fixed at the apex of the

dome to focus on almost flat texture materials.

The detailed design of our lighting dome is based on the relationship be-

tween illumination angle and reconstruction error. According to an empirical

experiment [49], modeling error is the lowest when the lighting source has the

illumination angle (elevation) of 55◦ for general surfaces, but the error rate is

almost the same between 40◦ and 70◦. Therefore, our design places light sources

at three elevations of 40◦, 55◦, and 70◦. This configuration is repeated for every

– 40 –

Textured Material

Digital camera

Figure 4.1: Apparatus for texture modeling. LEDs (marked by orange circles)

are installed inside the polycarbonate dome.

36◦ in azimuth, resulting in 30 different lighting conditions. We use power LEDs

(1W), accurately controlled by a driver circuit, to provide sufficient illumination.

4.2.2 Photometric Stereo Algorithm

We use a photometric stereo algorithm in [48] to construct a height map.

This method works well with a regular DSLR camera and multiple flashlights.

Before constructing a height map, we need to estimate the radiance (incident

light intensity) function L(x, y) at position (x, y) on the surface. For this, we use

a blank white paper as the surface material. This allows us to assume that the

normal vector and the albedo (reflection coefficient) are constant over the surface.

– 41 –

Therefore, a photographed image of the paper represents the radiance function.

Using the radiance function L(x, y), the photographed light intensity I(x, y)

can be represented by

I(x, y) = β(x, y)L(x, y), (4.1)

where β(x, y) is a BRDF (Bidirectional Reflectance Distribution Function). The

BRDF is a function that defines the light reflectance and depends on the surface

parameters such as albedo, normal vector, incident light vector, and view vec-

tor. Many different models exist for BRDF, and we use the Lambertian model

since it is effective for perfectly diffuse surfaces1 in spite of its simple form [48].

Specifically,

β(x, y) = a(x, y)n(x, y)T l(x, y), (4.2)

where a is the albedo, n is the normal vector, and l is the incident light vector. l

is determined using the lighting dome. Thus, a and n are the only unknowns in

(4.1) and (4.2), and they need to be determined at each point (x, y).

For robust estimation of a and n, we use multiple photographs taken under

N lighting conditions (N = 30). Then the problem reduces to an optimization

problem as follows:

(a(x, y),n(x, y)) =

argminN∑i=1

∣∣Ii(x, y)− a(x, y)n(x, y)T li(x, y)L(x, y)∣∣2 . (4.3)

To solve this optimization problem, we initialize n = (0, 0, 1) and then find a

using singular value decomposition (SVD). Then a new n is computed with this

1Diffuse reflection occurs on a non-glitter and irregular surface, which is the main target of

haptic texture modeling and rendering.

– 42 –

(a) Wood

(b) Denim

Figure 4.2: Examples of real materials and reconstructed height maps. Details

of each material are well preserved.

a using SVD. This procedure is terminated until the changes in n and a become

negligible. In most cases, twenty iterations are sufficient for the convergence.

Finally, the normal vectors n(x, y) are integrated over the surface to con-

struct the height map h(x, y) [50].

Fig. 4.2 shows two examples of real materials and their reconstructed height

maps. Since our modeling relies on imaging, it can capture any variations of

– 43 –

surface irregularities; e.g., see an anisotropic and inhomogeneous texture in Fig.

4.2a. Empirical observation of the modeled height maps indicates that the reso-

lution of our height map models is around 10µm.

4.3 Friction and Stiffness Modeling

The friction and stiffness of a material are also major components that affect

texture sensation. They should also be captured from the real material for realis-

tic re-creation of haptic textures. For that, we use two parametric models: Dahl

model [51] for friction and Hunt-Crossley model [52] for stiffness. Texture sam-

ples used for modeling are usually thin and allow only shallow deformation. They

show much simpler viscoelastic and frictional responses than general deformable

objects with sufficient volume. Thus, simpler parametric models are more ade-

quate than data-driven non-parametric models [53], which are more suitable for

large objects with complex viscoelastic properties. Therefore, our framework is

hybrid; texture modeling is data-driven while friction and stiffness modeling are

parametric.

4.3.1 Data Collection

We use a force-feedback device (PHANToM Premium 1.5 High Force, Ge-

omagic) to capture the position of its stylus with high resolution when a user

strokes a textured surface. A 3D force/torque sensor (M3701A, Sunrise Instru-

ments) is mounted between a custom aluminum tooltip and the PHANToM sty-

lus. The experimenter strokes a textured surface using the stylus to collect fric-

tion data and pokes it for stiffness data. The modeling processes run concurrently

– 44 –

with data collection. They are finished when the gradients of model parameters

become smaller than pre-defined thresholds. This parameter identification pro-

cedure takes 30–60 s (50–100 repetitions) with our system.

4.3.2 Dahl Model Identification

The Dahl friction model has been frequently used in haptics applications.

This simple but useful model can accurately portray the behavior of various real

frictional responses with a reasonable complexity. The original form of Dahl

model is given by

dfDdx

= κ

(1− fD

fCsgn(v)

)α, (4.4)

where fD is the frictional force, x is the tangential displacement, v is the tangen-

tial velocity, κ is the stiffness coefficient, and α is a constant exponent. fC is the

Coulomb friction force, such that

fC = µkfnorm, (4.5)

where µk is the Coulomb friction coefficient and fnorm is the normal load. For

easier calculation, we employ a discrete-time representation of the Dahl model

[54], such that

fD(t+ 1) = fC(t)sgn(v(t))+

(fD(t)− fC(t)sgn(v(t)))e− κfC (t)

|x(t)−x0|, (4.6)

where x0 is the initial displacement that is reset to x(t) when v(t) becomes zero.

Since the original Dahl model is focused to model the near-static friction, the

original Dahl model does not consider the tangential velocity. To add the effect

– 45 –

of the dynamic friction, we add a simple viscous term after we finish calculating

the Dahl model friction as other researchers did [55]. Then the total friction fF (t)

becomes

fF (t) = µbv(t) + fD(t)

= µbv(t) + fC(t− 1)sgn(v(t− 1))+

(fD(t− 1)− fC(t− 1)sgn(v(t− 1)))e− κfC (t−1)

|x(t−1)−x0|, (4.7)

were µb is the viscous friction coefficient.

Data for parameter identification are collected by manually stroking the

target surface with the PHANToM stylus and collect resulting forces. Assuming

the target surfaces are flat, the force readings are divided into normal load fnorm

and tangential friction force fF .

We identify the parameters in (4.7) separately according to the movement

state (pre-sliding and sliding), as shown in Algorithm 1. When the tangential

velocity of the stylus is below Tv1 = 0.5 mm/s, we consider that the system is in

the pre-sliding regime. In this regime, the tangential displacement |x(t) − x0| is

very small, and (4.7) can be approximated to a linear function of κ and fD(t) by

using the Taylor expansion of the exponential term. Then (4.7) is simplified to

fF (t) ' µbv(t) + fC(t− 1)sgn(v(t− 1))

+ (fD(t− 1)− fC(t− 1)sgn(v(t− 1)))

·(

1− κ

fC(t− 1)|x(t− 1)− x0|

), (4.8)

If we replace fD(t − 1) with fF (t − 1) − µbv(t − 1) and fC(t − 1) with

– 46 –

µkfnorm(t− 1), then we can express κ as

κ =µkfnorm(t− 1)

|x(t− 1)− x0|

· fF (t)− fF (t− 1)− µbv(t) + µbv(t− 1)

sgn(v(t− 1))− fF (t− 1) + µbv(t− 1). (4.9)

Since we can read the values of fF , fnorm, v from the force sensor and the

PHANToM, we use this equation to (4.9) to return κ in the estKappa function

in Algorithm 1.

In the sliding regime where the tangential velocity exceeds Tv2 = 3 mm/s,

the exponential term e− κfC (t)

|x(t)−x0| in (4.7) quickly converges to zero. In that

case, the friction force can be approximated by the sum of the Coulomb friction

and the viscous friction as follows:

fF (t) ' µbv(t) + µkfnorm(t− 1)sgn(v(t− 1)). (4.10)

This equation is in the estMu function in Algorithm 1 to identify µk and µb by

applying the least square method.

The algorithm stops the repetition when the changes of all the three param-

eters become lower than TC = 10−9.

4.3.3 Hunt-Crossley Model Identification

The Hunt-Crossley model has a form of

f(t) = Kxn(t) +Bxn(t)x(t), (4.11)

where K and B are the stiffness and damping parameters of the object, x(t) is

the normal displacement, and n is a constant exponent between 1 and 2. This

– 47 –

Algorithm 1 The friction parameter estimation algorithm

1: procedure Estimator(κ0, µb0, µb0, Tv1, Tv2, TC)

2: κ← κ0

3: µk ← µk0

4: µb ← µb0

5: repeat

6: read current v, x, x0, fnorm, fF

7: if v ≤ Tv1 then

8: κ← estKappa(v, x, x0, fnorm, fF , µk, µb)

9: ∆κ← κ− κ10: κ← κ

11: else if v ≥ Tv2 then

12: µk, µb ← estMu(v, x, x0, fnorm, fF , κ)

13: ∆µk ← µk − µk14: ∆µb ← µb − µb15: µk ← µk

16: µb ← µb

17: end if

18: until ∆κ,∆µk,∆µb < TC

19: return κ, µk, µb

20: end procedure

– 48 –

model has been successful in capturing deformation behavior for haptics [56, 57]

owing to its balanced performance between modeling accuracy and the ease of

parameter identification.

For parameter identification, we use the iterative estimation algorithm pro-

posed by Haddadi and Hashtrudi-Zaad [58]. We supply input data to the iden-

tification process by repeatedly pushing the target material downward with the

PHANToM stylus. Since we try to make no tangential movement in this process,

the resulting forces can be assumed to be generated from only the stiffness. In

addition, we discard the input samples that have higher force amplitude than 2 N

because we only want to model the deformation of fine texture geometry near the

surface, not the whole deformation of the object. Users usually exert no more

than 2 N of normal force when they stroke an object surface [59], so we use this

value as the cut-off threshold of input samples.

4.4 Texture Rendering

Given a real texture sample, we obtain a geometry model, a friction model,

and a stiffness model as described in Section 4.2 and 4.3. We then combine

the three models to re-create a virtual textured surface similar to the real one.

For rendering, we use a PHANToM Premium 1.5 High Force model for its high

nominal position resolution (0.007 mm). Our geometry models have very high

positional resolution on the order of 10µm, and other haptic devices with low

position resolutions are not suitable.

We assume that a flat, thin texture material is overlaid on a stiff base object.

– 49 –

Under this assumption, we can calculate the reaction force F by decomposing it

to three components:

~F = ~Ft + ~Fb + ~Ff , (4.12)

where ~Ft is the stiffness force perturbed by the texture, ~Fb is the stiffness force

of the base under the texture, and ~Ff is the friction force between the tool tip

and the surface.

The texture’s stiffness force ~Ft is formulated as

~Ft =(Ktd

ntt +Btd

ntt dt

)~N(~x), (4.13)

where Kt, Bt, and nt are the Hunt-Crossley model parameters of the texture

material, ~N(~x) is the texture normal vector at the horizontal position ~x, and dt

is the penetration depth into the texture material. dt is computed by

dt =

0, z > h(~x)

|h(~x)− z|, 0 < z < h(~x)

h(~x), z < 0

(4.14)

where ~x and z are the horizontal and vertical position of the tool tip, and h(~x)

is the height of the texture material at ~x.

Similarly,

~Fb =(Kbd

nbb +Bbd

nbb db

)z, (4.15)

where Kb, Bb, and nb are the Hunt-Crossley model parameters of the base object,

and z is the vertical unit vector.

We perturb only Ft to simulate the texture, and Fb always points upward

from the surface. The magnitude of Ft is limited since the maximum penetration

– 50 –

depth into the textured surface is bounded by the thickness of the material. This

allows us to improve the stability of texture rendering since the directional change

of F also remains bounded for deep penetration of the tool. Furthermore, the

user can perceive the thickness of the surface by an abrupt change in the stiffness

coefficient from the texture material to the stiff base if the object is soft enough.

This also occurs in reality when we stroke a thin, soft object.

For ~Ff , we use the discretized version of Dahl model in (4.7). We use | ~Ft+ ~Fb|

as the magnitude of normal force fnorm in calculating the Dahl friction model.

This is to associate the magnitude of ~Ff with the height change of texture.

4.5 User Study 1: Assessing the realism of the virtual

textures

We evaluated the performance of our texture modeling and rendering algo-

rithms by means of a user study. The emphasis was on the perceptual similarity

between virtual and real textures. This user study was approved by the Institu-

tional Review Board at the author’s institution (PIRB-2017-E070).

4.5.1 Methods

Participants

We recruited twenty participants (ten males and ten females; 20–28 years old

with an average of 22.7; all right-handed) who had no experiences in using haptic

interfaces. None of them reported any known sensory or motor impairment.

Participants signed an informed consent form after we explained the goals and

– 51 –

Denim Velvet Scrunchedpaper

Sponge Cork

Aluminumplate

Rubber mat

Towel Acryl MDF

Figure 4.3: Ten materials used in the user study. Three materials that showed the

lowest scores in the pilot experiment are highlighted with blue-dashed squares.

Note that some materials (denim, rubber mat, scrunched paper, towel, and

sponge) have nonhomogeneous textures.

procedure of the experiment. Participants were paid KRW 10,000 (approximately

USD 9) after the experiment.

Texture Materials

We applied the modeling and rendering methods described in Section 4.2–

4.4 on eighteen texture materials. We performed a pilot experiment to assess the

similarity between real and virtual textures. Based on the results of the pilot

experiment, we selected seven materials with the highest similarity scores as a

promising group (acryl, cork, denim, sponge, rubber mat, towel, and velvet) and

three materials with the lowest scores as an unpromising group (aluminum plate,

MDF, and scrunched paper). The main experiment was carried out with these

ten texture samples, which are shown in Fig. 4.3.

– 52 –

Performance Measures

Given a pair of haptic textures, participants evaluated their similarity for

each of the following four criteria: friction, hardness, roughness, and overall sim-

ilarity. The first three are the most salient perceptual dimensions of texture

perception [60]. Our framework models the relevant physical properties indi-

vidually and then renders them together using a force-feedback interface. Each

measure was rated in a scale of 0–100.

Task and Procedure

During the experiment, participants sat in a chair in front of a desk on

which a monitor and a PHANToM 1.5 High Force were placed (Fig. 4.4). A real

texture material was laid on a plastic stand beside the PHANToM. We tried to

(a) (b)

Figure 4.4: Experimental setup. (a) Screenshot of the program. Two white

squares represent the locations of two textures. The black sphere indicates the

position of the PHANToM stylus. (b) Physical environment. A barrier (indicated

by a semi-transparent plane) blocked the participant’s view.

– 53 –

remove all irrelevant sensory cues. Participants’ vision to the PHANToM and

texture samples was blocked by a screen. Instead, rectangles were displayed on

the monitor screen along with an indicator showing the stylus position (Fig. 4.4a).

Sound occurring during participants’ interaction with textures was canceled off

using headphones playing white noise.

The experiment consisted of two phases. In the first phase, participants

were provided with pairs of only real texture samples and were asked to rate

their similarity. This was to measure the upper and lower bounds of each perfor-

mance measure since humans tend to avoid giving extreme values (e.g., see [56]).

Additionally, this phase helped participants stabilize their perceptual bases and

scales for similarity scores. Participants were randomly presented with 15 pairs

of materials, ten of which with the same materials including both the promising

and unpromising groups and the other five with different materials giving clearly

different textures. Participants could freely explore the real texture samples while

touching them by moving the PHANToM (Fig. 4.4b). The PHANToM had a

round tip (radius 5 mm), and this tip was in contact with the real samples.

In the second phase, participants evaluated the similarity between a virtual

texture and its corresponding real texture. Their positions were randomly shuf-

fled. The order of the ten texture pairs was randomized for each participant.

Participants could freely explore the real and virtual textures. The maximum

force output of the PHANToM was restricted to be 10 N to ensure rendering

stability.

Participants were allowed to take a break whenever necessary. On average,

– 54 –

the experiment took about 50 minutes for each participant.

4.5.2 Results

During the experiment, we collected 2000 similarity scores (20 participants

× (15 + 10) pairs × 4 measures). Significant outliers were detected and discarded

using box plots.

Fig. 4.5 shows the means of the four similarity measures for the ten real-

virtual texture pairs and for the upper and lower bounds obtained from the data

of real-real texture pairs. Blue bars indicate the results of the promising group,

green bars those of the unpromising group, and yellow bars the upper and lower

bounds found by comparing between real materials. Materials marked with A

are significantly different from the lower limit, but not from the upper limit.

Materials those marked with B are significantly different from the upper limit,

but not from the lower limit. The materials marked with C showed no significant

differences from the lower limit nor the upper limit. Red error bars represent

standard error. The upper and lower limits were all similar (overall: 77.99 and

35.41, roughness: 75.97 and 35.55, and friction: 74.84 and 37.10), except those

of stiffness (85.52 and 38.15), which were slightly higher than the others. The

means and standard deviations of the four measures are also listed in Table 4.1.

The table also show the relative ratios of similarity scores to their upper limits.

The similarity scores did not pass the Shapiro-Wilk normality test. Thus,

we performed a non-parametric Kruskal-Wallis test to examine the statistical

differences among the materials. Results showed that the material had a statis-

tically significant effect on all the four measures (overall: χ2(11) = 126.86, p <

– 55 –

0

20

40

60

80

100

Ove

rall

Sim

ilarit

y Sc

ore

Acryl Cork Denim Sponge Rubbermat

Towel Velvet Aluminumplate

MDF Scrunchedpaper

Lowerlimit

Upperlimit

A B B B A A A B B B

(a) Overall similarity scores

0

20

40

60

80

100

Stiff

ness

Sim

ilarit

y Sc

ore

Acryl Cork Denim Sponge Rubbermat

Towel Velvet Aluminumplate

MDF Scrunchedpaper

Lowerlimit

Upperlimit

C A B B A A A B B B

(b) Stiffness similarity scores

0

20

40

60

80

100

Rou

ghne

ss S

imila

rity

Scor

e

Acryl Cork Denim Sponge Rubbermat

Towel Velvet Aluminumplate

MDF Scrunchedpaper

Lowerlimit

Upperlimit

A C C B A A A B B B

(c) Roughness similarity scores

0

20

40

60

80

100

Fric

tion

Sim

ilarit

y Sc

ore

Acryl Cork Denim Sponge Rubbermat

Towel Velvet Aluminumplate

MDF Scrunchedpaper

Lowerlimit

Upperlimit

A B C CB B C B B B

(d) Friction similiarity scores

Figure 4.5: Average similarity scores with standard errors. Blue, green, yellow

bars represent the promising group, the unpromising group, and the upper and

lower bounds respectively. A: significantly different from the lower limit, but not

from the upper limit. B: significantly different from the upper limit, but not from

the lower limit. C: not significantly different from the lower limit nor the upper

limit.

– 56 –

Table 4.1: Means and standard deviations of the similarity scores.

Materials Promising group Unpromising group All

OverallM: 56.26 (0.72) M: 23.43 (0.30) M: 46.41 (0.60)

STD: 25.39 (0.33) STD: 18.26 (0.23) STD: 27.87 (0.36)

StiffnessM: 66.25 (0.77) M: 34.00 (0.40) M: 56.58 (0.66)

STD: 25.18 (0.29) STD: 26.52 (0.31) STD: 29.51 (0.35)

RoughnessM: 57.36 (0.76) M: 22.73 (0.30) M: 46.97 (0.62)

STD: 28.97 (0.38) STD: 21.55 (0.28) STD: 31.25 (0.41)

FrictionM: 49.82 (0.67) M: 22.52 (0.30) M: 41.63 (0.56)

STD: 27.88 (0.37) STD: 23.82 (0.32) STD: 29.47 (0.39)

* The parenthesized values are the ratios of the preceding values to their

upper limits.

0.001; stiffness: χ2(11) = 92.2, p < 0.001; roughness: χ2(11) = 113.17, p < 0.001;

friction: χ2(11) = 94.91, p < 0.001).

We then ran the Dun-Bonferroni multiple comparison tests to the effect of

material. Test results allowed us to classify the materials into three groups of A,

B, and C. The materials in group A showed a significant difference from the lower

limit, but not from the upper limit. Their rendering performance is relatively

good. Inversely, the materials in group B were significantly different from the

upper limit, but not from the lower limit. So their rendering performance is

low. Group C showed no significant difference from the upper limit nor the lower

limit. Therefore, we can regard the materials in group C as having moderate

performance. No materials showed significant differences from both the upper

and lower limits.

Based on the overall similarity score, acryl, rubber mat, towel, velvet were

classified to group A. Cork, denim, sponge, aluminum plate, MDF, and scrunched

– 57 –

Table 4.2: Correlation coefficients of the overall similarity score with different

measures.

Measures Correlation coefficient

Roughness similarity score 0.87

Stiffness similarity score 0.78

Friction similarity score 0.81

Minimum of the similarity scores 0.96

Maximum of the similarity scores 0.83

Mean of the similarity scores 0.90

paper were sorted into group B. No samples belonged to group C. Fig. 4.5 also

shows the grouping results of the other three measures.

4.5.3 Discussion

The results of the user study revealed the advantages and shortcomings of our

texture modeling and rendering method. The four materials (acryl, rubber mat,

towel, and velvet) in group A showed high overall similarity scores comparable to

the upper limit (Fig. 4.5a). These results indicate that our method is appropriate

for replicating textures with low friction (acryl; Fig. 4.5d) or with low stiffness

(rubber mat, towel, and velvet; Fig. 4.5b).

The three materials in the unpromising group received very low overall simi-

larity scores as observed in the pilot experiment (Fig. 4.5a) and were all classified

into group B. The two materials (aluminum plate and MDF) have very high stiff-

ness, and this result is likely to due to the fact that such high stiffness models

are almost impossible to capture or render using an impedance-type haptic in-

terface. Scrunched paper is not as stiff, but all the three materials produce crisp

– 58 –

vibrations when they are scanned with a rigid tool. However, the force-feedback

haptic interface used (PHANToM) does not have such a high bandwidth, and

it is unable to generate such salient vibrotactile stimuli even if very fine textu-

ral variations are preserved in the photometric stereo-based texture models. In

fact, some participants emphasized that no crisp vibrations existed in the virtual

textures compared to the real textures.

Among the rest three materials (cork, denim, and sponge) that were in the

promising group but belonged to group B (Fig. 4.5a), sponge has particularly

different physical characteristics. Although the sponge showed a moderate fric-

tion and a high compliance, it has a porous structure. Many holes on the sponge

surface are not represented very well in our texture model; especially the modeled

depths are shallower. This could have caused a problem in re-creating the sensa-

tions of scanning on the holes in virtual rendering. Reasons for the low similarity

scores of cork and denim are not clear at the moment, except that their friction

model identification was not very successful (Fig. 4.5d). We are looking into this

issue.

Another interesting observation is that the four similarity scores showed sim-

ilar trends even though each physical property (texture geometry, stiffness, and

friction) was modeled independently. The perceptual effects of the three physical

dimensions seem highly correlated. Moreover, it also appears that the worst-

modeled physical property dominates the perceived realism. To examine this hy-

pothesis, we calculated the correlation coefficients between the overall similarity

and six other different metrics—the stiffness similarity, the roughness similarity,

– 59 –

the friction similarity, the minimum of the three similarities, the average, and the

maximum. The correlation coefficients are shown in Table 4.2. The correlations

were generally high (0.78–0.96). The correlation with the minimum similarity out

of the three individual similarities was the highest with an exceptionally strong

correlation (0.96). This agrees with our hypothesis and is an interesting research

topic for more careful validation. Among the three similarity scores, the corre-

lations were the highest with roughness (0.87), followed by friction (0.81) and

then stiffness (0.78). This result reconfirms that both stiffness and friction are

important in texture perception.

As an afterthought, also measuring the perceived similarity of texture ge-

ometry in the experiment could have contributed to elucidating the perceptual

weights of the three physical properties, texture geometry, stiffness, and friction,

in the perceptual realism of virtual textures. We will consider this point in our

next iteration.

4.6 User Study 2: Comparision btw. two modalities

In this user study, we compare the perceptual performance between the two

major approaches, that is, force vs. vibration, of haptic texture modeling and

rendering. We expect that the two modalities have different advantages and dis-

advantages as to realistic texture rendering. To our knowledge, no results of

such comparative perceptual analysis are available in the literature. A recent ex-

ception is [61], where the authors investigated the effects of friction, vibrational

contact transient, and vibration texture on realism individually and combinato-

– 60 –

Denim Velvet Cork

Rubber

mat

Towel Acryl

Bamboo

mat

Coated

Hardboard

WoodTile

Figure 4.6: Ten real materials used in the user study.

rially. Our work is in the context of how to combine the two modalities to render

more complete haptic surfaces.

We chose the photometric stereo-based method proposed in this section and

the LPC-based method as representative method of two haptic cues because they

are known to provide the best realism—the highest similarity to real textures.

Therefore, they are suitable for studying the relative roles of force and vibration

cues for virtual texture perception. In this user study, we comparatively assess

the similarity of the virtual textures rendered using the two methods to their real

counterparts. This study was approved by the IRB of the author’s institution.

4.6.1 Methods

We used ten textured materials shown in Figure 4.6, selected for diversity. A

half of them (bamboo, cork, rubber mat, tile, and wood) had visible fine geometric

features while the others did not. Their stiffness and friction characteristics were

also widely different.

Participants evaluated the perceptual similarity between real and virtual

– 61 –

(a) (b)

Barrier

Haptic Device

Texture Sample

The position of the haptic device’s stylus

The locations of two textures

Figure 4.7: Experimental setup. (a) Physical environment. (b) Screenshot of the

program.

textured surfaces using five criteria: geometry, roughness, hardness friction, and

overall similarity. The first four measures are the most salient dimensions of tex-

ture perception [60]. They were chosen to examine the effects of texture rendering

modality on individual perceptual dimensions. Each measure was rated in a scale

of 0–100 (0: completely different and 100: identical). Strohmeier and Hornbæk

used example objects to define the meanings of the measures [62]. In this work,

we used only haptic expert participants to maintain the same understanding.

The meanings of the measures were fully explained to participants before the

experiment.

Participants sat in a chair in front of a desk on which a monitor and the

Omega.3 device were placed (Figure 4.7). They operated the Omega.3 with their

right hands. In front of it, we put a stand on which a real texture sample was

put and a virtual texture was rendered. Their relative locations were randomly

decided in each trial. To block other sensory cues, participants’ view to the

Omega.3 and texture samples was blocked by a screen. Instead, the position of

– 62 –

the Omega.3’s stylus relative to real and virtual textures was indicated on the

monitor using a sphere cursor and two white rectangles (Figure 4.7). Participants

wore noise-cancelling headphones playing white noise to cancel off any auditory

cues.

Each real texture was compared with three virtual textures: a flat virtual

wall without texture, force texture, and vibration texture. This resulted in 30

texture pairs. A flat wall was regarded as a baseline condition. Each texture

pair was presented only once in random order. On each trial, participants freely

explored the real and virtual textures and then rated all five similarity scores.

Participants were allowed to take a break whenever necessary. On average, the

experiment took about 50 minutes for each participant.

We recruited ten participants (8 males and 2 females; 22–30 years old with

average 25.1; all right-handed) who had experiences in using haptic devices. We

preferred expert participants since in pilots we learned that breaking down per-

ceived similarity to the four specific criteria was not easy for novice users. Expert

participants are likely to have higher resolution to such cognitive tasks.

4.6.2 Results and Discussion

The average similarity scores for the three rendering methods are shown for

each criterion in Figure 4.8. No significant outlier was detected based on boxplot

inspections. Geometry, roughness, stiffness, and friction are critically correlated

to the four perceptual dimensions of haptic texture—shape, roughness, hardness,

and stickiness, respectively, all of which influence the overall similarity. Thus,

the scores of each method indicate the effectiveness of the method in delivering

– 63 –

Figure 4.8: Average similarity scores for the five criteria. Error bars indicate

standard errors. Pairs grouped by asterisks were significantly different by Tukey’s

HSD tests (∗ : 0.01 < p < 0.05, ∗∗ : p < 0.01).

the corresponding perceptual information.

In all cases, the similarity scores were ordered as force feedback (FF) >

vibration feedback (VF) > virtual wall (WALL). All the five scores passed the

Shapiro-Wilk normality test and Mauchly’s sphericity test. We performed two-

way repeated measure ANOVA on each measure using texture material and ren-

dering method as independent variables. Texture material was significant for all

the measures with p < 0.001. This is expected, and we do not discuss it further.

In what follows, we focus on the effects of rendering method on each measure.

Tukey’s HSD test was applied for post-hoc multiple comparisons when rendering

method was significant.

Geometry

FF resulted in a good average score (632), and VF a borderline score (51).

WALL’s score (45) was in the low side. Rendering method had an significant effect

on geometry similarity (F (2, 18) = 12.86, p < 0.001). FF was significantly better

– 64 –

than VF (p = 0.0033) and WALL (p < 0.001), but VF was not significantly better

than WALL (p = 0.25). This can be evidence that force feedback is superior in

delivering fine geometrical surface features to vibration feedback. The average

mechanical momentum produced by vibration feedback is zero, and it can be an

inherent limit for rendering texture geometry. Also, there is still sizable room for

improvement even for force feedback considering the scores.

Roughness

The mean scores of judging similarity based on roughness were 59, 52, and

42 for FF, VF, and WALL. Rendering method was significant for their differ-

ences (F (2, 18) = 10.4, p < 0.001). In post-hoc comparisons, both FF and VF

significantly outperformed WALL (p < 0.001 and p = 0.0332), but FF and VF

showed no significant difference (p = 0.13). These results suggest that force and

vibration cues have similar modeling and rendering performance for conveying

the roughness of real textures. However, roughness similarity should be made

better for both methods to achieve very high realism. Combining the two cues

may compensate for their relative weaknesses resulted from their bandwidths.

Stiffness

The means for stiffness similarity were 67, 61, and 58, all better than those

for geometry and roughness similarity. Although the sensory cues provided for

stiffness perception were identical for the three rendering methods, it seems that

the judgment of stiffness similarity was affected by other similarities to some

2In rating similarity, people tend to avoid extreme scores. In similar experiments, scores in

a 1–100 scale for the same real objects were around 80 [56, 31, 63].

– 65 –

extent. Rendering method showed significance (F (2, 18) = 3.639, p = 0.0471),

and the only significant different pair was the pair between FF and WALL (p =

0.022). Overall, our method for stiffness enabled fairly good match with real

surfaces.

Friction

The friction modeling and rendering was quite realistic with the means of

75, 74, and 69. There also appears to be some carry-over effects from the overall

similarity, but rendering method was not significant (F (2, 18) = 0.69, p = 0.52).

Friction showed the highest similarity among the four perceptual dimensions.

Overall Similarity

Finally, the overall similarity scores were 59, 51, and 44, and rendering

method was significant for the differences (F (2, 18) = 5.13, p < 0.001). FF-

VF and FF-WALL had significant differences (p = 0.039 and p < 0.001), but

VF-WALL did not (p = 0.13). These results are well aligned with the results

of the individual perceptual dimensions. Force feedback could deliver more real-

istic textures than vibrotactile feedback. However, even the perceptual realism

of force feedback cannot be said sufficient. Recall that both force and vibration

cues exist in real interaction.

4.7 Conclusions

In this chapter, we have presented a geometry-based haptic texture modeling

and rendering algorithms that provide high-resolution geometry profiles estimated

– 66 –

by photometric stereo. Our method enables the height-map of a real materials

to be captured with the resolution about 10µm, which is ten times better than

the previous methods[29]. We also model the parametric Dahl friction model

and Hunt-Crossley stiffness model to support realistic rendering of the modeled

textures. A user study that estimated the perceived similarity between real and

virtual textures showed that our framework is capable of re-creating real textures

with a reasonable level of realism if the target material is relatively soft with low

friction. The study also disclosed interesting research issues for further improve-

ment of our framework. In the following chapters, combining our model with the

data-driven method of contact vibration will be addressed.

– 67 –

V. Hybrid Texture Rendering

In this section, we present a hybrid haptic texture rendering framework that

combines the physics-based model and the data-driven model. At first, we devise

a new force-feedback texture-rendering algorithm adequate for hybrid rendering.

Through a user study, we verify that the proposed algorithm is more suitable for

our hybrid texture framework than other well-know algorithms. To model the

contact-acceleration occurring during the exploration of a texture, we use an LPC-

based method. The frequency characteristics of the two models are understood to

lead them to merge into a hybrid framework. A user study assessing the perceived

similarity between the real textures and their virtual simulations indicated that

our framework can achieve a reasonably high level of realism. In particular, our

framework could model and render inhomogeneous textures with a quality similar

to homogeneous textures.

5.1 Hybrid Rendering Approach

After we construct all the models as described in Chap. III and Chap. IV, we

use these models to render a hybrid haptic texture. Our hybrid texture rendering

algorithm consists of three parts: a part that generates the force-feedback signal,

another part that synthesizes the contact acceleration vibration signal, and a

signal-process part to adjust the force-feedback signal and the vibration signal

so that the combination of two signals does not lead to an exaggeration and

– 68 –

Velocitycalculation

Geometry model

Hybrid filtering

Voice-coil actuator

Force-feedback

device

Force generation algorithm

LPC based model

Mechanically coupled

𝑥, 𝑦, 𝑧

Contact position

Normalforce

𝐹𝑁

Tangential Velocity

𝑣𝑇

𝐹𝑥 , 𝐹𝑦 , 𝐹𝑧

Texture force

Contact acceleration

𝑎𝐶

Filtered acceleration ෦𝑎𝑐

Filtered texture force ෩𝐹𝑥 , ෩𝐹𝑦 , ෩𝐹𝑧

Figure 5.1: Schematic diagram of our hybrid rendering process.

generates a feel that is most similar to the target material. Fig. 5.1 depicts the

whole process of our texture rendering.

In the first subsection, we propose a force-generation algorithm for a haptic

texture. We also compare our algorithm with two existing ones in terms of

stability. To prevent the surface we render from being ‘alive’, we estimate and

present the conservativity and passivity of the algorithms. We then present the

vibration synthesis process using an LPC-based model in the following subsection.

The last subsection introduces our hybrid-rendering algorithm, focusing on the

combination of force-feedback and vibrotactile signal.

5.1.1 Force-generation Algorithms

After Minsky’s first algorithm, many force-feedback algorithms rendering a

haptic texture using a height map have been proposed. Generally, these algo-

rithms elicit the rough feel of the surface by perturbing the normal force from

the surface. This approach leads to an increased perceived roughness for the

– 69 –

virtual texture; however, is also prone to induce an instability of the surface

when the fluctuation of the normal force exceeds the mechanical limit of the

force-feedback device. Such artifacts occurring during the exploration of a vir-

tual surface greatly reduce the realism of that surface. Therefore, the objective

of many haptic texture-rendering algorithms is to find an optimal trade-off be-

tween realism and stability of the rendered surface for their platform or type of

application.

In Fig. 5.2, we present two popular algorithms to render a haptic texture.

Algorithm A renders a virtual texture by aligning the direction of the normal

force to the gradient of the texture height map [64]. By introducing the fast-

varying gradient of the texture height map, algorithm A elicits the sensation

of roughness effectively. However, algorithm A is prone to introduce artifacts

during exploration. By contrast, algorithm B is known for rendering highly stable

𝑑𝐹

𝑛

𝐹

𝑛

𝑑

𝐹𝐹𝑧

𝐹𝑥

HIP movement direction

Not moving

A B

Figure 5.2: Schematic diagrams for two popular texture rendering algorithms.

Circles represent the position of the haptic tool in the virtual space. Thick

straight lines represent the nominal surface, and dashed lines show textured sur-

faces.

– 70 –

surfaces [18]. Algorithm B changes the magnitude of the friction force to render

a textured surface. In most cases, the direction of the friction force does not

change very rapidly. Moreover, the magnitude of the force falls to zero when the

user stops.

As these algorithms do, we also need to find or devise a force-feedback algo-

rithm to render the haptic texture by considering the characteristics of the height

map constructed using the photometric stereo. As reported in Chap. IV, the pho-

tometric stereo estimates the height map with a resolution that is approximately

ten times higher than in prior studies. This means that our height map contains

much smaller height fluctuation and has a higher spatial frequency than previous

height maps. In this case, the rendering algorithm A in Fig. 5.2 is inadequate

although it is well known for a rich sensation of roughness. This algorithm aligns

the direction of normal force with the gradient of the texture. Therefore, algo-

rithm A can potentially induce a stability problem if it is applied to our height

maps. By contrast, algorithm B renders a virtual surface with high stability as

it only adjusts the magnitude of the frictional force using a normalized texture

height. However, this algorithm is less effective for conveying the fine roughness

of the texture transferred by a small variation of the height around the surface.

Based on these considerations, we propose a new force-feedback algorithm

inspired by both algorithms A and B. Our approach is to modulate the magnitude

of the frictional force, as in algorithm B. The difference is that the magnitude

depends on the gradient of the height, not the normalized height. This enables

our algorithm to deliver a fine roughness sensation, superior to that of algorithm

– 71 –

B.

Using these algorithms, we calculated the force output to render virtual

texture as follows. In our framework, we assume that a flat and thin texture

material is overlaid with a stiff base object. First, we calculate the original force

Fo with no consideration for the texture. Then, the target force output Fo can

be divided into three components:

~Fo = ~Fm + ~Fb + ~Ff (5.1)

where ~Fm is the stiffness force from the material, ~Fb is the stiffness force from

the base under the texture, and ~Ff is the friction force. The stiffness force of the

texture ~Fm can be calculated using (4.11) using the penetration depth into the

material x. The base object’s force ~Fb is calculated assuming a linear stiffness

model. This simple model is sufficient for the base object because we do not

consider the deformation of the base object. The stiffness coefficient of the base

object was set to half the maximum linear stiffness allowed by the force-feedback

device. The direction of ~Fb is also set to the surface’s normal. The friction force

~Ff is calculated using the Dahl model, and its direction is set to the opposite of

the user’s tangential moving direction. After calculating all three components of

~Fo, we apply our texture rendering algorithm to generate the final force output

~Ft. Formulas are given in table 5.1.

5.1.2 Stability Analysis

To compare the stabilities of these three algorithms, we test the conserva-

tivity of the force field generated by each algorithm. When an algorithm creates

– 72 –

𝑥

𝑧

𝑥

𝑧A B𝑑𝑥𝑚𝑎𝑥

𝑑𝑧𝑚𝑎𝑥

Figure 5.3: The path to calculate the energy gain of force algorithms. The red

dashed line represents the direction of the trajectory, and the blue line in path

A is the texture height map. We tested 2D trajectory for a straightforward

calculation.

a non-conservative force field, the confined energy might create artifacts. To

test that, we calculate the energy gain following the fixed trajectory depicted

in Fig. 5.3. Algorithm A is tested using path A, and the friction-based algo-

rithms (Algorithm B and the proposed algorithm) are tested on path B. Since

the friction-based algorithms dissipate energy during the slip phase, their trajec-

tory has limited movement in the tangential position x. For ease of calculation,

we use a simple sinusoidal function h(x) = A sin(2πx/l) as the height map of the

texture. Then the energy generated while cycling through the trajectory is

∆E =κ0A

2

2+κ0L

2

4π2

(1−

√L2 + 4A2π2

L2

)(5.2)

Generally, Algorithm A generates non-conservative force fields and knows to

cause the feel of an active surface as well. In the case of Algorithm B and the

– 73 –

proposed algorithm, the energy gain is in the order of

∆E =κ0µ

2dxmaxd

zmax (5.3)

if we ignore some constant factors. dxmax and dzmax are the parameters used

in the Dahl friction model.

Considering that dxmax is generally in the range 10−2 m to 10−6 m and dzmax is

limited by the thickness of the texture material, and the energy gain is negligible.

To summarize, the proposed algorithm generates a highly stable texture surface.

As the performance of the algorithm in delivering a sensation of roughness

is hard to quantize, we conducted a dedicated user study. More detail on this is

provided in section 5.2.

5.1.3 Vibration Synthesis

To synthesize the contact acceleration of texture, we use the same filter vector

~h optimized in section ??, but in the reverse direction (Fig. 5.4). We use this

filter as an autoregressive moving average (ARMA) filter that uses the history of

the contact acceleration ag(l) as an autoregressive term and uses the history of

the white noise eg(l) as a moving-average term. The power of this white noise is

set to be the same as that of the residual e(k) in the modeling process.

The filter coefficients for the current normal force fN and tangential velocity

vT are determined by barycentric interpolation using fN and vT as interpolation

variables. The tangential velocity of the current user can be acquired using

the position encoder of the force-feedback device. However, the user’s current

normal force cannot be estimated as the user is not actually interacting with any

– 74 –

𝐻(𝑧)

1/𝑃(𝑧)𝑎𝑔(𝑙)𝑒𝑔(𝑙)

Σ+

+

Figure 5.4: A diagram of vibration synthesizing using an LPC-based model [1].

The inverse model 1/P (z) synthesizes the contact vibration ag(l) of the texture

by taking white noise eg(l) as a moving average term of the ARMA filter H(z).

real object. Instead, we use the normal force generated by the force-feedback

algorithm. Given the values of vT and fN , we find four neighbor filters and

interpolate their coefficient independently to generate a new filter. Then, we

synthesize the next contact acceleration sample using the new filter, the history

of the contact acceleration, and the history of the white noise. As long as the

user maintains the contact and moves faster than the threshold velocity (1 cm/s),

this process is repeated to synthesize the continuous vibration signal.

5.1.4 Hybrid Rendering Algorithm

The force-feedback generation and vibration synthesis are performed by

two asynchronized and dedicated threads as vibration synthesis requires a much

higher update rate (force-feedback: a few kHz, vibration: tens of kHz). In each

iteration, the force generating thread computes vt and fn. Based on their value,

the force generating thread turns the vibration synthesis thread on or off.

As we mentioned earlier, the independent modeling and rendering of the

– 75 –

force-feedback and vibrotactile signal simultaneously can potentially exaggerate

the sensation caused by the texture. To prevent this, we use the cut-off frequency

fc introduced in section 3.5. In our hybrid texture framework, the inhomogeneity

of the texture is transferred by force-feedback signal, and the characteristic of the

material itself is sent in the form of the vibrotactile signal. Therefore, we apply

a low-pass filter with the cut-off frequency fc to the generate a force signal to

ensure that the signal only contains the inhomogeneity information. Inversely, the

synthesized vibrotactile signal is high-pass filtered with the same cut-off frequency

to remove low-frequency components. This process can be regarded as a cross-

talk removing process in the viewpoint of the duplex theory of texture perception.

After filtering, the signals are fed to the corresponding devices.

5.2 User Study 1: Comparison of Force-Feedback Ren-

dering Algorithms

We compared three force-feedback texture-rendering algorithms (Algorithm

A: Norm, Algorithm B: Fric1, Proposed algorithm: Fric2) described in sec-

tion 5.1.1 by means of a user study. Through this user study, we attempted

to find the best match for an LPC-based vibration texture rendering algorithm

in a hybrid haptic texture framework. In the user study, we placed the emphasis

on the perceived realism of the virtual textures rendered. The level of realism

was evaluated as a function of the subjective similarity score between the virtual

textures and their corresponding real material. This study was approved by the

IRB of POSTECH (PIRB-2019-E014).

– 76 –

Denim Velvet Cork

Rubber

mat

Towel Acryl

Bamboo

mat

Coated

Hardboard

WoodTile

Figure 5.5: Texture materials used in the user studies.

5.2.1 Method

The participants of this study were asked to assess the subjective similarity

score between real and virtual textures using three criteria: roughness, friction,

and overall similarity. Roughness and friction are two major dimensions of texture

perception. As all force-feedback texture rendering algorithms adjust either the

roughness or the friction of the virtual surface to elicit the texture sensation,

we examine the effects of the type of the algorithm on these dimensions as well.

Each measure was rated on a scale of 0–100 (0: completely different and 100:

identical).

We used ten textured materials, as shown in Fig 5.5, selected for diversity.

Half of them (bamboo, cork, rubber mat, tile, and wood) had fine visible geomet-

ric features while the others did not. Their stiffness and friction characteristics

were also widely different.

We recruited fifteen participants (9 males and 6 females; 19–31 years old

with an average of 22.9; all right-handed) who had no experience in using haptic

– 77 –

interfaces. None of them reported any existing sensory or motor impairment. The

participants all signed an informed consent form after we explained the goals and

procedure of the experiment. To prevent any misunderstanding of the similarity

score measures, the meaning of the measures was explained using real texture

samples. The participants were paid KRW 15,000 (approximately USD 13) after

the experiment.

Here, we assumed a tool-mediated interaction between the participants and

both real and virtual textures. To this end, we attached a custom aluminum

stylus-shaped grip to a Force Dimension’s Omega.3 force-feedback device. The

TactileLab’s Haptuator BM3C installed inside the grip facilitated the partici-

pant’s interaction with the virtual textures. Under the grip, a sharp tooltip was

combined to let participant interact with the real textures in the same man-

ner as with the virtual textures. In front of the Omega.3 device, a stand was

placed to balance the difference in height between the real texture samples and

the virtual textured surface. During the experiment, participants operated the

Omega.3 device to stroke texture samples by holding the grip of the Omega.3

device. Participant’s view toward the texture samples was blocked by a screen

barrier. Instead, the position of the Omega.3’s stylus relative to the real and vir-

tual textures was indicated on the monitor using a sphere cursor and two white

rectangles (Fig. 5.6). Participants wore noise-canceling headphones playing white

noise to cancel off any auditory cues.

The experiment consisted of a training session and the main session. During

the training session, ten virtual texture samples were rendered in random order,

– 78 –

Barrier

Haptic Device

Texture Sample

The position of the haptic device’s stylus

The locations of two textures

Figure 5.6: Experimental setup of the user studies. The left figure is a picture

of the physical environment, and the right one is a screenshot of the experiment

program.

one by one. Participants were asked to stroke the virtual texture with adequate

pressure on the surface (0 N–12 N; maximum linear force of Omega.3 device)

for ten seconds to proceed to the next sample. This training session prevented

haptic novices from damaging the real texture samples and penetrating the virtual

texture surfaces. The main session contained ninety trials (ten materials × three

rendering methods × three repetitions). The order of the trials was randomized

for each participant. In every trial, participants evaluated the similarity between

a virtual texture and its corresponding real texture. Their positions were shuffled

randomly. Participants could freely explore the real and virtual textures and were

allowed to take a break whenever necessary. On average, the experiment took

about 80 minutes for each participant.

– 79 –

Figure 5.7: Average similarity scores of user study 1 for the three criteria. Error

bars indicate 95% confidence intervals. Pairs grouped by asterisks were signifi-

cantly different according to Tukey’s HSD tests (∗ : 0.01 < p < 0.05, ∗∗ : p <

0.01).

5.2.2 Results

The average similarity scores for the three rendering methods are shown

in Figure 5.7 for each criterion. No significant outlier was found by a boxplot

inspection. Error bars indicate 95% confidence intervals.

The average similarity scores for friction were all similar for the three ren-

dering methods (Norm: 61.0, Fric1: 61.6, Fric2: 60.3). However, in roughness

– 80 –

(Norm: 66.4, Fric1: 56.3, Fric2: 66.2) and overall similarity (Norm: 64.3, Fric1:

57.3, Fric2: 63.7), Fric1 showed a lower average score than the other two. For

statistical analysis, we performed a two-way repeated-measure ANOVA on each

measure using texture material and rendering method as the independent vari-

ables. The data for all the three measures passed the Shapiro-Wilk normality test

and Mauchly’s sphericity test, which are the prerequisites of ANOVA. Table 5.2

shows the results of ANOVA for each measure.

Regarding the similarity score for roughness, only the effect of rendering

method was found to be significant. The roughness similarity score mostly de-

pended on the main effect of rendering method (η2p = 0.657), rather than on

material (η2p = 0.093) or the interaction between them (η2p = 0.087). By contrast,

the similarity score for friction exhibited a significant correlation with the type of

material and the interaction term, but not with rendering method. The overall

similarity score was significantly affected by all the three factors with relatively

even contributions (η2p = 0.251, 0.191, and 0.124, respectively).

To direct the focus to the effect of rendering method, we ran Tukey’s HSD

test for post-hoc multiple comparisons on the roughness and overall similarity

score. In both measures, the differences between Norm and Fric2 were not sig-

nificant, but Fric1 was statistically different from Norm and Fric2 (see Figure

5.7).

5.2.3 Discussion

The rendering method showed a significant effect on the roughness similarity

score. In particular, the Fric1 method scored much lower in terms of similarity

– 81 –

than the other two methods. The Fric1 method elicits the sensation of roughness

by modulating the magnitude of the frictional force, with the variation of the

force depending on the normalized height. This algorithm can simulate a rough

texture when the height variation is large compared to the thickness. However, if

the height variation of the texture is small and changes quickly, the fluctuation of

the frictional force becomes too small to be perceived. As a result, a user cannot

feel the roughness from the virtual surface rendered. In practice, materials with

visible geometry (whose height variation might be large) lead to a smaller score

difference between Fric1 and Fric2 than the other materials. By contrast, the

Fric2 method adjusts the magnitude of the frictional force using the gradient of

the height map. Although the height variation is small compared to the thickness

of the surface, the Fric2 method can fully deliver the rough feel of the surface

provided. The ANOVA results on roughness suggest two conclusions. First, the

Fric1 method is inadequate for hybrid texture rendering. Second, to render a

hybrid haptic texture, the Fric2 method is as effective as the Norm method, the

most widely used haptic texture rendering method.

The friction similarity score was not significantly affected by the type of

rendering method. Instead, the type of material had a significant impact on the

score. In most materials, the higher the maximum friction force, the lower the

similarity score except for Acryl.

The ANOVA results of the overall similarity score are similar to the aver-

aged results of the roughness and friction score. Considering that the roughness

and friction are two major axes of haptic texture perception, this tendency is

– 82 –

quite natural. As the friction similarity score was not affected by the rendering

method, our discussion on the roughness similarity score holds similarly for the

overall similarity score. Among the three rendering methods, the Fric1 method

scored a considerably lower score than the two other methods with no statistical

difference in performance. Considering the superior stability of the Fric2 method

as explained in section 5.1.2, we conclude that the Fric2 method is the most

effective force-feedback texture rendering algorithm among the three methods

proposed in this user study.

5.3 User Study 2: Assessing the Realism of Hybrid

Haptic Texture

In the previous section, we compared the force-feedback texture-rendering

algorithms for hybrid texture rendering. Based on the result of the user study

and stability analysis, we conclude that the Fric2 algorithm is the most suitable

for our hybrid framework. In this user study, we compare the realism of our

hybrid framework to the previous haptic texture rendering algorithms that solely

use either force-feedback or vibrotactile. The photometric stereo-based method

(FF) and the LPC-based method (LPC) were used as the representatives of the

force and vibration approaches. As in user study 1, we proposed a subjective

similarity score between the virtual textures and the corresponding real material

for various rendering methods. This study was approved by an IRB of POSTECH

(PIRB-2019-E014).

– 83 –

Figure 5.8: Average similarity scores of the user study 2 for the three criteria.

Error bars indicate 95% confidence intervals. Pairs grouped by asterisks were

significantly different for Tukey’s HSD tests (∗ : 0.01 < p < 0.05, ∗∗ : p < 0.01).

5.3.1 Method

We used the same experiment apparatus and procedure as for the user study

1. The only two differences were the number of participants and the rendering

methods used in the experiment. In user study 2, we recruited twenty haptic

novices (11 males and 9 females; 20–27 years old with an average age of 23.6

years; all right-handed). As this is the main experiment in this work, we wanted

to improve the reliability of the user study by using more participants.

– 84 –

As we concluded that Fric2 was the most suitable algorithm for hybrid haptic

texture rendering, we used the Fric2 algorithm in this user study. To render the

force-only virtual texture, we directly rendered the force signal generated in the

force-generation module introduced in section 5.1.1 with no hybrid filtering and

vibration. In the case of vibration-only textures, we applied no texture-rendering

algorithm in the force-generation module in order for the resulting force signal to

only render the flat surface. No hybrid filtering was applied in this case either.

The physical parameters for the Dahl model and the Hunt-Crossley model were

all shared among the rendering methods.

The same training session as in user study 1 followed in the main session

consisting of ninety trials (ten materials × three rendering methods × three

repetitions).

5.3.2 Results

The average similarity scores for the three rendering methods are shown for

each criterion in Fig. 5.8.

We applied a two-way repeated-measure ANOVA on the data of each mea-

sure using rendering method and texture material as the independent variables.

The data for all the three measures passed the Shapiro-Wilk normality test and

Mauchly’s sphericity test. ANOVA results for each measure are detailed in Ta-

ble 5.3.

The average scores for roughness similarity were ordered as Hybrid (66.5) ¿

FF (62.4) ¿ LPC (59.5), and this was statistically significant. Tukey’s HSD test

for post-hoc multiple comparisons showed that Hybrid had a significantly better

– 85 –

score than FF and LPC (Figure 5.8). The effect sizes in terms of η2p showed that

rendering method was the most dominant factor (Table 5.3).

For friction similarity, the rendering method did not cause significant differ-

ences in the average scores. The texture sample and the interaction term were

significant with similar effect sizes.

Lastly, the overall similarity scores showed a very similar trend to that of

the roughness similarity scores. The effect of rendering method was statistically

significant, and their average scores were ordered as Hybrid (65.30) ¿ FF (61.3)

¿ LPC (59.0). Hybrid had a significantly better score than FF and LPC accord-

ing to Tukey’s post-hoc tests (Figure 5.8). The texture material, however, was

not significant for the overall similarity. The effect size was greatest with the

rendering method was the largest (Table 5.3).

5.3.3 Discussion

The hybrid rendering method showed the best subjective performance in

terms of roughness similarity with statistical significance. The other two meth-

ods, LPC and FF, were not distinguished in delivering the sensation of roughness

on average. However, they exhibit different patterns when the roughness simi-

larity scores are listed for individual texture materials and rendering methods,

as in Table 5.4. The materials we used in this user study can be grouped into

two categories on the basis of their texture homogeneity. In most inhomogeneous

materials, FF scored a higher value for roughness similarity than LPC. The only

exception was Tile, which generates very strong contact vibration for texture

perception. Conversely, LPC generally received higher scores than FF for homo-

– 86 –

geneous textures except for Acryl. Hence, we can deduce that the FF algorithm

is appropriate for inhomogeneous textures and LPC is suitable for homogeneous

ones. The results also imply that our hybrid rendering method inherits the ad-

vantages of both methods. This inference can explain the superior performance

of hybrid rendering method for rendering texture roughness.

The friction similarity score showed a similar trend with user study1. No

significant effect was found in the type of rendering method. Instead, the type of

material and the interaction effect appeared significant.

As for the overall similarity score, the hybrid method outperformed FF and

LPC with statistical significance. These results are well aligned with the results

of the roughness and friction similarity scores. Table 5.5 displays the average

overall similarity score for each material and rendering method. While the overall

similarity scores of LPC and FF largely depend on the texture homogeneity, the

hybrid method shows consistency compared to the others.

The above results allow us to conclude that the hybrid rendering method is

more effective than the prior ones in terms of realism (similarity to real textures)

and applicability.

5.4 Conclusion

In this section, we proposed a hybrid rendering procedure using force-feedback

and vibrotactile rendering simultaneously. To this end, we presented new force-

feedback texture-rendering algorithm using friction modulation for our high-

resolution height map. We also proposed a hybrid texture-rendering algorithm

– 87 –

combining the force-feedback and vibrotactile feedback with no exaggeration of

the texture perception. We evaluated our research using two user studies that es-

timated the perceived similarity between the real and virtual textures. The first

user study comparing the force-feedback texture-rendering algorithm indicated

that the proposed force-feedback rendering algorithm is suitable for our hybrid

texture framework. In the second user study, we compared the textures rendered

using the hybrid framework with the same textures rendered using solely either

force or vibration. The results demonstrate that the hybrid texture framework

achieves better realism and applicability than the previous methods.

– 88 –

Table 5.1: Formulas of the three force-texture rendering algorithms. nt is a

gradient of the texture height map. h(x) stands for the normalized texture height.

θt is the angle between the moving direction of the user and nt.

Algorithm A ~Ft = |~Fm|~nt + ~Fb + ~Ff

Algorithm B ~Ft = ~Fm + ~Fb + (1− h(x))~Ff

Proposed ~Ft = ~Fm + ~Fb + (1 + sin θt)~Ff

– 89 –

Table 5.2: ANOVA Statistics of the user study 1

Roughness

Factor Df F p η2p

Method* 2 26.873 < 0.001 0.657

Material 9 1.433 0.181 0.093

Method:Material 18 1.340 0.163 0.087

Friction

Factor Df F P η2p

Method 2 0.292 0.749 0.020

Material* 9 3.976 <0.001 0.221

Method:Material* 18 3.429 <0.001 0.197

Overall

Fctor Df F P η2p

Method* 2 4.693 0.017 0.251

Material* 9 3.316 <0.001 0.191

Method:Material 18 1.980 0.118 0.124

Significant factors are marked by *.

– 90 –

Table 5.3: ANOVA Statistics of the user study 2

Roughness

Factor Df F p η2p

Method* 2 13.121 <0.001 0.484

Material* 9 2.279 0.021 0.140

Method:Material* 18 4.581 <0.001 0.247

Friction

Factor Df F p η2p

Method 2 1.685 0.204 0.107

Material* 9 6.324 <0.001 0.311

Method:Material* 18 8.061 <0.001 0.365

Overall

Factor Df F p η2p

Method* 2 13.182 <0.001 0.485

Material 9 1.625 0.115 0.104

Method:Material* 18 5.066 <0.001 0.266

Significant factors are marked by *.

– 91 –

Table 5.4: Aggregated roughness similarity scores

Inhomogenous

Material Bamboo Denim Rubbermat Tile Towel Wood

LPC 44.00 58.67 58.22 61.33 55.78 56.67

FF 60.00 69.22 70.56 54.64 62.67 69.67

Hybrid 62.11 70.78 73.22 68.78 65.22 71.89

Homogenous

Material Acryl Hardboard Cork Velvet

LPC 66.44 61.87 65.11 67.00

FF 72.89 57.22 57.16 50.02

Hybrid 72.78 58.24 61.33 60.98

Table 5.5: Aggregated overall similarity scores

Inhomogenous

Material Bamboo Denim Rubbermat Tile Towel Wood

LPC 52.49 58.87 58.44 59.20 53.76 52.02

FF 62.69 70.62 67.18 55.20 60.60 68.22

Hybrid 70.18 65.00 69.62 66.47 57.84 70.09

Homogenous

Material Acryl Hardboard Cork Velvet

LPC 65.09 57.29 60.18 73.04

FF 69.93 51.91 55.11 51.64

Hybrid 64.96 57.84 65.04 65.91

– 92 –

VI. Conclusions

In this dissertation, we have proposed a hybrid framework for haptic texture

modeling and rendering that uses force-feedback and vibrotactile rendering simul-

taneously. This framework is suitable for modeling and rendering inhomogeneous

textures realistically and efficiently. To this end, we present a geometry-based

texture-modeling algorithm and new force-feedback texture-rendering algorithm

using friction modulation.

To construct a physics-based model to generate force-feedback, we have pre-

sented a geometry-based haptic texture modeling and rendering algorithms that

provide high-resolution geometry profiles estimated by the photometric stereo.

Our method enables the height-map of real materials to be captured with the

resolution about 10µm. We also model the parametric Dahl friction model and

Hunt-Crossley stiffness model to support the realistic rendering of the modeled

textures. A user study that estimated the perceived similarity between real and

virtual textures showed that our framework is capable of re-creating real textures

with a reasonable level of realism if the target material is relatively soft with low

friction.

In the following user study, we compared the perceptual advantages and

disadvantages between the physics-based model and the data-driven model. In

particular, the perceptual similarity of a virtual texture to a real texture is rated

using five criteria of geometry, roughness, hardness, friction, and overall similar-

– 93 –

ity. According to the results, force feedback provides better realism for geometry

than vibration feedback, which also leads to the better overall realism. However,

each method alone is not sufficient for highly realistic texture rendering.

Then, we devise a hybrid texture-rendering algorithm to combine the force-

feedback and vibrotactile feedback without the exaggeration of texture percep-

tion. By applying a band-limited filter, we can discriminate the frequency band-

width for each model. The filtered signal is fed to an ordinary data-driven model

to synthesize the spectral-identical signal. The synthesized signal is combined

with the force-feedback signal generated by the physics-based model to render

realistic virtual textures.

We evaluated our research using two user studies that estimate the perceived

similarity between real and virtual textures. The first user study that compares

the force-feedback texture-rendering algorithm showed that our proposed force-

feedback rendering algorithm is suitable for our hybrid texture framework. In

the second user study, we compared textures rendered using the hybrid frame-

work with the same textures rendered by either force or vibration only. The

results demonstrate that the hybrid texture framework exhibits better realism

and applicability than the previous methods.

Our hybrid framework still has much room for further improvement. First,

a thermal modeling and rendering model has been left as futurework to limit the

scope of this research. Adding a thermal sensation would greatly advance the

realism of the rendered virtual textures. Next, the hybrid rendering algorithm

can be replaced with a more complicated realistic algorithm. We expect that

– 94 –

machine-learning based techniques can be a good solution to our problem.

– 95 –

요 약 문

햅틱 질감은 우리가 일상생활중에 느낄수 있는 기본적인 햅틱 특성 중 하나이다.

우리가 주변 환경과 상호작용을 할 때 발생하는 촉각 신호의 대부분이 햅틱 질감의

형태로제공되기때문에,사실적인햅틱질감을가상으로재생하여사용자에게제공

하는 것은 햅틱 연구자들에게 있어서 오랜 염원이었다. 산업적인 측면에서 보아도,

사실적인 햅틱 질감은 다양한 가상현실 어플리케이션의 사실성 및 몰입감을 높일

수 있기 때문에 중요도가 높은 기술이라고 할 수 있다.

햅틱질감을모델링및렌더링하기위한기존기술은크게두가지분류로나눌수

있다. 하나는 물체의 성질을 설명하기 위해 적합한 물리적 모델을 이용한 물리 기반

(Physics-based) 질감 모델이며, 다른 하나는 질감을 표현하기 위한 물성을 정하고,

이를 측정하여 비매개변수 모델을 이용하여 표현하는 데이터 기반(Data-driven) 질

감 모델이다. 물리 기반 모델의 경우 명확한 물리 수식을 기반으로 질감을 표현할

수 있기 때문에 필요한 데이터의 양이 적고 물리적으로 명확한 의미를 가진다는 장

점이존재하지만,경우에따라물리모델이복잡해질수있고,사실성을높이기위한

복잡한물리모델은모델링및렌더링이사실상어렵다는단점이있다. 한편,데이터

기반 모델의 경우 물리 기반 방식에서 표현할 수 없는 복잡한 물리적 관계를 표현할

수 있어서 더 사실성 있는 질감의 표현이 가능하고 복잡한 물리 시뮬레이션이 필요

없다는 장점이 있지만 입력 변수의 종류가 늘어남에 따라서 필요한 데이터의 양이

기하급수적으로 증가하여, 다양한 종류의 질감을 표현하는 것이 어렵다는 단점이

있다. 특히, 물체의 표면이 비균일한 특성을 가지는 질감(Inhomogeneous Texture)

– 96 –

의 경우 접촉 위치에 따라 다른 반응을 보이기 때문에 데이터 기반 방식으로 처리

하기가 어렵다.

이러한 문제를 해결하기 위해서 이 학위논문에서는 햅틱 질감을 모델링 하고 렌

더링하기 위한 혼용 체계(Hybrid Framework)를 제안한다. 각기 분명한 장단점을

가지는 물리 기반 방식과 데이터 기반 방식의 질감 모델링 및 렌더링 방식을 통합적

으로 이용하여 물리 질감 모델의 단점인 낮은 사실성을 높이고, 데이터 기반 방식의

단점인 낮은 적용성을 확장시켜 결과적으로 넓은 적용성과 높은 사실성을 가지는

물리 질감을 모델링 및 렌더링 할 수 있는 통합적인 시스템을 개발하였다. 이 프레

임워크는 크게 세 부분으로 이루어진다. 첫 번째는 선형 예측 부호화(LPC; Linear

Predictive Coding) 기반 모델을 이용하여 햅틱 질감을 데이터 기반 방식으로 모델

링 하는 방식이다. 이 부분에서 우리는 기존에 균일한 질감에만 적용이 가능했던

데이터 기반 방식을 비균일 질감에 적용하기 위한 방식을 제시한다. 다음으로는

Photometric Stereo 방식을 이용해서 물리 기반 질감 모델의 모델링 성능을 향상

시켰다. 기존에 햅틱 질감에 이용된 방식들에 비해서 그 정밀성을 향상시킴으로써

물리 기반 모델의 사실성을 높이기 위한 방식을 제시한다. 마지막으로 세 번째 부분

은 앞서 언급된 두 햅틱 질감 모델을 하나로 합쳐서 렌더링하기 위한 알고리즘이다.

두 신호가 서로 이질감 및 간섭을 일으키지 않도록 하기 위해서 두 신호 사이에 필

터링 과정을 거쳐 렌더링을 수행하였다. 완성된 프레임워크를 통해서 제시된 가상

질감은 사용자 실험을 통해서 얼마나 실제 질감과 유사한지 비교되었다. 기존에

각각 사용되던 물리 기반 모델/데이터 기반 모델과 비교를 하였을 때 제시된 혼용

체계는 더욱 사실성이 높은 가상 질감을 재생할 수 있음이 확인되었다.

– 97 –

References

[1] J.M. Romano, T. Yoshioka, and K.J. Kuchenbecker. Automatic filter design

for synthesis of haptic textures from recorded acceleration data. In Proc. of

IEEE ICRA, pages 1815–1821, May 2010.

[2] Mark Hollins, SJ Bensmaıa, and R Risner. The duplex theory of tactile

texture perception. In Proceedings of the 14th annual meeting of the inter-

national society for psychophysics, pages 115–121, 1998.

[3] Mark Hollins, Richard Faldowski, Suman Rao, and Forrest Young. Per-

ceptual dimensions of tactile surface texture: A multidimensional scaling

analysis. Percept. Psychophys., 54(6):697–705, 1993.

[4] Mark Hollins, Sliman Bensmaıa, Kristie Karlof, and Forrest Young. Indi-

vidual differences in perceptual space for tactile textures: Evidence from

multidimensional scaling. Percept. Psychophys., 62(8):1534–1544, 2000.

[5] Delphine Picard, Catherine Dacremont, Dominique Valentin, and Agnes

Giboreau. Perceptual dimensions of tactile textures. Acta psychologica,

114(2):165–184, 2003.

[6] D Picard, C Dacremont, D Valentin, and A Giboreau. About the salient

perceptual dimensions of tactile texture space. Touch, Blindness, and Neu-

roscience, pages 165–174, 2004.

– 98 –

[7] George A Gescheider, Stanley J Bolanowski, Tyler C Greenfield, and Katya-

rina E Brunette. Perception of the tactile texture of raised-dot patterns: A

multidimensional analysis. Somatosensory & motor research, 22(3):127–140,

2005.

[8] Mark Hollins, Aaron Fox, and Carianne Bishop. Imposed vibration influences

perceived tactile smoothness. Perception, 29(12):1455–1465, 2000.

[9] Mark Hollins and S Ryan Risner. Evidence for the duplex theory of tactile

texture perception. Perception & psychophysics, 62(4):695–705, 2000.

[10] Mark Hollins, Sliman Bensmaıa, and Sean Washburn. Vibrotactile adapta-

tion impairs discrimination of fine, but not coarse, textures. Somatosensory

& motor research, 18(4):253–262, 2001.

[11] Margaret Diane Rezvan Minsky. Computational haptics: the sandpaper sys-

tem for synthesizing texture for a force-feedback display. PhD thesis, Mas-

sachusetts Institute of Technology, 1995.

[12] Margaret Minsky and Susan J Lederman. Simulated haptic textures: Rough-

ness. In Proceedings of the ASME dynamic systems and control division,

volume 58, pages 421–426, 1996.

[13] A Hardwick, S Furner, and J Rush. Tactile display of virtual reality from

the world wide web—a potential access method for blind people. Displays,

18(3):153–161, 1998.

– 99 –

[14] Thomas Harold Massie. Initial haptic explorations with the phantom: Vir-

tual touch through point interaction. PhD thesis, Massachusetts Institute of

Technology, 1996.

[15] Cagatay Basdogan, C Ho, and Mandayam A Srinivasan. A raybased haptic

rendering technique for displaying shape and texture of 3d objects in virtual

environments. In ASME Winter Annual Meeting, volume 61, pages 77–84,

1997.

[16] Chih-Hao Ho, Cagatay Basdogan, and Mandayam A Srinivasan. Efficient

point-based rendering techniques for haptic display of virtual objects. Pres-

ence: Teleoperators and Virtual Environments, 8(5):477–491, 1999.

[17] Vincent Hayward and Dingrong Yi. Change of height: An approach to the

haptic display of shape and texture without surface normal. Experimental

Robotics VIII, pages 570–579, 2003.

[18] Gianni Campion. The Synthesis of Three Dimensional Haptic Textures: Ge-

ometry, Control, and Psychophysics, chapter On the Synthesis of Haptic

Textures, pages 73–97. Springer Science & Business Media, 2011.

[19] Roberta L. Klatzky, Susan J. Lederman, Cheryl Hamilton, Molly Grind-

ley, and Robert H. Swendsen. Feeling textures through a probe: Effects of

probe and surface geometry and exploratory factors. Percept. Psychophys.,

65(4):613–631, 2003.

– 100 –

[20] Miguel A Otaduy and Ming C Lin. A perceptually-inspired force model for

haptic texture rendering. In Proc. of ACM APGV, pages 123–126. ACM,

2004.

[21] Michael A Costa, Mark R Cutkosky, and Sonie Lau. Roughness perception

of haptically displayed fractal surfaces. In Proc. ASME Dynamic Systems

and Control Division, volume 69, pages 1073–1079, 2000.

[22] Steven S Wall and William Seymour Harwin. Modelling of surface identifying

characteristics using fourier series. In ASME Dynamic Systems and Control

Division (Symposium on Haptic Interfaces for Virtual Environments and

Teleoperators), pages 65–71, 1999.

[23] Dinesh K. Pai and Peter Rizun. The WHaT: A wireless haptic textue sensor.

In Proc. of HAPTICS, pages 3–9. IEEE, 2003.

[24] Yasushi Ikei, Kazufurni Wakamatsu, and Shuichi Fukuda. Vibratory tactile

display of image-based textures. IEEE Comput. Graph. Appl., 17(6):53–61,

1997.

[25] Dimitrios A Kontarinis and Robert D Howe. Tactile display of vibratory

information in teleoperation and virtual environments. Presence: Teleoper-

ators & Virtual Environments, 4(4):387–402, 1995.

[26] A.M. Okamura, J.T. Dennerlein, and R.D. Howe. Vibration feedback models

for virtual environments. In Proc. of IEEE ICRA, pages 674–679, May 1998.

– 101 –

[27] S. Andrews and J. Lang. Interactive scanning of haptic textures and surface

compliance. In Proc. of 3DIM, pages 99–106. IEEE, 2007.

[28] S. Andrews and J. Lang. Haptic texturing based on real-world samples. In

Proc. of IEEE HAVE, pages 142–147, Oct 2007.

[29] J. Lang and S. Andrews. Measurement-based modeling of contact forces

and textures for haptic rendering. IEEE Trans. Vis. Comput. Graphics.,

17(3):380–391, March 2011.

[30] Vijaya Lakshmi Guruswamy, Jochen Lang, and Won-Sook Lee. IIR filter

models of haptic vibration textures. IEEE Trans. Instrum. Meas., 60(1):93–

103, January 2011.

[31] J.M. Romano and K.J. Kuchenbecker. Creating realistic virtual textures

from contact acceleration data. IEEE Trans. Haptics, 5(2):109–119, April

2012.

[32] H. Culbertson, J.M. Romano, P. Castillo, M. Mintz, and K.J. Kuchenbecker.

Refined methods for creating realistic haptic virtual textures from tool-

mediated contact acceleration data. In Proc. of IEEE HAPTICS, pages

385–391, March 2012.

[33] H. Culbertson, J. Unwin, B.E. Goodman, and K.J. Kuchenbecker. Generat-

ing haptic texture models from unconstrained tool-surface interactions. In

Proc. of IEEE WHC, pages 295–300, April 2013.

– 102 –

[34] H. Culbertson, J.J. Lopez Delgado, and K.J. Kuchenbecker. One hundred

data-driven haptic texture models and open-source methods for rendering

on 3d objects. In Proc. of IEEE HAPTICS, pages 319–325, Feb 2014.

[35] Norbert Wiener, Norbert Wiener, Cyberneticist Mathematician, Norbert

Wiener, Norbert Wiener, and Cyberneticien Mathematicien. Extrapolation,

interpolation, and smoothing of stationary time series: with engineering ap-

plications. MIT press Cambridge, 1949.

[36] Arsen Abdulali and Seokhee Jeon. Data-driven modeling of anisotropic hap-

tic textures: Data segmentation and interpolation. In Proc. of EuroHaptics,

pages 228–239. Springer, 2016.

[37] Arsen Abdulali, Waseem Hassan, and Seokhee Jeon. Sample selection of

multi-trial data for data-driven haptic texture modeling. In Proc. of IEEE

WHC, pages 66–71, 2017.

[38] Juhani Siira and Dinesh K Pai. Haptic texturing-a stochastic approach. In

Proc. of IEEE ICRA, volume 1, pages 557–562. IEEE, 1996.

[39] D. F Green and J. K. Salisbury. Texture sensing and simulation using the

phantom: Towards remote sensing of soil properties. In Proceedings of the

Second PHANToM Users Group Workshop, 1997.

[40] Jason P Fritz and Kenneth E Barner. Stochastic models for haptic texture. In

Proc. of SPIE, pages 34–44. International Society for Optics and Photonics,

1996.

– 103 –

[41] Nils Landin, Joseph M Romano, William McMahan, and Katherine J

Kuchenbecker. Dimensional reduction of high-frequency accelerations for

haptic rendering. Lecture Notes on Computer Science (Eurohaptics 2010,

Part II), LNCS 6192:79–86, 2010.

[42] Seungmoon Choi and Katherine J. Kuchenbecker. Vibrotactile display: Per-

ception, technology, and applications. Proc. of the IEEE, 101(9):2093–2104,

2013.

[43] Ron Lazebnik. Using Frequency Decomposed Parallel Neural Networks For

System Identification. PhD thesis, Case Western Reseve University, 2002.

[44] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning

internal representations by error propagation. Technical report, La Jolla

Institute for Cognitive Science, The University of California, San Diego,

1985.

[45] J. Romero, A. Garcıa-Beltran, and J. Hernandez-Andres. Linear bases for

representation of natual and artificial illuminants. Journal of the Optical

Society of America, 14(5):1007–1014, 1997.

[46] S. Cholewiak, K. Kim, H. Z. Tan, and B. Adelstein. A frequency-domain

analysis of haptic gratings. IEEE Trans. Haptics, 3(1):3–14, 2010.

[47] Fausto Bernardini and Holly Rushmeier. The 3d model acquisition pipeline.

In Computer graphics forum, volume 21, pages 149–172. Wiley Online Li-

brary, 2002.

– 104 –

[48] James A Paterson, David Claus, and Andrew W Fitzgibbon. BRDF and

geometry capture from extended inhomogeneous samples using flash pho-

tography. In Computer Graphics Forum, volume 24, pages 383–391. Wiley

Online Library, 2005.

[49] A. Spence and M. Chantler. Optimal illumination for three-image photo-

metric stereo acquisition of texture. In Proc. of Texture, pages 89–94, 2003.

[50] Li Zhang, Brian Curless, Aaron Hertzmann, and Steven M Seitz. Shape

and motion under varying illumination: Unifying structure from motion,

photometric stereo, and multiview stereo. In Proc. of IEEE ICCV, pages

618–625. IEEE, 2003.

[51] Philip R Dahl. Solid friction damping of mechanical vibrations. AIAA

journal, 14(12):1675–1682, 1976.

[52] KH Hunt and FRE Crossley. Coefficient of restitution interpreted as damping

in vibroimpact. J. Appl. Mech., 42(2):440–445, 1975.

[53] Sunghoon Yim, Seokhee Jeon, and Seungmoon Choi. Data-driven haptic

modeling and rendering of viscoelastic and frictional responses of deformable

objects. IEEE Trans. Haptics, 9(4):548–559, 2016.

[54] Mohsen Mahvash and Allison M Okamura. Friction compensation for a force-

feedback telerobotic system. In Proc. of IEEE ICRA, pages 3268–3273, 2006.

– 105 –

[55] Feei Wang, Terril Hurst, Daniel Abramovitch, and Gene Franklin. Disk

drive pivot nonlinearity modeling. ii. time domain. In American Control

Conference, volume 3, pages 2604–2607. IEEE, 1994.

[56] Seokhee Jeon, Seungmoon Choi, and Matthias Harders. Rendering virtual

tumors in real tissue mock-ups using haptic augmented reality. IEEE Trans.

Haptics, 5(1):77–84, 2012.

[57] Seokhee Jeon and Seungmoon Choi. Real stiffness augmentation for haptic

augmented reality. Presence: Teleop. Virt., 20(4):337–370, 2011.

[58] Amir Haddadi and Keyvan Hashtrudi-Zaad. A new method for online pa-

rameter estimation of hunt-crossley environment dynamic models. In Prof.

of IEEE IROS, pages 981–986, 2008.

[59] Seungmoon Choi, Laron Walker, Hong Z. Tan, Scott Crittenden, and Ron

Reifenberger. Force constancy and its effect on haptic perception of virtual

surfaces. ACM Trans. Appl. Percept., 2(2):89–105, April 2005.

[60] Shogo Okamoto, Hikaru Nagano, and Yoji Yamada. Psychophysical dimen-

sions of tactile perception of textures. IEEE Trans. Haptics, 6(1):81–93,

2013.

[61] Heather Culbertson and Katherine J. Kuchenbecker. Importance of matching

physical friction, hardness, and texture in creating realistic haptic virtual

surfaces. IEEE Transactions on Haptics, 10(1):63–74, 2017.

– 106 –

[62] Paul Strohmeier and Kasper Hornbæk. Generating haptic textures with

a vibrotactile actuator. In Proceedings of the CHI Conference on Human

Factors in Computing Systems, pages 4994–5005. ACM, 2017.

[63] Sunghwan Shin and Seungmoon Choi. Geometry-based haptic texture mod-

eling and rendering using photometric stereo. In Proc. of IEEE HAPTICS,

pages 262–269, 2018.

[64] Hugh B Morgenbesser and Mandayam A Srinivasan. Force shading for haptic

shape perception. In American Society of Mechanical Engineers, Dynamic

Systems and Control Division (Publication) DSC, volume 58, pages 407–412,

1996.

– 107 –

Acknowledgements

학부 과정까지 포함해서 정말 오랜 시간을 포항에서 보냈습니다. 그리고 그 긴

시간 동안 정말 많은 사람들에게 도움을 받고, 배우고, 영향을 받으며 제가 지금의

저라는 사람이 되고, 박사 과정을 어떻게든 마칠 수 있었던 것 같습니다. 먼저 제

대학원 생활을 지도해주신 최승문 교수님께 감사드립니다. 교수님은 학문적인 내용

뿐만 아니라 인생의 선배이자 훌륭한 학자로서의 입장에서 제게 조언을 해 주시고

또 본받을 만한 모습을 보여주시면서 제 롤 모델이 되어주셨습니다. 제 박사 연

구와 논문을 심사해주시고 좋은 지적을 해 주셨던 한성호, 이승용, 조민수, 전석희

교수님께도 감사드립니다. 많은 교수님들의 지도을 통해서 저 혼자만의 생각을 더

발전시켜 연구를 진행할 수 있었습니다.

연구실 생활은 다소 개인적인 성향이 있었던 제게는 새로운 배움의 연속이었습니

다. 오랜 기간 연구실에 있으면서 많은 선배, 친구, 후배들을 만났지만 모두 좋은

사람이어서 즐겁게 연구실 생활을 할 수 있었습니다. 지금은 사회의 여러 분야로

진출해 있는 성훈이형, 재봉이형, 인이형, 인욱이형, 갑종이형, 건혁이형, 경표형,

명찬이형, 종만이형, Reza 등 선배들에게는 연구실의 전반적인 생활부터 실제로 연

구를 수행하면서 필요한 많은 기술을 배울 수 있어 고마웠습니다. 동년배인 호진,

용재, 승재, 종호는 좋은 친구가 되어주었습니다. 성인이 된 후 만난 친구들이지만

스스럼 없이 편하게 대할 수 있는 친구들 덕분에 연구실 생활이 한결 편했던 것 같

습니다. 후배들인 호준이, 준석이, Phoung, 인석이, 성호, 한슬이, 상윤이, 성원이,

혜진이,선웅이,겨레,효승이,채용이,지완이,범수,준경이에게는제가선배로서줄

수있는가르침또는도움을주지못해서항상미안한마음이있습니다. 그래도이런

– 108 –

선배를 많이 도와주고 말벗이 되어주어서 고맙게 생각합니다. 다들 현명한 친구들

이니, 연구실 생활을 통해서 본인이 원하는 길을 잘 찾아가길 바랍니다. 연구실에서

학생들이 연구에 전념할 수 있게 도와주신 오송이 선생님께도 감사드립니다.

마지막으로, 오랜 시간 제가 제 뜻대로 할 수 있도록 믿음을 보내주신 부모님께

감사드립니다. 저마저도 이 길이 제 길이 아닐까 하고 의심한 적이 많았지만, 항상

제게신뢰를보내주신부모님이아니었으면끝까지공부를마칠수없었을것입니다.

그리고오랜기간바쁘다는핑계에도기다려주고많은도움을준민희에게고맙다는

말을 하고 싶습니다.

Curriculum Vitae

Name : Sunghwan Shin

Education

2006 – 2010 Department of Physics and Computer Science and Engi-

neering (Double Majors), Pohang University of Science and

Technology (B.S.)

2010 – 2018 Department of Computer Science and Engineering, Pohang

University of Science and Technology (Direct Ph. D Course)

Affiliation

1. Haptics and Virtual Reality Lab., Department of Computer Science and

Engineering, Pohang University of Science and Technology