transparency analysis and haptic synchronization scheme for force-reflecting teleoperation

DEPT. OF INFO. & COMM., GISTNetworked Media Lab.

Networked Media LaboratoryDept. of Information & Communications

Gwangju Institute of Science & Technology (GIST)[email protected]

http://nm.gist.ac.kr/~shlee

1

Transparency Analysis and Haptic Synchronization Scheme for Force-reflecting Teleoperation

Seokhee Lee

Lab. Seminar 2009.08.15


Contents

Introduction Requirements of Force-reflecting teleoperation EBA-based Teleoperation Delay Jitter Problem and Related Work

Transparency Analysis-based Ap-proach Transparency Analysis Haptic Synchronization Simulation and Experiment Results

Conclusions and Future Work

2


Requirements of Force-reflecting Teleoperation

Stability The primary requisite for safe system If the output response are bounded for all bounded inputs,

the system is said to be stable. Instability

Uncontrollable oscillations and chaotic behavior Sometimes, serious damages to the user

Transparency Transparency ≈ haptic realism Mathematically more difficult to analyze since the ultimate

goal is to make the user experience a “good feeling” Optimal transparency

The user cannot distinguish between direct and tele-interaction with a remote environment.

3


EBA-based Teleoperation

EBA (energy bounding algorithm) Stability algorithm of a haptic simulation system (Kim & Ryu, 2004)

EBA passifies virtual environment and restricts the energy generated in the ZOH (zero order hold) within a consumable energy limit in the haptic device.

Can be applied to teleoperation to ensure robust stability regardless of the amount of time delays and packet losses (Seo et al. 2008).

4


EBA-based Teleoperation Master EBA

5

Slave EBA

,max ,max

,min ,min

,max 1 ,max

,min ,min

,01 1

2

0

,max

with the following bounding laws:( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

where( ) min( , ( )),

( ) ( ),

( ),

( 1)

(

m m m m

m m m m

m m m

m m

MD MDm n

mk

m

if n n then n n

if n n then n n

n c n

n n

P nc

X k

n

2

22 2

2

2,min 2 2

2

( 1) ( 1)) ,

( ) ( )

( 1) ( 1)( ) ,

( ) ( )

where is a positive constant.

m mm m

m m

m mm m m

m m

m

F n F nc c

X n X n

F n F nn c c

X n X n

c

Control law Control law

Bounding law Bounding law

( ) ( 1) ( ) ( )where

( ) ( 1)( ) ( ) 0( )

( ) ( ) ( 1)

mEBA mEBA m m

md mEBAm m

m

m m m

F n F n n X n

F n F nn for X n

X nX n X n X n

,max ,max

,min

( ) ( 1) ( ) ( )where

( ) ( 1)( ) ( ) 0

( )( ) ( ) ( 1),

( ) ( ) ( ),with the following bounding laws:

( ) ( ) ( ) ( )

( ) (

sEBA sEBA s s

s sEBAs s

s

s s s

s sd s

s s s s

s s

F n F n n e n

F n F nn for e n

e ne n e n e n

e n x n x n

if n n then n n

if n n

,min

,max 1 ,max

,min ,min

,01 1

2

0

2

2,max 2 2

2,min 2 2

) ( ) ( )

where( ) min( , ( )),

( ) ( ),

( ),

( 1)

( 1) ( 1)( ) ,( ) ( )

( 1) ( 1( )

( )

s s

s s s

s s

SD SDs n

sk

s ss s s

s s

s ss s s

s

then n n

n c n

n n

P nc

e k

F n F nn c ce n e n

F n F nn c c

x n

2

2

),

( )

where is a positive constant.s

s

x n

c


Delay Jitter Problem Delay jitter effect of haptic event

Delayed data transmission, out-of-order arrivals, and empty sampling instances

Teleoperation with delay jitter Instability Transparency deterioration

EBA with delay jitter EBA guarantees stable teleoperation over net-

work delay and packet losses Limitation

It cannot overcome transparency deterioration according to time-varying network situation.

6


7

Related Work Adaptive buffering control using moving-average smooth-ing technique (Wongwirat & Ohara, 2006) Buffering time is adjusted to twice the moving average delay. Problem: the authors mentioned the importance of an optimum buffer

size for the transparency but it remained further study. Adaptive buffering control with interpolation scheme (Berestesky et al. 2004) Stability is guaranteed by compressing and expanding buffered data. Problem: although this scheme improved performance of position

tracking and stability, it did not focus on transparency of force. VTR (virtual time rendering) (Ishibashi et al. 2004)

Dynamically adapting the buffering time to improve the interactivity of haptic events in haptic-based NVEs

Problem: little attention has been given to the transparency in the force-reflecting teleoperation.


8

Transparency Analysis-based Approach Transparency analysis

Quantifies the force feedback distortions caused by network delay and packet loss.

Predicts the maximum allowable delay and loss for the predefined transparency requirements.

Haptic synchronization based trans-parency analysis Improves transparency over time-varying delay.

By controlling the playout time of the transmitted haptic event with the transparency-related parameters

By synchronizing the local haptic event with the trans-mitted event according to the transparency analysis


Transparency Analysis Definition of Transparency

Similarity between the force feedback for a slave robot (F-

sEBA) and that for a user in master side (FmEBA) Force feedback increase according to the

user's input Robot keeps in contact with a wall and a user feels the force

feedback by moving haptic device facing the wall with con-stant velocity (vm).

From the control law in master EBA

9

/

1

/

,max ,max1

/

1 ,max1

( ),

( )

min( , ( ))

inter

inter

inter

T

in m mn

T

in m mn

T

m m mn

F v n

F v n

v c n

,

: update time period ( ): interaction time ( )

: increase of ( ): maximum value of ( )

inter

in mEBA

in max in

secT secF F NF F N


Transparency Analysis Approximation of force

feedback increase Assumption: c1m≥2c2m

γm,max(n) in bounding laws con-verges into c2m when Fm(n-1) increases monotonically.

Force feedback decrease caused by delay Network delay reduces Tinter as

much as the delayed time If the network delay in-

creases, the force feedback decreases in proportional to c2m∙vm.

10

,max 2in m m interF c v T

, 2de delay m m delayF c v T

, : decrease according to delay

: network delayde delay mEBA

delay

F F

T


Transparency Analysis Force feedback decrease

(Fde,loss) caused by loss Transparent loss time (Ttr,loss): time

period when the force feedback continuously increases even though there exist the packet losses

Maximum allowable delay (Tal,delay) and loss (Tal,loss) Predefined transparency require-

ments Maximum allowable force feedback

(Fal) Maximum allowable force feedback

decrease by packet loss (Fal,loss)

11

,,

2

loss mEBA losstr loss

m m

F FT

c v

, 2 ,( )de loss m m loss tr lossF c v T T

max: maximum force feedback without delay and lossF

,, ,

2

al lossal loss tr loss

m m

FT T

c v

max ,,

2

al loss alal delay

m m

F F FT

c v

,

: the latest received force feedback before the packet losses

: the latest updated FmEBA

before the packet losses: loss time (time period when

the pack

loss

mEBA loss

loss

F

F

Tet losses happen)


Haptic Synchronization Transparency im-

provement Output time control of

delayed force Controls playout time of

transmitted event. With transparency-re-

lated parameters Output time control of

local position Synchronizes local event

with transmitted event. To minimize the de-

crease of interaction time caused by delay

12


Haptic SynchronizationOutput time control of delayed force Ideal target output time xn

f

The time at which the event should be output in the case where network de-lay jitter is always smaller than an estimated maxi-mum network delay jitter Jmax

Target output time tnf

The time when the haptic event should be output in the case where network de-lay jitters exists

13

1 1 max 1 1 ,1

1 ,

1 1

( ), if , otherwise

( ) ( 2)

f f fal delayf

fal delay

f f f fn n

D A J D T Tx

T T

x x T T n

1 1

1

*

( 2)

( 1)

f f

f fn n n

f fn n n

t x

t x S n

t t S n

: output time of force, : arrival time

: generation time of force

fn n

fn

D A

T

0 1

*

: slide time: total slide time

0, ( 1)

: modified target output time of force

n

n

n n n

fn

SS

S S S S n

t


Haptic SynchronizationOutput time Dn

f

By comparing the arrival time An and the target output time tn

f Virtual time expansion

Delays the target output time Total delay increase, loss rate decrease

To minimize the transparency degradation caused by delay

Only when packet loss time is larger than allowable packet loss time

Virtual time contraction Advances the target output time Total delay decrease, loss rate increase

To minimize the transparency degradation caused by loss

Only when the haptic interactions do not happen

14

,if

loss al loss

fn n n

T T

S A t

Virtual time expan-sion

Virtual time contrac-tion

,

1

if [( or 0) and

( )]

( , , )else 0

m mEBA

f fn n al delay

fn n n n

n

X F

t T T

S min r S t A

S

*

, ,

ff n n nn f

n

A if A tD

t otherwise


Haptic Synchronization Output time control of local position

In order to minimize the decrease of the interaction time Tin-

ter caused by delay, the playout time of the local position Xm is synchronized with the transmitted haptic event Fds.

Ideal target output time xnP

Output time DnP

If the virtual-time expansion or contraction is executed for transmitted event, the target output time and output time of Xm are also changed.

15

1 1 1 1

1 1

( )

( ) ( 2)

p p f f

p p p pn n

x T t T

x x T T n

1 1

( 2)

p p

p pn n n

p pn n

t x

t x S n

D t

: generation time of positionpnT

: target output time of positionpnt


Simulation Results MATLAB/SIMULINK sim-

ulation Verification of the the trans-

parency analysis Wall contact motion

Slave robot keeps in con-tact with the wall.

Haptic device movement vm=0.05 m/s

16

1 2

0.5 , 20000PD controller: _ 200 /EBA: 1000 , 200

s e

s s

M Kg K N mproportional gain N m

c N m c N m

1 2EBA: 1000 , 200m mc N m c N m


Simulation Results Delay effect

Transparency analysis Fde,delay=10∙Tdelay

Delay increase of 100ms -> force decrease of 1N

Simulation results With delay 0~600ms

Loss effect Transparency analysis

Ttr,loss=0.002 s Fde,loss=10 ∙(Tloss-0.002)

Packet losses for 50 ms -> force decrease of 0.48N

Simulation results With loss time 2~300ms

17

Simulated results closely follow predicted values -> the transparency analysis is valid


Experimental Results

Verification of the pro-posed scheme Transparency improvement

over network delay jitter Assumptions

Maximum allowable force feedback decrease by packet loss (Fal,loss)=1N

Time-varying network delay

18

1 2

PD controller: _ 50 / _ 0.1 /EBA: 1000 , 50s s

proportional gain N mderivative gain Ns m

c N m c N m

1 2EBA: 1000 , 50m mc N m c N m

300 , 50Avg ms Std ms


Experimental Results Wall contact motion

vm=0.2 m/s Transparency com-

parison Each force feedback

with different schemes is compared with the transparent force feed-back (FsEBA).

RMS force feedback er-rors Moving-average adap-

tive buffering (MAB) = 3.7 N

VTR = 3 N Skipping = 2.6 N Proposed scheme = 1 N

19

Transpar-ent force feedback


Remote Calligraphy System

20

Master

Haptic data

transport

Haptic synchronization

FmdMaster EBA

Xm

FmEBA

Slave

Haptic synchronization

Haptic data

transport

Xsd

Fs

Xm

Slave EBA

Xs

FsEBA

PD control

+-

es

Fs

Fds

Xdm

HIP position

Virtual brush position

HIP

0

0.5

1

1.5

2

0 10 20 30 40 50 60 70 80 90

Forc

e (N

)

Time (s)

FmEBAFsEBA

No network de-lay Standard shape

and force Transparent force

feedback


Remote Calligraphy System The user writes the character as feeling the similar

force feedback to the transparent force feedback Force feedback error is less than 0.2 N

The force errors between the reaction forces and the standard force 0.6 N (MAB), 0.4 N (VTR), 0.3 N (skipping), and 0.1 N (proposed scheme)

21

0

0.5

1

1.5

2

2.5

0 10 20 30 40 50 60 70 80 90

Forc

e (N

)

Time (s)

FmEBA without delay FmEBA with MABFmEBA with VTR FmEBA with skippingFmEBA with the proposed scheme

0

0.5

1

1.5

2

2.5

0 10 20 30 40 50 60 70 80 90

Forc

e (N

)

Time (s)

FsEBA without delay FsEBA with MABFsEBA with VTR FsEBA with skippingFsEBA with the proposed scheme


Remote Calligraphy System User can write the character most similarly to

the standard shape by using the proposed scheme.

With the other schemes, although the user thinks that he or she is writing the character well, unintended tick lines are drawn.

22

with MAB with VTR with skippingwith the proposed scheme


23

Conclusions Transparency analysis and haptic synchroniza-

tion scheme for EBA-based force-reflecting teleoperation Transparency analysis

The force feedback distortions according to network variations are quantified.

Haptic synchronization scheme based on transparency analysis

The optimization of the adaptation parameter (buffering time) of the scheme for realistic haptic interactions (transparency)

Simulation and experimental results Transparency analysis provides an acceptable quantification

method The scheme guarantees more transparent haptic interactions

over time-varying network delays


24

Future Work Accuracy and generality improvement

More simulations and experiments with various haptic in-teraction scenarios

Paper work Taylor & Francis, Cybernetics and Systems (regular issue) Taylor & Francis, International Journal of Human-Computer

Interaction (regular issue) Springer’s Multimedia Tools and Applications (special issue:

Multimodal Interaction and Multimodal Content Manage-ment)

Manuscript submission deadline: 1 October 2009 Notification of acceptance: 1 December 2009

IEEE Transaction on Systems, Man, and Cybernetics (regular issue)


25

Questions & Comments

END


ReferencesJ.-P. Kim and J. Ryu, “Stable haptic interaction control using energy bound-

ing algorithm,” in Proc. IEEE/RSJ IROS, 2004.

C. Seo, J. Kim, J. Kim, J. Yoon, and J. Ryu, “Stable bilateral teleop-eration using the energy-bounding algorithm: Basic idea and feasibility tests,” in Proc. IEEE/ASME AIM, 2008.

O. Wongwirat and S. Ohara, “Haptic media synchronization for re-mote surgery through simulation,” IEEE Multimedia, 2006.

P. Berestesky, N. Chopra, and M. W. Spong, “Discrete time passiv-ity in bilateral teleoperation over the internet,” in Proc. IEEE ICRA, 2004.

Y. Ishibashi, H. Kasugai, and M. Fujimoto, “An intra-stream syn-chronization algorithm for haptic media in networked virtual environments,” in Proc. ACM SIGCHI ACE, 2004.

26

transparency analysis and haptic synchronization scheme for force-reflecting teleoperation

Documents

gistnetworked media

transparency of force

network delay

buffering time

networked media laboratorydept

based teleoperationeba

delay jittereba

haptic synchronization