Transcript
Page 1: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Tokyo, Japan, 4-5 February 2013

Cognitive neuroscience approach to telemedicine

Makio KASHINO, Ph.D. Senior Distinguished Scientist, Executive Manager

NTT Communication Science Laboratories [email protected]

ITU Workshop on “E-health services in low-resource settings:

Requirements and ITU role”

(Tokyo, Japan, 4-5 February 2013)

Page 2: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Tokyo, Japan, 4-5 February 2013 2

Human Information Lab, NTT CSL

���������� �

����������

������������������

�01���

23453#678479:�

25:;73453#678479:�

<=>7:�

!7?#7@A7347BC7�

������

������

���������� �

����������

������������������

�01���

23453#678479:�

25:;73453#678479:�

<=>7:�

!7?#7@A7347BC7�

������

������

Informatics

Neuroscience

Psychophysics

Man-machine interface

Perception, Emotion, Motor control

Page 3: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Tokyo, Japan, 4-5 February 2013 3

Challenges in telemedicine

!   Transmission delay and asynchrony

!   Guiding a remote collaborator

!   Face-to-face-like communication

Page 4: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Compensation for transmission delay

Tokyo, Japan, 4-5 February 2013 4

����� �������

�+�����

�� "����

����

����� ��

��������

Takamuku, Gomi (2011) ECVP2011

No delay With delay Delay+flow Delay Delay

Heavy

Smooth Smooth Awkward

Page 5: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Compensation for transmission delay

Tokyo, Japan, 4-5 February 2013 5 Takamuku, Gomi (2011) ECVP2011

Page 6: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Compensation for transmission delay

Tokyo, Japan, 4-5 February 2013 6 Takamuku, Gomi (2011) ECVP2011

Page 7: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Involuntary arm movement induced by vision

Tokyo, Japan, 4-5 February 2013 7

1717

�� ������������ ��������������������������

Movement duration: 1.0 sassisted by beeping soundsFixation marker

0.05 [m]

XY 0.

05

[m]

RLC

110 ms

85 ms

91 ms

norm

aliz

ed

Saijo et al. 2005 J.Nneurosci.

MFR (Manual Following Response)

����������

1717

�� ������������ ��������������������������

Movement duration: 1.0 sassisted by beeping soundsFixation marker

0.05 [m]

XY 0.

05

[m]

RLC

110 ms

85 ms

91 msno

rmal

ized

Saijo et al. 2005 J.Nneurosci.

MFR (Manual Following Response)

����������

1717

�� ������������ ��������������������������

Movement duration: 1.0 sassisted by beeping soundsFixation marker

0.05 [m]

XY 0.

05

[m]

RLC

110 ms

85 ms

91 ms

norm

aliz

ed

Saijo et al. 2005 J.Nneurosci.

MFR (Manual Following Response)

����������

Saijo, Murakami, Nishida, Gomi (2005) The Journal of Neuroscience

Manual Following Response (MFR)

・Involuntary ・Fast

Page 8: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Different visual processing systems for perception and motor control

Tokyo, Japan, 4-5 February 2013 8 Gomi, Abekawa, Nishida (2006) The Journal of Neuroscience

Fast motion of a coarse pattern is not salient perceptually, but effective to induce MFR

Page 9: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Audiovisual simultaneity

Tokyo, Japan, 4-5 February 2013 9

How to judge A-V timing in the presence of significant external and internal asynchrony?

Page 10: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Adaptation to A-V asynchrony induces…

Tokyo, Japan, 4-5 February 2013 10 Fujisaki, Shimojo, Kashino, Nishida (2004) Nature Neuroscience

Page 11: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Recalibration of A-V simultaneity

Tokyo, Japan, 4-5 February 2013 11

“Sim

ult

aneo

us”

res

ponse

A first V first

N=7

Test AV lag (ms)

adaptation (0ms)

adaptation (-235ms)

adaptation (+235ms)

Fujisaki, Shimojo, Kashino, Nishida (2004) Nature Neuroscience

Page 12: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Tokyo, Japan, 4-5 February 2013 12

Challenges in telemedicine

!   Transmission delay and asynchrony

!   Guiding a remote collaborator

!   Face-to-face-like communication

Page 13: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Perceptual attraction force: Buru-Navi

Tokyo, Japan, 4-5 February 2013 13

Amemiya, Gomi (2012) IEEE Transactions on Haptics

Amemiya, Ando, Maeda (2007) Laval Virtual 2007

How to generate “pull” or “push” sensation in the mobile devices having no external fulcrum?

Page 14: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Navigation by Buru-Navi

Tokyo, Japan, 4-5 February 2013 14 Amemiya, Sugiyama (2009) ASSETS

Successfully navigated ~90% (21/23) of visually-impaired people in a maze

Page 15: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Tokyo, Japan, 4-5 February 2013 15

Challenges in telemedicine

!   Transmission delay and asynchrony

!   Guiding a remote collaborator

!   Face-to-face-like communication

Page 16: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Implicit InterPersonal Information

Tokyo, Japan, 4-5 February 2013 16

Supported by JST CREST (PI: Makio Kashino [NTT], Collaborators: Sinsuke Shimojo [Caltech], Katsumi Watanabe [U Tokyo])

������������������������� �������

���� ���������������������� �������

!"#

!$#

!%#

!&#

!'#

!(#

)#

(#

'#

&#

%#

$#

"#

*#

+#

,#

()#

)# (# '# &# %# $# "# *# +#

-./#

01#

)#

()#

')#

&)#

%)#

$)#

")#

)# (# '# &# %# $# "# *# +#

-./#

01#

!"#

!$#

!%#

!&#

!'#

!(#

)#

(#

'#

&#

%#

$#

"#

)# (# '# &# %# $# "# *# +#

!"#$%&'()*+),#!(-'.)

01#

!"#

!$#

!%#

!&#

!'#

!(#

)#

(#

'#

&#

%#

$#

"#

)# (# '# &# %# $# "# *# +#

,#!(-'.)*+),#!(-'.)

01#

Gamma, 40-60 Hz (650ms) Theta, 5-8 Hz (500ms)

ERPs after Face1 presented (200-400ms)

Wavelet-based time-freq. plot, after Face2 presented

All before decision! (Lindsen, et al., NeuroImage, '10)

Two face presented sequentially.

22

ⓒ Caltechⓒ Caltech

IIPI (interpersonal interaction of involuntary and subconscious body movements and physiological responses) may play critical roles in communication → Decoding and controlling IIPI for better communication systems

Interpersonal interaction of brain and ANS activities

Interpersonal interaction of body movements

Page 17: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Acoustic IIPI for talker “presence”

Tokyo, Japan, 4-5 February 2013 17

Kobayashi, Ooishi, Enomoto, Kitagawa, Ueno, Ise, Kashino (in preparation)

Spontaneous utterances recorded and reproduced by the sound field sharing system “BoSC” (CREST Ise team)

→ Subjective rating, ANS activity, hormone concentration

Dynamic Static

Page 18: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Acoustic IIPI for talker “presence”

Tokyo, Japan, 4-5 February 2013 18

Kobayashi, Ooishi, Enomoto, Kitagawa, Ueno, Ise, Kashino (in preparation)

n  Significant differences also found for sympathetic nerve activity and salivary hormone concentration

n  Subjective rating 1. Artificial sound? 2. Feel talker presence? 3. Like the talker? 4. Interested in the content?

Page 19: Cognitive neuroscience approach to telemedicine - … · Cognitive neuroscience approach to telemedicine Makio KASHINO, Ph.D. Senior Distinguished Scientist, ... Wavelet-based time-freq

Concluding remark

Tokyo, Japan, 4-5 February 2013 19

!   Challenges in telemedicine may be overcome by taking advantage of brain mechanisms underlying human perception, emotion, motor control, and communication.


Top Related