Detecting CNV-like variation when remembering and generating continuous motion Hiroyuki Iizuka, Mika Sunagawa, Masataka Niwa, Hideyuki Ando and Taro Maeda

Graduate School of Information Science & Technology, Osaka University,

2-1 Yamadaoka, Suita, Osaka, 565-0871, Japan {iizuka, sunagawa.mika, hide, t_maeda}

@ist.osaka-u.ac.jp

Center for Information and Neural Networks, NICT,

1-3 Yamadaoka, Suita, Osaka, 565-0871 niwa@ist.osaka-u.ac.jp

Abstract— Our aim of this study is to investigate the possibility of estimating interruptibility from Electroencephalograms (EEG). We focus on contingent negative variation which is elicited in response to concentration. Our results support that our method could correctly detects the user’s internal states such as just watching or actively remembering even the external situations were completely same and also CNV is elicited when controlling continuous motion.

Keywords- Ambient interface;Contingent negative variation; EEG; interruptibility

I. INTRODUCTION

Ambient information society provides just the right amount of desired information to users at the appropriate time and place by the appropriate way [1]. Contrarily, while the concept of ubiquitous information society is based on “anywhere, anytime, anyone”, the concept of the ambient information society can be express as “information is given to users according to current time, place, and personal preferences”.

In order to realize such a user-friendly society, the most important and difficult thing for a system is to detect the right timings to give information to users. The wrong timings of interference make user’s work inefficient. The timing and length of the interruptions severely affect the work performance [2]. There are some studies that estimate human interruptibility from simple sensors such as cameras and microphones (e.g. [3]). However, these methods can estimate human interruptibility only in the obvious cases where the abrupt changes happen in sensory information. Except for these obvious situations, there is still difficulty in estimating interruptibility.

Therefore, our aim of this study is to investigate the possibility of detecting segments of thought from Electroencephalograms (EEG). In our experiment, two conditions are considered. One is that users are doing nothing but watching something. This condition can be regarded as the situation where the user passively obtains information from some media. Another is a situation where users are actively doing something like working on a desk or building something. In these passive and active conditions, we investigate to detect time between concentrations from EEG.

II. EXPLOITING CNV TO DETECT SEGMENTS

To detect such segments of thought, we focus on the Contingent Negative Variation (CNV). CNV is elicited when

people are concentrating on the interval between warning (S1) and imperative (S2) stimuli [4]. After the S1, subjects start concentrating on the task to which they have been asked to react (e.g. pushing a button) soon after S2 is given. During concentration, the brain wave develops negatively between S1 and S2, and it is known that the more a person concentrates, the larger is the CNV elicited. As another feature of CNV, the brain wave suddenly shifts to positive after the S2 stimuli.

In the experiment, the subject watches human continuous motion. Since it is already known that the CNV-like variation is elicited in response to discrete events, we investigate the possibilities to detect CNV-like variation depending on segments of concentration while watching continuous flows of motions. Since it is expected that significant still images in the continuous motions play the roles of S1 and S2 stimuli, CNV is used as cues of behavioural intentions, such that the segmentation of behavioral intentions can be captured with EEG automatically. We expect that negative variations like CNV would be elicited by significant still images in continuous motions during motion remembering. As well as watching and remembering the continuous motion passively, we also expect that the negative variations can be elicited even when the user generates the continuous motion. In our experiments this hypothesis is tested.

III. EXPERIMENTS

In order to investigate whether the CNV-like variation is elicited when participants are observing and generating continuous motion, we set up the following four tasks: letter-remembering (LR), motion-remembering (MR), motion-observing (MO) and motion-generation (MG).

LR task is performed to obtain the template of the brain waveform that represents the most typical brain waves related to externally given segmentation. Participants were given explicitly segmented visual stimuli, which were four alphabetic letters displayed successively on a monitor, and they were required to remember the letters. In the MR task, participants observed and remembered human motion and EEG was recorded during observation. MO condition investigates whether EEG changes depending on the user’s intention. In MR, the subject is actively performing task but is just watching in MO. MG task collects EEG when the participants generate the continuous motions. However, there are some difficulties in measuring EEG when they are allowed to move. Therefore, we measure EEG when they operate a controller to move a humanoid robot with Tsumori

IEEE Virtual Reality 201316 - 20 March, Orlando, FL, USA978-1-4673-4796-9/13/$31.00 ©2013 IEEE

interface where the subject’s voluntariness can be maintained different from the conventional command-type interface [5].

IV. RESULT

A. LR, MR and MO tasks �igure 1 shows the examples of averaged EEG data

recorded in the LR, MR and MO task. All EEG data was averaged around time when the subjects had manually selected the important still image in the continuous motion. In the LR task, CNV-like variation is observed in response to the externally given discrete events in which an alphabetic letter is display on a monitor. Soon after displaying the letters, the variation suddenly developed positively. In the MR task, the similar variations were observed in all subjects same as the LR task despite the fact that the participants were watching just continuous motion in the MR task. However, interestingly even the subject were watching the same continuous motion, the CNV-like variation was not observed in the MO task.

The correlation of MR-EEG with the template of LR-EEG is shown in �ig. 2. The correlation becomes highest near time of important frames in the continuous motion. On the other hand, the correlation becomes close to -1 in between those frames.

There are two findings from our results. One is that CNV-like variation can be elicited when subjects remember human continuous motions because segmented information is stored in our brain to remember human motions and the timing of segmentations plays a role of S1 and S2 stimuli as they concentrate on those stimuli in the conventional CNV experiments. Another one is that negative shifts of the CNV-like variation were shifted to 250-750 ms before the timings manually selected as significant frames.

�igure 1. EEG recorded in the LR, MR and MO tasks. A to � indicates different subjects.

�igure 2. Correlation (green line) between LR and MR in a trial. The dotted line shows the selected important frames and the red line show time when correlation becomes maximum.

B. MG task �igure 3 shows the EEG when performing the MG task.

These data are also averaged around the selected important frames in the continuous motion. We found that CNV-like variations were also elicited when generating the continuous motion. In the previous studies, it was difficult to measure EEG when subject is allowed to move. However, in our experiment, Tsumori interface made it possible.

�igure 3. EEG recorded in the MR task.

V. CONCLUSION

Our results support that our method with EEG could correctly detects the user’s internal states such as just watching or actively remembering even the external situations were completely same. It is difficult for the conventional methods which estimate interruptibility from sensors because the system just captures the situations. And also our method could detect the waves of concentrations in controlling continuous motion of a humanoid robot. It means that there must be segments in the continuous work, which can be detected from EEG data.

As a future work, the validity of the segments detected by EEG for the interruptibility signal in a daily work is tested. We need to investigate whether the interruption by our method is performed at an appropriate time for users or not.

AC�NOWLEDGMENT

This work was partially supported by grant-in aid (No. 22240008)

RE�ERENCES

[1] Global COE Program, Center of Excellence for �ounding Ambient Information Society Infrastructure by Osaka University, http:��www.ist.osaka-u.ac.jp�GlobalCOE

[2] McDaniel, M., Einstein, G.O., Graham, T., Rall, E.: Delaying Execution of Intentions:Overcoming the Costs of Interruptions. Applied Cognitive Psychology 18, 533�547 (2004) I. S. Jacobs and C. P. �ean, “�ine particles, thin films and exchange anisotropy,” in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1�63, pp. 271�350.

[3] Hudson, S., �ogarty, J., Atkeson, C., Avraham, D., �orlizzi, J., �iesler, S., Lee, J., & Yang, J. (2003). Modeling user behavior: Predicting human interruptibility with sensors: a Wizard of Oz feasibility study. ACM CHI. p. 257-164.

[4] Walter, W.G., Cooper, R., Aldrodge, V.J., McCallum, W.C., and Winter, A.L. (1�64). Contingent negative variation: An electric sign of sensorimotor association and expectancy in the human brain, Nature, vol. 203, pp. 203-380.

[5] Niwa, M, Okada, S., Sakaguchi, S., Azuma, �., Iizuka, H., Ando, H., and Maeda, T. (2010) Detection and Transmission of �Tsumori�:an Archetype of �ehavioral Intention in Controlling a Humannoid Robot, 20th International conference on artificial reality and telexistence.