Exploring EEG for Object Detection and Retrieval

Eva Mohedano Amaia Salvador Sergi Porta Xavi Giró Graham O’Healy Kevin McGuinness Noel O’Connor Alan Smeaton

ACM International Conference on Multimedia Retrieval (ICMR) 2015.June 23-26, 2015. Shanghai, China.

Outline

● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work

2

Motivation

Are EEG signals useful for Image Retrieval?

?

3

Motivation

If so, are they comparable to mouse interaction?

VS

4

Outline

● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work

5

Related Work: Rapid Image RetrievalWang, J., Pohlmeyer, E., Hanna, B., Jiang, Y. G., Sajda, P., & Chang, S. F. (2009, October). Brain state decoding for rapid image retrieval. In Proceedings of the 17th ACM international conference on Multimedia (pp. 945-954). ACM.

6

Related Work: Rapid Image RetrievalJun Yang, “A General Framework for Classifier Adaptation and its Applications to Multimedia”. Phd thesis. Carnegie Mellon University (2009).

Section 7.2: Adaptation of EEG-based Relevance ModelsCross-user adaptation to avoid re-training EEG-based SVM classifiers for relevance prediction.

7

Related Work: EEG Object Detection

8

Related Work: EEG Object Detection

Ref: Kapoor, Ashish, Pradeep Shenoy, and Desney Tan. "Combining brain computer interfaces with vision for object categorization." Computer Vision and Pattern Recognition, 2008. CVPR 2008. 9

Outline

● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work

10

Methodology: DatasetSubset of TRECVID INS 2013 Dataset● 3 Instances (4 local regions/instance)● For each topic, 1000 Images

○ 50 relevant (targets) and 950 non-relevant (distractors)

11

Methodology: RSVP

12

Methodology: Image Sorting

Unknown

200=10/190 images @ 5 Hz = 40 s200=10/190 images @ 10 Hz = 20 s

...Rest...

Round

Time between relevant (targets)

10/190 guarantees 5% of target, but they may not be apart.

...Rest Rest... Rest... ...

5 rounds x 200 = 1,000 images @ 5 Hz -> 200 seconds 13

Methodology: Feature Vectors

Cut EEG activity related to visual event

From 200 ms to 1 second after the target presentation. 14

Methodology: Feature Vectors

1

2

3

32

... ...

Band-pass 0.1-20Hz

Downsample1000 → 250 Hz

Downsample250 → 20Hz

Channel concatenation512D vector

15

Methodology: Classification

Linear SVM

Q1

Q2

Q3

Testing:Training:

Leave-one-out Cross Validation 16

Outline

● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work

17

Object Detection: Early results

Images with high classifier scores for queries 2 and 3.18

Object Detection: Early results

Images with high classifier scores (yet not relevant) for query 1.19

Object Detection: User diversityReceiver Operating Characteric prove EEGs are valid, but with significant variations among users.

20

Object Detection: User diversity

21

Object Detection: 5 vs 10 Hz

22

Outline

● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work

23

EEG vs Mouse for Retrieval

EEG

RSVP

Mouse

Clicks on thumbnails

Timer

24

Retrieval within our dataset

Linear SVM

EEG: Sorting according to SVM confidence score

Mouse: Top: Clicked images (relevant)Bottom: Observed but not clicked imagesMiddle: Remaining images randomly sorted

25

Retrieval within our dataset

26

Retrieval in a larger dataset

Subset of TRECVid INS 2013 ( 23.614 images)

27

Retrieval in a larger dataset

28

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).Software: Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., ... & Darrell, T. (2014, November). Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia(pp. 675-678). ACM. [web]

Retrieval in a larger datasetEEG

Top 10: Positives

Bottom 100: Negatives

Clicked: Positives

Observed & unclicked:Negatives

Linear SVM...

Testing:

29

Retrieval in a larger dataset

30

Retrieval in a larger dataset

31

Outline

● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work

32

Conclusions

● Are EEG signals useful for Image Retrieval?

YES!

33

Conclusions

● Not all users are the same...

34

Conclusions

EEG Mouse

Comparable performance for Image Retrieval35

Future work

Can RSVP benefit from relevance feedback?

36

Outline

● Motivation● Related Work● Methodology● Object Detection● EEG vs Mouse for Retrieval● Conclusions● Future work

37

38