![Page 1: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/1.jpg)
OPTIMAL TRANSPORT APPLIED TO BCI
Nathalie GAYRAUD Maureen CLERC Alain RAKOTOMAMONJYPhD Seminars of Inria
![Page 2: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/2.jpg)
OPTIMAL TRANSPORT APPLIED TO BCI
1. BRAIN COMPUTER INTERFACES(THE P300 SPELLER)
2. OPTIMAL TRANSPORT3. APPLYING OT TO BCI4. RESULTS5. DISCUSSION
2
OUTLINE
![Page 3: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/3.jpg)
BRAIN COMPUTER INTERFACES
3
![Page 4: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/4.jpg)
Brain Computer Interfaces
4
EEG
Acquisition
Pre processingFeature extraction
Classification
Command
Feedback
![Page 5: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/5.jpg)
The P300 Speller
P300 Speller session➢ User looks at a keyboard on the
screen.- Letters are flashing.
➢ User counts the number of times “his” letter flashed.
➢ Each time “his” letter flashes, it elicits a response.
➢ Amplitudes are typically highest over parietal brain areas
5
![Page 6: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/6.jpg)
The P300 Speller
Feature extraction➢ Time windowing➢ Signal pre-processing
- Frequency filtering- Spatial Filtering / Component
analysis (Xdawn)[1]- Downsampling
Classification➢ Feature vectors
are classified into two classes:Target / Nontarget
6[1] Rivet, B., Souloumiac, A., Attina, V., & Gibert, G. xDAWN algorithm to enhance evoked potentials: application to brain–computer interface. IEEE Transactions on Biomedical Engineering, 2009, vol. 56, no 8, p. 2035-2043.
![Page 7: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/7.jpg)
Variability
Types
● Same-session● Cross-session● Cross-patient
7
Sources
● Equipment dddd ● Electrode position dddddd● Mental state dddddd ● Physiological differences dddd● Environmental dddddd
![Page 8: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/8.jpg)
Towards a zero-calibration BCI
8
![Page 9: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/9.jpg)
Towards a zero-calibration BCI
9
![Page 10: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/10.jpg)
Towards a zero-calibration BCI
10
![Page 11: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/11.jpg)
REGULARISED DISCRETE OPTIMAL TRANSPORT WITH CLASS LABELS
11
![Page 12: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/12.jpg)
Optimal Transport
Find a transportation that minimises a cost function.
12
“What is the optimal way to transport mass from domain A to domain B?”
![Page 13: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/13.jpg)
Optimal Transport
Find a transportation that minimises a cost function.
13
“What is the optimal way to transport mass from domain A to domain B?”
![Page 14: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/14.jpg)
Optimal Transport
Find a transportation that minimises a cost function.
14
“What is the optimal way to transport mass from domain A to domain B?”
![Page 15: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/15.jpg)
Regularised Optimal Transport
Entropic regularisation
Allows for a faster implementation using Sinkhorn algorithm [2]
15
“What is the optimal way to transport mass from domain A to domain B?”[2] Cuturi, Marco. "Sinkhorn distances: Lightspeed computation of optimal transport." Advances in Neural Information Processing Systems. 2013.
![Page 16: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/16.jpg)
Regularised Optimal Transport
Entropic regularisation
Allows for a faster implementation using Sinkhorn algorithm [2]
16
“What is the optimal way to transport mass from domain A to domain B?”[2] Cuturi, Marco. "Sinkhorn distances: Lightspeed computation of optimal transport." Advances in Neural Information Processing Systems. 2013.
![Page 17: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/17.jpg)
Regularised Optimal Transport
Entropic regularisation
Allows for a faster implementation using Sinkhorn algorithm [2]
17
“What is the optimal way to transport mass from domain A to domain B?”
![Page 18: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/18.jpg)
Regularised Optimal Transport
Entropic regularisation
Allows for a faster implementation using Sinkhorn algorithm [2]
18
“What is the optimal way to transport mass from domain A to domain B?”
![Page 19: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/19.jpg)
Regularised Optimal Transport with Class Labels
Regularise by class label when available for one of the two datasets [3]
19
“What is the optimal way to transport mass from domain A to domain B?”[3] Courty, Nicolas, Rémi Flamary, and Devis Tuia. "Domain adaptation with regularized optimal transport." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg, 2014.
![Page 20: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/20.jpg)
Regularised Optimal Transport with Class Labels
Regularise by class label when available for one of the two datasets [3]
20
“What is the optimal way to transport mass from domain A to domain B?”[3] Courty, Nicolas, Rémi Flamary, and Devis Tuia. "Domain adaptation with regularized optimal transport." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg, 2014.
![Page 21: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/21.jpg)
Regularised Optimal Transport with Class Labels
Regularise by class label when available for one of the two domains
21
“What is the optimal way to transport mass from domain A to domain B?”
![Page 22: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/22.jpg)
Regularised Optimal Transport with Class Labels
Regularise by class label when available for one of the two domains
22
“What is the optimal way to transport mass from domain A to domain B?”
![Page 23: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/23.jpg)
APPLYING CL-REGULARISED OT TO P300
23
![Page 24: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/24.jpg)
Applying CL-Regularised OT to P300 Speller Data
Transport a set of unlabeled feature vectors onto the domain of a set of labeled feature vectors.
24
Training
Input: Sets Xe, Ye
➢ Train classifier➢ Compute probability vector μe
Testing
Input: Set Xn
➢ Compute probability vector μn➢ Compute γ➢ Transport feature vectors: Xn
- Input to classifier
Output: Set Yn
![Page 25: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/25.jpg)
Experiments
Dataset➢ EEG signals recorded during
P300 speller sessions at the CHU of Nice.
➢ Adult patients suffering from Amyotrophic Lateral Sclerosis (ALS).
➢ 12 Subjects- 1 Session per subject
(calibration)➢ 12 electrodes
25
Pairwise Transfer Learning
Train with one session
Test with one session
132 experiments
![Page 26: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/26.jpg)
Example
26
Before Transportation
After Transportation
1st Xdawn Component 2d Projection
![Page 27: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/27.jpg)
RESULTS
27
![Page 28: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/28.jpg)
Results
28
Average performance (area under ROC curve) of an existing classifier over 11 experiments.
➢ Best performance before transport: 60% ➢ Best performances after transport: 62,7%
![Page 29: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/29.jpg)
CONCLUSIONS & FUTURE WORKS
29
![Page 30: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/30.jpg)
Conclusions & Future Works
➢ Optimal Transport application enhances the generalisation capacity of existing classifiers
➢ Computation is fast enough to allow online simulations
➢ Include more information in the existing set➢ Combination of more than one existing set➢ Use Optimal Transport theory to reduce dimensionality
30
![Page 31: OPTIMAL TRANSPORT APPLIED TO BCI · BRAIN COMPUTER INTERFACES (THE P300 SPELLER) 2. OPTIMAL TRANSPORT 3. APPLYING OT TO BCI 4. RESULTS 5. DISCUSSION 2 OUTLINE. BRAIN COMPUTER INTERFACES](https://reader033.vdocuments.site/reader033/viewer/2022060922/60ae24afeca27e30984a0095/html5/thumbnails/31.jpg)
Thank you !
31Python toolbox used for Optimal Transport Computation: http://pot.readthedocs.io/en/latest/