evaluation of a robot as embodied interface for brain computer interface systems
DESCRIPTION
Evaluation of a robot as embodied interface for Brain Computer Interface systems. Evaluation of a robot as embodied interface for Brain Computer Interface systems. L. Tonin, F. Piccione, S. Silvoni, K. Priftis, Emanuele Menegatti. K. Priftis Department of General Psychology - PowerPoint PPT PresentationTRANSCRIPT
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Evaluation of a robot as embodied Evaluation of a robot as embodied interface for Brain Computer interface for Brain Computer
Interface systemsInterface systems
L. Tonin, F. Piccione, S. Silvoni, K. Priftis, L. Tonin, F. Piccione, S. Silvoni, K. Priftis, Emanuele MenegattiEmanuele Menegatti
IRCCS San Camillo,Venice, Italy
Dept. Of General Psychology
University of Padua
Intelligent Autonomous Systems Lab University of Padua
2May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
What is BCI?What is BCI?
• There are four general steps in a BCI system:
1. Signal acquisition2. Classification and recognition3. Interaction with device4. Feedback to Subject
•According to Birbaumer et all.(1999): “Brain Computer interfaces are systems that allow to translate in real-time the electrical activity of the brain, in commands to control devices“
• Our target for the use of BCI system are patients severely paralyzed (locked-in) due to injury or disease
3May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Original P300-based BCI of IRCCS San CamilloOriginal P300-based BCI of IRCCS San Camillo
• This BCI system is based on P300 wave detection
• P300 is event-related potential
• Visual interface gives to patient stimulus to activate P300 signal
• The P300 detection will control the movement of cursor
• Directional cues are represented by the single flash of one out four target-direction arrows
• Participants focus their attention on the flashing arrow corresponding to the desired direction of motion
• Validated on 5 tetraplegic patients and 7 healthy patients [Piccione et al. Clinical Neurophysiology 2006]
4May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
P300-based BCI of IRCCS San CamilloP300-based BCI of IRCCS San Camillo
Focusing own attention, patient will generate P300 signal:
5May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Goals of our projectGoals of our project
In this project we aim at investigating two main issues:
• Giving to locked-in patients the control of a physical actuactor to improve their rehabilitation experience
• Developping telepresence of the patient with the body of the robot as medium
The questions are:
Can the robot lead patient to full and better control of BCI via a higher patient engagement in the task?
Might patients identify them-selves into the robot and think of it as an extension of their perceptions and actions (i.e. their body)?
6May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Robot as a physical actuatorRobot as a physical actuator
• To be able to move a body again, might give new strength and will to the locked-in patients
1. Custom built holonomous robot with hexagonal structure and three omnidirectional wheels
2. The holonomous nature of the robot is very important to replicate the motion and the apparence of the interface used by our BCI
The robot is fitted with:• Omnidirectional camera• Audio board• WiFi connectivity
7May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
BCI and robot connectionBCI and robot connection
BCI system SERVER
RobotClient
Command
Action result
• Interaction between robot and BCI is based on client-server system with TCP-IP protocol via IEEE 802.11
Security about connection between BCI and robot is guaranteed by:
• System protocol and network protocol check data received;
• Virtual Private Network between BCI and robot with autentication and encryption
8May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Test and DemoTest and Demo
We carried out many tests to integrate BCI and robot. These tests accomplished at IRCCS San Camillo in Venice, and in IAS Laboratory ofUniversity of Padua:
• Robot and BCI computer connected with wired network
• Testing healthy subject performances to move robot
•BCI and robot connected in larger network with VPN connection(robot was in Padova, and BCI system in Venice)
9May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Test and DemoTest and Demo
During the scientific festival Discovery on Film in Rovereto (May 2008).
A healthy subject at San Camillo in Venice was able to move the robot using the BCI at museum of Rovereto (about 200 Km from Venice).
The test was successful and subject was able to reach the target.
You can see it at: www.sperimentArea.tv the web TV of
the Museo Civico of Rovereto
10
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Robot as actuator and generator of interfaceRobot as actuator and generator of interface
Robot is able to send back to patient images and sounds about environment.
The images token by omnidirectional camera can be integrated in BCI interfaceto allow patient to see through robot’s camera.
A new integrated interface for BCI is under clinical validitation by prof. Priftis of Psychology Department of University of Padua
11
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Robot as actuator and generator of interfaceRobot as actuator and generator of interface
Figure 6: classification accuracy (%) of the 5 healthy subjects who performed Experiment 1, and classification accuracy (%) of one subject who performed Experiment 2.
12
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Robot sensesRobot senses
13
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Robot as a Telepresence mediumRobot as a Telepresence medium
• The patient is able to see and listen with the senses of the robot and so he is able both to extend own social life and communicate own needs
• Telepresence for museum visits: more autonomous navigation,obstacle avoidance, new graphical user interface
• Telepresence for household rehabilitation: consumer robot Rovio holonomous robot with frontal camera for remote perception.
14
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
Robot as a Telepresence for museum visitsRobot as a Telepresence for museum visits
TCP/IP
The BCI Server
Internet
Patient in Venice, robot in Agrigento, Sicily, 1467 km
The Client robots developed The Client robots developed at University of Palermo, at University of Palermo, Sicily, ITALYSicily, ITALY
Pioneer3-AT (Outdoor Robot)
PeopleBot (Indoor Robot)
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
STEP 1: Choose of the real robot
The Graphical Interface
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
STEP 2:The Session Start and Sensors and Actuators are Activated
The Graphical Interface
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
STEP 2.1:
The Applet for the visualization of the navigation environment through the robot camera
The Graphical Interface
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
STEP 3: Robot Movements Activation through the
use of the 4 directional arrows
The Graphical Interface
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
The Graphical Interface
ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009
21
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
• Next step: fully autonomous robot, robot has a map of the environment, robot presents four topological choices (e.g. the four closest masterpieces), subject selects destinations
Robot as a Telepresence for museum visitsRobot as a Telepresence for museum visits
22
May 12, 2009 BCI Workshop ICRA 2009, Kobe
Evaluation of a robot as embodied interface Evaluation of a robot as embodied interface for Brain Computer Interface systemsfor Brain Computer Interface systems
• Next step: fully autonomous robot, robot has a map of the environment, robot presents four topological choices (e.g. the four closest masterpieces), subject selects destinations
Robot as a Telepresence for museum visitsRobot as a Telepresence for museum visits
The Experimental Results: Simulation
The BCI-Robot moving in the DINFO Department.Time T1: Command Go Forward
BCI- Web_Interface Snapshot of Robot Position
BCI- Web_Interface
Snapshot of Robot Position
Time T10: Command Stop
Many Thanks to You and to: Dept. of Information Engineering, University of Palermo, Palermo,
ItalyRosario Sorbello: [email protected] Chella: [email protected] Anzalone: [email protected] Cinquegrani: [email protected]
Dept. of Information Engineering, University of Padova, Padova, Italy
Emanuele Menegatti: [email protected]
Dept. of Neuro-rehabilitation, S.Camillo Hospital, Venice, ItalyLaboratory: [email protected]
Dept. of General Psychology, University of Padova, Padova, ItalyKonstantinos Prifitis: [email protected]