Scientific Progress Report Labex SMART

December 2014

Laboratory of Excellence SMART (ANR-11-LABX-65) is supported by French State funds managed by the ANR within the Investissements

d'Avenir Programme under reference ANR-11-IDEX-0004-02

2

3

Table of contents SMART Overview ......................................................................................................................... 5

SMART Projects .......................................................................................................................... 11

EDHHI................................................................................................................................................. 14

ISMES ................................................................................................................................................. 19

ONBUL ............................................................................................................................................... 27

SeNSE ................................................................................................................................................. 35

SMART-BAN ....................................................................................................................................... 49

SpinalCOM ......................................................................................................................................... 59

SMART Actions ........................................................................................................................... 65

SMART Perspectives ................................................................................................................... 85

SMART Publications ................................................................................................................... 89

4

5

SMART Overview

6

7

SMART Context In 2010 the French Government launched a program called “Investments for the Future” with the main

objective of supporting higher education, research and transfer to industry

http://www.enseignementsup-recherche.gouv.fr/pid24578/investissements-d-avenir.html (in French)

Within this program, there were specifically competitive calls in 2010 and 2011 for different

categories of projects among which projects called “Equipment of Excellence (Equipex), Laboratories

of Excellence (Labex), and Initiatives of Excellence (Idex). The latter aimed at grouping French

universities within larger structures called Idex. The Labex are local or national consortia of labs that

address a large, long term (8 to 10 years), scientific program. The Equipex program finances new

scientific equipment.

SMART, was submitted to the second call in 2011 and selected by the international committee in

February 2012. The main topic of the project is research on human-machine interactions. The

consortium is composed of eight laboratories affiliated to seven legal institutions: University Pierre

and Marie Curie, CNRS, INSERM, Institut Mines-Telecom, University Paris 8, IRCAM, EPHE.

SMART is within the Idex Sorbonne Universités (SU).

SMART Vision Be it in our homes or workplaces, in the streets of our cities where we stride or the public spaces

where we go for business, service, shopping, leisure, or travel, we are already surrounded with digital

systems, more or less complex computing and communicating devices and artifacts, with which we

interact. Access to this digital world opens enormous possibilities for new services and easier living.

SMART aims to design technologies that would make the interaction of humans with those devices

simpler, more efficient and more adaptive. This requires to include in those systems capacities to

better understand how humans act and interact, and hence to develop models of humans representing

their physical capabilities, psychological trends and behaviors. It also requires studying efficient and

natural interfaces and enhanced tools for a better interactivity between digital artifacts and humans.

The wide distribution in our environment of communicating and interacting devices, integrating

individually and collectively perception, computation, actuation and communication capacities at

different scales, and which may be mobile, creates an ambient intelligence also requires addressing

them both as integrated units and as a global networked system and conceiving infrastructures for their

connectivity. Those devices produce also massive amounts of data that need to be processed efficiently

to extract the relevant information and knowledge, and also stored and protected.

SMART Ambition The SMART Labex objective is to contribute to the foundational bases for facilitating the inclusion of

intelligent digital artifacts in our daily life for service and assistance. The project addresses underlying

scientific questions raised by the development of Human-centered digital systems and artifacts in a

comprehensive way.

An efficient and natural interaction with artifacts requires understanding human actions and behavior,

as well as the design of natural and friendly interfaces. When those artifacts and digital devices are

disseminated, networked, and sometimes mobile, it is also necessary to provide for their connectivity

and for knowledge extraction, sometimes from massive data, knowledge sharing and access.

Project actions unfold along five dimensions: a) basic research and novel concepts; b) methods,

technologies and tools for the design, operation, interfacing and networking of systems and artifacts

interacting with humans; c) exploration of novel applications and usage; d) education curricula, and e)

dissemination and exploitation of results.

As an illustrative main usage area, source of open topics and case studies for this project, the new

services induced by the digital society for e-health, including the ageing society and autonomous

living.

8

The research program is organized along five axes that address the main scientific questions and the

use case:

1. “Modeling of humans”: understanding and modeling of physiological and neurophysiological

functions, integrated representations of the musculoskeletal system, of the basic motor and

perceptive systems and of the learning and adaptation mechanisms, integrative architectures.

2. “Interfaces and Interaction with humans”: design of new devices that enhance the range of

interactions between human and machines (e.g., haptic devices), new interaction modalities

and signals including cognitive and emotional aspects.

3. “Humans at the convergence of digital and real environments”: large scale complex data

processing, knowledge emergence, digital and human decision-making and socially intelligent

computing.

4. “Autonomic Distributed Environments for Mobility”: networking, virtualization, self-

organized, self-healing and secure architectures of heterogeneous, autonomous and

cooperative mobile entities.

5. “Human autonomy and e-health”: innovative medical devices, from assisting robots to

implanted sensors and bio-mechatronic embedded systems, and personalized care in the

context of e-health for autonomous living

SMART Teams SMART is a specific blend of research teams of eight laboratories in applied mathematics, computer

science, robotics, neuroscience, medical imagery, networks and distributed systems, Human-Machine

interaction, electrical engineering in the same campus, with a clear and consistent research program

and an education and training program, experimenting new usages in living labs, and having close ties

with industry.

- The Institute of Intelligent Systems and Robotics - ISIR (UPMC, CNRS): Autonomous and

interactive Robotics systems; Mobility; cognitive systems; Robotics and Neuroscience; assistance

to surgical and functional rehabilitation; micromanipulation; Manipulation; Haptics.

- Paris 6 Computer Science Lab - LIP6 (UPMC, CNRS): Decision-making, Intelligent Systems and

Operations Research, Databases and Machine-Learning, Networks and Distributed Systems,

Systems-on-chip.

- Human and Artificial Cognition Laboratory - Chart-Lutin (UPMC, University Paris 8, EPHE):

pragmatic and semantic interactions of human and artificial systems.

- Electronics and Electromagnetism Laboratory - L2E (UPMC): micro and nano-electronics,

communication, physiological parameters monitoring.

- Information processing and Communication Electronics Laboratory – LTCI (Institut

MinesTelecom, CNRS): Signal processing and image, pattern recognition, 3D object modeling,

conversational agents, multimedia (speech, audio, music, images, video), document analysis,

multimodal biometrics.

- Laboratory Jacques-Louis Lions - LJLL (UPMC, CNRS): Mathematical modeling of physical

phenomena in physics, mechanics, biology, medicine, chemistry, information processing,

Economics, finance;

- Laboratory of Biomedical Imagery - LIB (UPMC, CNRS, INSERM): Medical imaging, modeling,

image and signal processing, magnetic resonance imaging, microscopy, optical imaging, nuclear

medicine imaging, ultrasound, Alzheimer's, cardiovascular disease, medical oncology,

neurosciences.

- Laboratory of the Technology of Music and Sound – STMS (UPMC, CNRS, IRCAM):

Instrumental Acoustics, Acoustic and Cognitive Spaces and Sound, Perception and Design

Analysis and sound synthesis, Analysis of musical practices, Real-Time Musical Interactions.

9

SMART Strategy

Financing transversal projects to investigate the main scientific topics of the project in a

multidisciplinary approach

Financing post-doctoral grants, to attract young scientists with high potential to preform

advanced research on novel topics and to explore new venues.

Doctoral program for financing PhD grants for excellent post-graduate students to perform

novel cross-disciplinary research on the project’s topics in participating laboratories.

Financing internships for master students.

Financing invited visiting professors to bring very high-level senior professionals to partner

laboratories for participating and advising in research projects and educating local students.

Involvement in educational curricula in relation with the research program for

dissemination of results to the younger generation.

Industrial partnership for exploitation of results.

International partnership and cooperation.

SMART Figures

Duration: 94 months (March 2012-Dec. 2019). Total budget: 5 M€.

Personnel directly involved in SMART: 88 faculty and researchers, 4 visiting professors, 19

PhDs, 4 Post-docs, 20 Master Interns.

6 projects were launched in September 2013 for durations ranging between 12 and 48 months.

SMART Actions (2012-2014) 3 calls for internships

2 calls for PhDs

1 call for Post-docs

1 call for visiting professors

1 call for projects

Invited colloquia: Rodney Brooks, Claude Berrou

Regular seminars

Organization of workshops

SMART Governance SMART is coordinated by a Director (Raja Chatila), a Deputy Director (Mohamed Chetouani) and a

Project Manager (Zoitsa Siaplaoura). Three main bodies are involved in managing and overseeing the

Labex: the Executive Committee (EXCOM), the Steering Committee (STEERCOM), and the

Scientific Advisory Board (SAB).

The EXCOM is the operational body of the project. It is composed of the director and deputy director,

a representative of each of the five programs and the person in charge of the Education curricula.

Members:

Habib Benali (LIB), Mohamed Chetouani (ISIR), Patrick Gallinari (LIP6), Patrick Garda (LIP6),

Benoît Girard (ISIR), Christophe Marsala (LIP6), Catherine Pelachaud (LTCI), Franck Petit (LIP6),

Agnès Roby-Brami (ISIR), Pierre Sens (LIP6), Jean-Luc Zarader

The STEERCOM oversees the general operation of the Labex. It is chaired by a representative of the

main partner institution, the Idex Sorbonne Universités, and composed of:

One representative from each of the other seven legal institutions partners of the Labex

The directors of the eight member laboratories or their representatives

A representative of the Doctoral Training Institute

The SMART Labex director

10

Institutions:

Sorbonne Universités: Véronique Atger

CNRS: Wilfrid Perruquetti

EPHE: François Jouen

Institut Mines Télécom: Yves Grenier

INSERM: Franck Lethimonnier

IRCAM: Hugues Vinet

UPMC: Paul Indelicato

Université Paris 8: Mario Barra

Doctoral Training Institute: Jean-Dominique Polack

Laboratories:

ISIR: Agnès Roby-Brami

L2E: Aziz Benlarbi-Delaï

Laboratoire CHART-LUTIN: Charles Tijus

LIB: Pascal Laugier

LIP6: Jean-Claude Bajard

LJLL: Benoît Perthame

LTCI: Olivier Cappé

STMS: Gérard Assayag

The SAB is composed of members external to the Labex partners, belonging to French and foreign

higher education and research institutions and industry in the main scientific domains of SMART:

Etienne Burdet, Imperial College, London, UK

Justine Cassell, Carnegie Mellon University, Pittsburgh, USA

Peter Ford Dominey, INSERM, Lyon, FRANCE

Rachid Guerraoui, EPFL, Lausanne, CH

Philippe Roy, CAP DIGITAL, Paris, FRANCE

Stuart Russell, University of California, Berkeley, USA

Alessandro Vinciarelli, Glasgow University, UK

11

SMART Projects

12

13

PROGRESS REPORT

EDHHI

Date : 07/03/2014

Partners ISIR

Serena Ivaldi (postdoc)

Mohamed Chetouani (professeur) Salvatore Anzalone (postdoc)

Anna-Lise Jouen (postdoc)

Charles Ballarini (stagiaire) *

CHART-LUTIN

Ilaria Gaudiello (doctorante) Sebastien Lefort (stagiaire) *

Elisabetta Zibetti (maitre de conference)

Joelle Provasi (maitre de conference)

Stages financés par EDHHI

· Sebastien Lefort, né le 13/12/1981

o Responsable: Elisabetta Zibetti, CHART-LUTIN

o Ecole: Ecole Pratique des Hautes Etudes (EPHE) - 4-14 rue Ferrus 75014 Paris

· Charles Ballarini , né le 01/04/1988

o responsable: Mohamed Chetouani, UPMC

o Ecole : EPITA

Liste de publications

International Journals

Ivaldi, S.; Anzalone, S.M.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2014) Robot initiative in a team learning task

increases the rhythm of interaction but not the perceived engagement.

Frontiers in Neurorobotics. Vol 8, No 5, DOI 10.3389/fnbot.2014.00005.

Short papers in International W orkshops

Ivaldi, S.; Anzalone, S.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2013). Cues for making a humanoid child more

human-like during social learning tasks. Workshop on Towards social humanoid robots: what makes

interaction human-like? - IROS 2013.

Rousseau, W.; Anzalone, S.; Chetouani, M.; Sigaud, O.; Ivaldi, S. (2013). Learning object names through shared

attention. Workshop on Developmental Social Robotics - IROS 2013.

Ivaldi, S, ; Anzalone, S.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2014). Robot initiative increases the rhythm

of interaction in a team learning task. Workshop Timing in Human-Robot interaction, in HRI 2014, Bielefeld,

Germany.

14

EDHHI: Engagement During Human-Humanoid Interaction Responsible of the project:

Elisabetta ZIBETTI & Joëlle PROVASI [CHART- LUTIN 4004]

Mohamed CHETOUANI [ISIR UMR7222]

Partners:

Institut des Systèmes Intelligents et de Robotique (ISIR) UPMC/ CNRS UMR 7222: M. Chetouani

(Prof.), S. Ivaldi (Post-Doc), S. Anzalone (Post-Doc)

Laboratoire Cognitions Humaine et Artificielle et plate-forme LUTIN (CHART-LUTIN) UP8/EPHE:

I. Gaudiello (Post-Doc), J. Provasi (Mcf), E. Zibetti (Mcf)

Web site: http://www.smart-labex.fr/index.php?perma=EDHHI

1 The Project at a glance

Context and Objectives

EDHHI, a one-year project, advances the current understanding about the factors influencing

effective human-humanoid physical interaction in cooperative tasks.

First we aim at identifying the factors influencing the human engagement towards the robot

and the task for determining the metrics destined to automatically assess the engagement and the

acceptability of the human subjects. Engagement (Sidner et al, 2005) in collaborative interactions is

characterized by a dynamic and continuous exchange of verbal and non-verbal signals - gestural and

postural – carrying out information and communication content (Delaherche et al., 2012). Dynamics

can be modulated by inter-individual or social factors such as degree of extraversion or the a-priori

attitude toward robots. Therefore, we investigate them and take them into account as potential bias of

the engagement metrics during human-humanoid interaction.

Our second aim is to identify the factors influencing the robot functional and social

acceptability, which could be used to enhance interactions behaviors. By functional acceptability we

mean the willingness to use technology for the tasks for which the robot has been created (Salvini et

al., 2010) and by social acceptability we mean the facility to share statements, space and routines with

a non-human agent (Pesty & Duhaut, 2011).

To investigate these questions, in EDHHI, we design an experimental protocol involving

physical and verbal interaction between human participants and the iCub robot during several

cooperative tasks such as handling, assembly, and decision-making. We focus on the interplay

between cognitive and personality differences and behavioral features (speech, motion, gaze, posture)

that can have an impact on the effectiveness of the interaction. Methodologically, we combine the

processing of conventional signals (utterance, posture, contacts) and explicit measures such as

responses to personality tests and post-experimental questionnaires.

References

Delaherche, E., Chetouani, M., Mahdhaoui, A. and Saint-Georges, C. and Viaux, S. and Cohen, D. (2012).

Interpersonal Synchrony : A Survey Of Evaluation Methods Across Disciplines. IEEE Transactions on

Affective Computing. Vol 3 No 3 Pages 349 - 365.

Pesty, S., Duhaut, D. (2011). Acceptability In Interaction: From robots to Embodied Conversational Agents.

Computer graphics theory and applications.

Salvini, P., Laschi, C., & Dario, P. (2010). Design for acceptability: improving robots’ coexistence in human

15

society. International Journal of Social Robotics , 2 (4), 451-460.

Sidner, C. L., Lee, C., Kidd, C. D., Lesh, N., & Rich, C. (2005). Explorations in engagement for humans and

robots. Artificial Intelligence, 166(1), 140-164.

2 Scientific progress and results

Work carried out and outcomes achieved over the period

Between September and December 2013:

We selected few dependent variables to evaluate: i) human engagement towards the collaborative

task: e.g. distances; gaze and speech duration... ii) human acceptability towards the robot: e.g.

response (compensative or reciprocal)

We finalized the experimental design of the tasks and we set up the experimental protocol and

material

We submitted experimental protocol to the ethical committee of CERES for Approbation of the

EDHHI protocol by the Ethical Committee Conseil en Ethique pour les Recherches en Santé

(CERES), Université Paris Descartes.

We developed software tools for implementing different experimental conditions of the

cooperative interaction between humans and iCub in order for the latter to be able to execute the

selected task in an effective and collaborative way.

Between January and April 2014:

We performed pre-experimental tests on dyadic interaction between ten humans adult while

executing a simple cooperative task in order to tune the interaction task scenario.

We received the approbation of the EDHHI protocol by the Ethical Committee

We started an intense experimental phase after recruiting 60 adults participants

Between April and August 2014:

We coded, analyzed (video, audio, interview, questionnaire) and draw conclusion on the

influence of selected experimental variations on the engagement of the human towards both the

robot and the task.

Defense of the Master (M2) dissertation “Evaluation de l’engagement lors d’Interactions Homme-

Robot: Effets de l’extraversion et de l’attitude vis-à-vis des robots sur l’émission de signaux

sociaux " by Sebastien Lefort.

During this period we devoted part of our time in disseminating the EDDHI projects and presenting its

first advances (invited seminars) and in preparing publications.

Figure 2 Interaction between a human participant and the humanoid iCub for a cooperative task

Figure 1 the robot, the experimenter and a voluntary participant during the experiment

16

Results

We assessed the influence of extroversion and negative attitude toward robots on speech and gaze

during a cooperative task, where a human must physically manipulate a robot to assemble an object.

The experiments were carried out with the humanoid iCub and N=56 adult participants.

We found that extrovert people tend to talk more and longer to the robot, whereas they do not look at

the robot more than introverts. On the contrary, we found that people with negative attitude towards

robots tend to look less at the robot than people with a positive attitude.

Correlation between the participants’ extraversion score (computed by the NEO-PI-R) and their gaze, utterance

frequency and duration during the assembly task

Our results suggest that the engagement models classically used in human-robot interaction should

take into account personality traits.

Correlation between the participants’ negative attitude towards robots score (computed by the NARS) and their

gaze, utterance frequency and duration during the assembly task

3 Recruitment

From the 1st October to the 30th of May 2014 four internships students have joint the EDDHI team on

the following topics.

[1] Evaluation de l’engagement lors d’Interactions Homme-Robot: Effets de l’extraversion et de

l’attitude vis-à-vis des robots sur l’émission de signaux sociaux

[2] Interfaces for experiments of human-humanoid interaction

[3] Analysis of behaviors in human-robot interaction experiments

[4] Acceptability in HRI: implicit and explicit methods

Only two of them (1 and 2) have been hired on the EDHHI budget.

Topic Student name Year Dates Supervisor Laboratory Funded by

Evaluation de l’engagement

lors d’Interactions Homme-

Robot

Sebastien Lefort M2- EPHE From October 2014

(8 months)

Elisabetta

Zibetti CHART EDDHI

Interfaces for experiments of

human-humanoid

interaction

Charles Ballarini M2- EPITA From October 2014

(8 months) Serena Ivaldi ISIR EDDHI

Analysis of behaviors in

human-robot interaction

experiments

Anais Jeannel de

Thiersant L3 -Psycho Pratt

From March 2014

(2 months)

Joëlle Provasi

and Serena

Ivaldi

CHART-

ISIR CHART

Acceptability in HRI:

implicit and explicit

methods

Ilaria Gaudiello 3rd PhD Thesis From Oct 2013

(6 months)

Elisabetta

Zibetti CHART CHART

F igur e 2: D em onst r at ion of t he assembly t ask : 1) t he par t icipant asks t he r obot t o gr asp t he t wo cy l inder s; 2)

t he par t icipant gr abs t he r obot ar m s and dem onst r at es how t o m ove t hem t o al ign t he t wo cy l inder s; 3) t hepar t icipant fi xes t he cy l inder s w i t h som e t ape whi le t he r obot is holding t hem ; 4) t he par t icipant r et r ieves

t he assembled ob ject fr om t he r obot .

Variable Ex t r over sion scor e

Gaze frequency (num./ s) r 2= -0,13 ; p= N.S.

Gaze durat ion (s) r 2= 0,098 ; p= N.S.

U t t er ance fr equency (num./ s) r 2= 0,318 ; p< 0,05

U t t er ance dur at ion (s) r 2= 0,321 ; p< 0,05

Table 1: Cor r elat ion bet ween t he par t icipant s’ ex-t r over sion scor e (com put ed by N EO-P I -R [4]) and

t hei r gaze and ut t er ance fr equency and dur at iondur ing t he assembly t ask .

frequency and durat ion of ut terances (see Table 1). This

can also be seen in the scat ter graphs in Figure 3.To summarize, the more an individual is ext rovert , the

more he/ she will talk to the robot during an assembly taskto provide inst ruct ions. On the cont rary, as ext roversion

does not have influence on the gaze frequency or durat ion,int roverts will not look at the robot ’s face less than ext ro-

verts.Therefore, with reference to the research hypothesis ex-

pressed in Sect ion 2, we accept Hypothesis 1, and rejectHypothesis 2.

4.2 Effect of negativeattitude towards robotsThepart icipants’ averageNARSscorewas45,55 (σ= 12,74),

which is a neut ral value for the at t itude towards robots. Ta-ble 2 reports the Pearson’s correlat ion between the NARS

score of the part icipants and their gaze and ut terance fre-quency and durat ion. The results indicate that the nega-

t ive at t itude does not influence the verbal signal, as there

Fu

ncti

on

al

acce

pta

bil

ity

0

0,225

0,45

0,675

0,9

Robot ics expert ise

0 0,25 0,5 0,75 1

So

cia

l a

cce

pta

bil

ity

0

0,25

0,5

0,75

1

Robot ics expert ise

0 0,25 0,5 0,75 1

0

0,225

0,45

0,675

0,9

N egat ive at t itude towards robot ics

score

0 20 40 60 80

So

cia

l a

cce

pta

bil

ity

0

0,25

0,5

0,75

1

N egat ive at t itude towards robot ics

score

0 20 40 60 80

Fre

que

ncy o

f u

tte

rances

0,05

0,167

0,283

0,4

Extroversion score

55 77 99 121 143 165

y = 0,0009x + 0,1336

Du

ratio

n o

f utt

era

nces

0,05

0,175

0,3

0,425

0,55

Extroversion score

55 77 99 121 143 165

y = 0,0014x + 0,1183

F igur e 3: Scat t er gr aphs show ing t he fr equency(number / seconds) and dur at ion (seconds) of ut t er -

ances of t he par t icipant s (N = 56) , in funct ion of t hei rex t r over sion scor e.

F igur e 2: D em onst r at ion of t he assembly t ask : 1) t he par t icipant asks t he r obot t o gr asp t he t wo cy l inder s; 2)

t he par t icipant gr abs t he r obot ar m s and dem onst r at es how t o m ove t hem t o al ign t he t wo cy l inder s; 3) t hepar t icipant fi xes t he cy l inder s w i t h som e t ape whi le t he r obot is holding t hem ; 4) t he par t icipant r et r ieves

t he assembled ob ject fr om t he r obot .

Variable Ex t r over sion scor e

Gaze frequency (num./ s) r 2= -0,13 ; p= N.S.

Gaze durat ion (s) r 2= 0,098 ; p= N.S.

U t t er ance fr equency (num./ s) r 2= 0,318 ; p< 0,05

U t t er ance dur at ion (s) r 2= 0,321 ; p< 0,05

Table 1: Cor r elat ion bet ween t he par t icipant s’ ex-t r over sion scor e (com put ed by N EO-P I -R [4]) and

t hei r gaze and ut t er ance fr equency and dur at iondur ing t he assembly t ask .

frequency and durat ion of ut terances (see Table 1). This

can also be seen in the scat ter graphs in Figure 3.To summarize, the more an individual is ext rovert , the

more he/ she will talk to the robot during an assembly taskto provide inst ruct ions. On the cont rary, as ext roversion

does not have influence on the gaze frequency or durat ion,int roverts will not look at the robot ’s face less than ext ro-

verts.Therefore, with reference to the research hypothesis ex-

pressed in Sect ion 2, we accept Hypothesis 1, and rejectHypothesis 2.

4.2 Effect of negativeattitude towards robotsThepart icipants’ averageNARSscorewas45,55 (σ= 12,74),

which is a neut ral value for the at t itude towards robots. Ta-ble 2 reports the Pearson’s correlat ion between the NARS

score of the part icipants and their gaze and ut terance fre-quency and durat ion. The results indicate that the nega-

t ive at t itude does not influence the verbal signal, as there

Fu

ncti

on

al

acce

pta

bil

ity

0

0,225

0,45

0,675

0,9

Robot ics expert ise

0 0,25 0,5 0,75 1

So

cia

l a

cce

pta

bil

ity

0

0,25

0,5

0,75

1

Robot ics expert ise

0 0,25 0,5 0,75 1

0

0,225

0,45

0,675

0,9

N egat ive at t itude towards robot ics

score

0 20 40 60 80

So

cia

l a

cce

pta

bil

ity

0

0,25

0,5

0,75

1

N egat ive at t itude towards robot ics

score

0 20 40 60 80

Fre

que

ncy o

f u

tte

rances

0,05

0,167

0,283

0,4

Extroversion score

55 77 99 121 143 165

y = 0,0009x + 0,1336

Du

ratio

n o

f utt

era

nces

0,05

0,175

0,3

0,425

0,55

Extroversion score

55 77 99 121 143 165

y = 0,0014x + 0,1183

F igur e 3: Scat t er gr aphs show ing t he fr equency(number / seconds) and dur at ion (seconds) of ut t er -

ances of t he par t icipant s (N = 56) , in funct ion of t hei rex t r over sion scor e.

17

4 Publications

Journals

Ivaldi, S.; Anzalone, S.M.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2014) Robot initiative

in a team learning task increases the rhythm of interaction but not the perceived engagement.

Frontiers in Neurorobotics. Vol 8, No 5, DOI 10.3389/fnbot.2014.00005

Anzalone, S.M.; Boucenna, S.; Ivaldi, S.; Chetouani, M. (2015) Evaluating the quality of the

interaction with social robots. The International Journal of Social Robotics (under revision)

Short papers in International Workshops

Ivaldi, S.; Anzalone, S.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2013). Cues for making a

humanoid child more human-like during social learning tasks. Workshop on Towards social

humanoid robots: what makes interaction human-like? - IROS 2013.

Rousseau, W.; Anzalone, S.; Chetouani, M.; Sigaud, O.; Ivaldi, S. (2013). Learning object

names through shared attention. Workshop on Developmental Social Robotics - IROS

2013.

Ivaldi, S, ; Anzalone, S.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2014). Robot initiative

increases the rhythm of interaction in a team learning task. Workshop Timing in Human-

Robot interaction, in HRI 2014, Bielefeld, Germany.

5 Events

Invited seminars

S. Ivaldi. “iCub learning from humans through physical interaction”, invited talk in ICRA

2014, Hong Kong, June 2014.

S. Ivaldi. “Human-robot interaction with iCub”, invited talk at IAS13, Padova, July 2014.

E. Zibetti. (2014): Robotics in Social Cognitive Sciences. A powerful “mindtool” for studying

and improving Human Behavior. Présentation dans le cadre du "German-French Worskhop

Robotics and Social Sciences": Les ateliers INNOROBO. 20 March 2014. Cite internationale -

Centre de Congres Lyon, France.

M. Chetouani. (2014): Impaired social interaction and robotics. French-German Worskhop

Robotics and Social Sciences" : Les ateliers INNOROBO. 20 March 2014. Cite internationale

- Centre de Congres Lyon, France.

S. Ivaldi. « Robot learning through human interaction » seminar Telecom-ParisTech. Janvier

2014.

S. Ivaldi. « Advancements of project EDHHI » seminar SENSE project (SMART). December

2013.

18

19

ISMES

20

ISMES Interfaces SensoriMotrices Embarquées pour la rééducation et la

Suppléance

Embedded Sensorimotor Interfaces for rehabilitation and assistance

Responsible of the project:

Agnès Roby-Brami [ISIR]

Partners:

ISIR: W. Bachta, N. Jarrassé, A. Roby-Brami

STMS: F. Bevilacqua

LIB: V. Marchand-Pauvert, P.F. Pradat, R. Katz, A. Lackmy-Vallée

Salpétrière Hospital: Physical Medicine and Rehabilitation (P. Pradat-Diehl), neurology (PF. Pradat).

Web site: http://ismes.isir.upmc.fr/

1 The Project at a glance

Context and Objectives

The aim of the project is to study the benefit of techniques associating sensors and effectors-

stimulators that we call “sensori-motor interfaces”. Those embedded interfaces will allow online

measurements of motor activity and augmented sensory feedback based on a physiological analysis of

human action. Enriched sensory feedback allows to compensate the impairments of sensory loops and

to reinforce the learning of new compensatory actions. The project addresses two scientific challenges:

the first is to establish the necessary models to represent the motor actions in a parsimonious way from

the sensors. The second is to find the efficient encoding of motor behavior to provide pertinent

multisensory signals, easily interpretable by the central nervous system.

The clinical objective is to improve the autonomy of disabled persons thanks to sensori-motor

learning, rehabilitation and assistive technology. For that purpose, our approach is to analyze and

rehabilitate the human activities in an enriched context closer to the daily life activity and to develop

assistive technology as a function of patients' needs. The project is thus closely related to clinical

neuro-rehabilitation.

The multidisciplinary central task of the project is to

develop sensorimotor interfaces for a better analysis of

human motor actions in healthy subjects and neurological

patients. We develop specific interactive devices using

multisensory signals (light touch, vibration, sound) in three

contexts: for a better command of an upper-limb prosthesis

in amputees; for the rehabilitation of gait and posture in

neurological patients (light touch); for the rehabilitation of

arm coordination in stroke patients (coupling gesture-sound).

21

2 Scientific progress and results

Work carried out and outcomes achieved over the period

A significant part of the SMART budget is devoted to equipment. We had to acquire the

equipement and to install experimental set-ups, with a special effort to homogeneise protocols between

ISMES laboratories.

A complete set-up for the analysis of hand and finger posture is now operational at ISIR. This

set-up uses data fusion from different sensors (IMU, 6DoF electromagnetic device, dataglove,

force sensors) totally or partially embedded in instrumented objects with an interactive table

(coll. E. Burdet, Imperial College).

We have acquired an up-to-date hand and elbow prosthesis that will be the basis of the

experimental platform for the command of the prosthesis (engineer to hire).

We have acquired a force plateform and duplicated the set up for the analysis of the effect of

light touch on the equilibrium. A similar set-up used at ISIR is now installed in clinical setting

at Salpétrière. (Coll A. Wing and R. Reynolds, Birmingham)

We have developed and duplicated a braked elbow orthosis to induce controlable arm

dyscoordination for experimental purpose. This orthesis is fitted with interactive interfaces for

the coupling of movement and sound (Musical objects –MO- developed by STMS), visual and

force feedback (ISIR).

We have shared the experience for the extraction and fusion of data and signal processing from

embedded sensors (coll. Popovic, Pavle Savic project).

The internal and external meeting at the beginning of the project SMART were an opportunity to

build or reinforce multidisciplinar collaboration between laboratories (e.g F. Bevilacqua gave a

seminar in LIB).

Manual dexterity and Prosthesis

The analysis of finger coordination for grasping objects

in a bimanual task showed that synergies could be

summarized by 4 Principal Components, with a

particularly good functional correlation with the hand

anatomy (Jarrassé et al. 2014). Preliminary experiments

with instrumented objects in healthy subjects and some

stroke patients (Jarrassé et al, 2013, Martin et al. in

preparation) allowed to define quantitatively prehension

strategies and to advance in the analysis and

interpretation of IMU signals.

A series of experiments have been performed on the

cortical control of phantom movements made by amputees (coll J. Graaf, ISM). Preliminary

experiments with healthy subjects wearing a fake prosthesis were performed in order to test the effect

of sound feedback on the timing of prehension gestures.

An article has been submitted to question a socio-anthropological approach of prosthesis and corporeal

integration of techniques.

22

Light touch and equilibrium

The effect of light touch on posture has been analysed in healthy

subjects. We have demonstrated that is is possible to drive the

center of pressure in closed loop thanks to a kinesthesic feedback

(Verité et al. 2013, 2014). The use of such a tactile feedback in

neurological patients has been examined and a protocol has been

submitted to a local ethic committee.

In a related study, an active cane has been developed, which is

servo-controlled by the human gait thanks to IMU (Ady et al. 2013).

This will be the basis for the development of the interactive handle

of the cane to provide tactile feedback to the user.

Gesture-sound coupling

The sonification of arm movements is often based on the

movement of one point of the limb (coll. Legos ANR,

STMS). In an aiming experiment we showed the effect of

audiomotor coupling and heading for aiming (Boyer et al.

2013). The challenge for the rehabilitation of shoulder-

elbow coordination in stroke patients is to sonify the

temporal coordination of two variables (e.g. two angles).

Several modes of coupling were developed and tested in

healthy subjects, thanks to the braked orthosis developed in ISIR fitted with “musical objects”

(Bevilaqua et al. 2013) (see Françoise et al 2014 for another mode of coupling). Further investigations

in healthy subjects were performed and are under analysis before proposing experiments in stroke

patients. More generally, the effect of sound and mechanical constraints (for example those provided

by an exoskeleton, Jarrasse et al in press), should be combined for an efficient rehabilitation.

Preliminary effect of sound was also investigated in the context of hand prosthesis. This project is

complementary of the use of music during gait rehabilitation in Salpétrière (V. Marchand Pauvert).

References

Ady R, Bachta W, Bidaud P. (2013) Analysis of cane-assisted walking through nonlinear optimization. Robotics

and Automation ICRA 2013, 3866 – 3872 Bevilacqua F., Van Zandt-Escobar A., Schnell N., Boyer E. O., Rasamimanana N., Françoise J., Hanneton S.,

Roby-Brami A. (2013) Sonification of the coordination of arm movements. « Multisensory Motor Behavior :

Impact of sound ». Org Pr A. Effenecberg & Gerd Schmitz, Leibnitz University Hanover. September 2013 Boyer E.O., Babayan BM., Bevilacqua F., Noisternig M., Warusfel O., Roby-Brami A., Hanneton S., Viaud-

Delmon I. (2013) From ear to hand: the role of the auditory-motor loop in pointing to an auditory source.

Front Comput Neurosci. 2013 Apr 22;7:26. doi: 10.3389/fncom.2013.00026.

Françoise J., Schnell N., Bevilacqua F. (2014) MaD: Mapping by Demonstration for Continuous Sonification

ACM SIGGRAPH 2014 Emerging Technologies, Aug 2014, Vancouver, Canada, France. ACM, pp.16:1-16:1

Jarrassé N, Kuhne M, Roach N, Hussain A, Balasubramanian S, Burdet E, Roby-Brami A (2013). Analysis of

grasping strategies and function in hemiparetic patients using an instrumented object. Proceedings of the 13th

International Conference on Rehabilitation Robotics (ICORR). Pages 1-8.

Jarrassé N, Ribeiro AT, Sahbani A, Bachta W, Roby-Brami A. (2014) Analysis of hand synergies in healthy

subjects during bimanual manipulation of various objects. J Neuroeng Rehabil. 2014 Jul 30;11:113. doi:

10.1186/1743-0003-11-113.

Jarrasse N, Proietti T, Crocher V, Robertson J, Sahbani A, Morel G and Roby-Brami A (2014) Robotic

exoskeletons: a perspective for the rehabilitation of arm coordination in stroke patients. Frontiers in Human

Neuroscience, in press.

Vérité F., Bachta W., Morel G., (2013) Closed-loop control of a human Center-Of-Pressure position based on

somatosensory feedback. IEEE Intelligent Robots and Systems (IROS).

Vérité F., Bachta W., Morel G., (2014) Closed loop kinesthetic feedback for postural control rehabilitation.

IEEE Transactions on Haptics, Special Issue: Haptics in Rehabilitation and Neural Engineering. IEEE Trans

Haptics. 2014 Apr-Jun;7(2):150-60. doi: 10.1109/TOH.2013.64.

23

3 Future Work

As planned, we have acquired the equipment by the end of 2014. The immediate perspective in

2015 is to undertake and complete the experiments (use of human synergies for the control of hand

prosthesis, evaluation of the light touch in neurological patients in Salpétrière and sonification of

interjoint coordination). The experiments will be performed thanks to the doctoral students

contributing to the project. The link with the clinics will be developed thanks to C. Kemlin,

physiotherapist in Salpetrière. We shall also hire one engineer who will develop the command of the

hand prosthesis and 2 post doctoral fellows: one will work on the mechanisms of light touch for

assistance, E. Boyer will continue on the sonification project.

We are currently organizing the visit of Pr Archambault, Mc Gill University (in June 2015) by

submitting to invited professors grants. In addition, we plan to be visited by a Spanish doctoral student

for 2 months.

From a dissemination point of view, we plan to organize a workshop on the use of embedded

sensorimotor interfaces for rehabilitation and assistance.

4 Recruitment

Funded by SMART:

Claire KEMLIN, who is a physiotherapist in Salpetrière Hospital: half time in LIB-ISIR for two

years beginning in October 2014 (convention UPMC-APHP)

Ragou ADY, PhD student at ISIR recruited for extra 7 months from Nov 2014 (assistance to

equilibrium)

Jean Baptiste CAZENEUVE mechanical engineer, recruited at ISIR for 2 months (Nov-Dec 2014).

Planned:

Eric Boyer, Post-doc (April-September 2015) in STMS

Engineer 18 months at ISIR beginning spring 2015 (task: prosthesis)

Post-Doc 1 year at ISIR beginning spring 2015 (task: light touch for equilibrium)

Not funded by SMART:

Fabien VÉRITÉ: PhD student since 2012 at ISIR (AMN), full time on the project

Manelle MERAD PhD student since 2014 at ISIR (Doctoral school SMAER), full time on the

project

Adrienne GOUZIEN, medical resident in leave (APHP). Contract for 10 months (IUIS UPMC)

Tommaso PROIETTI PhD Student (ISIR, Bourse Ile de France), participating part time

Eric BOYER PhD Student (STMS, ANR LEGOS), participating part time

Jules Françoise, PhD Student (STMS), participating part time

Interns:

Funded by SMART:

Alejandro VAN-ZANDT ESCOBAR (STMS: May-September 2013)

Other funding sources:

Adriano TACILO RIBEIRO, (ISIR) Master 2 ENSTA-UPMC 2013.

Sandra MARTIN, (ISIR) M2 Cogmaster 2013.

Dijana NUIC, (STMS), M2 VHMA 2014.

Wahid TOUNSI (ISIR), M2 2014

24

5 Publications

Journal articles

Boyer E.O., Babayan BM., Bevilacqua F., Noisternig M., Warusfel O., Roby-Brami A.,

Hanneton S., Viaud-Delmon I. (2013) From ear to hand: the role of the auditory-motor loop in

pointing to an auditory source. Front Comput Neurosci. 2013 Apr 22;7:26. doi:

10.3389/fncom.2013.00026.

Vérité F., Bachta W., Morel G., (2014) Closed loop kinesthetic feedback for postural control

rehabilitation. IEEE Transactions on Haptics, Special Issue: Haptics in Rehabilitation and

Neural Engineering. IEEE Trans Haptics. 2014 Apr-Jun;7(2):150-60. doi:

10.1109/TOH.2013.64.

Jarrassé N, Ribeiro AT, Sahbani A, Bachta W, Roby-Brami A. (2014) Analysis of hand

synergies in healthy subjects during bimanual manipulation of various objects. J Neuroeng

Rehabil. 2014 Jul 30;11:113. doi: 10.1186/1743-0003-11-113.

Gonzalez, F. and Gosselin, F. and Bachta, W. (2014). Analysis of Hand Contact Areas and

Interaction Capabilities During Manipulation and Exploration. IEEE Transactions on

Haptics. In press.

Conferences

Vérité F., Bachta W., Morel G., (2013) Closed-loop control of a human Center-Of-Pressure

position based on somatosensory feedback. IEEE Intelligent Robots and Systems

(IROS).

Bevilacqua F., Van Zandt-Escobar A., Schnell N., Boyer E. O., Rasamimanana N.,

Françoise J., Hanneton S., Roby-Brami A. (2013) Sonification of the coordination of arm

movements. « Multisensory Motor Behavior : Impact of sound ». Org Pr A. Effenecberg

& Gerd Schmitz, Leibnitz University Hanover. September 2013

Roby-Brami A., Van Zandt-Escobar A., Jarrassé N., Robertson J., Schnell N., Boyer E. O.,

Rasamimanana, Hanneton S., Bevilacqua F. (2014) Toward the use of augmented auditory

feedback for the rehabilitation of arm movements in stroke patients. 17th European

congress of physical rehabilitation medicine. Marseille May 2014.

Françoise J., Schnell N., Bevilacqua F. (2014) MaD: Mapping by Demonstration for

Continuous Sonification ACM SIGGRAPH 2014 Emerging Technologies, Aug 2014,

Vancouver, Canada, France. ACM, pp.16:1-16:1

Ady R., Bachta W., Bidaud, P. (2014). Development and control of a one-wheel telescopic

active cane. IEEE RAS/EMBS BioRob Pages 461 – 466

Other related publications (SMART not acknowledged)

Jarrassé N, Kuhne M, Roach N, Hussain A, Balasubramanian S, Burdet E, Roby-Brami A

(2013). Analysis of grasping strategies and function in hemiparetic patients using an

instrumented object. Proceedings of the 13th International Conference on

Rehabilitation Robotics (ICORR). Pages 1-8.

Ady R, Bachta W, Bidaud P. (2013) Analysis of cane-assisted walking through nonlinear

optimization. Robotics and Automation ICRA 2013, 3866 – 3872.

Jarrasse N, Proietti T, Crocher V, Robertson J, Sahbani A, Morel G and Roby-Brami A

(2014) Robotic exoskeletons: a perspective for the rehabilitation of arm coordination in

stroke patients. Frontiers in Human Neuroscience, in press.

25

6 Events

Workshop "Intégration corporelle de la technique" (Body integration of technic) Défi-sens

CNRS and ISCC http://ict2012.isir.upmc.fr/ in 2012.

Participation to "LEGOS days" ANR project STMS, 18-19 March 2014

Seminars

Milica Djuric-Jovicic (Belgrade university) ISIR 20/06/2013

F. Bevilaqua, LIB (Decembre 2013).

Hugh Herr (MIT), ISIR 13/6/2014 "On the Design of Bionic Leg Devices: The Science of

Extreme Interface".

Meeting with C. Lenay and O. Gapenne (UTC) April 2014.

Meeting with J. Mizrahi (Technion) ISIR 5 March 2014

Internal events

Kick off- meeting: 21/11/2013

Meeting: 12/09/2014

General audience events

N. Jarrassé : participation to the documentary "Bras de fer" for the series "LA BOITE

NOIRE"

Roby-Brami in « La Grande Équation », Normand Mousseau (Université de Montréal)

Broadcast radio Ville-Marie (Québec), April 2014

N. Jarrassé: RFI Broadcast "Autour de la question" November 2014 "Quel défi pour

l'humain avec les technosciences ?"

N. Jarrassé: Participation to Semaines Sociales de France, Lille Nov 2014 "l'homme et les

technosciences: le defi".

26

27

ONBUL

28

ONBUL: Online Budgeted Learning Responsible of the project: Ludovic DENOYER [LIP6]

Partners:

LIP6: Ludovic DENOYER, Patrick GALLINARI, Gabriella CONTARDO

ISIR: Benoit GIRARD, Mehdi KHAMASSI, Nassim AKLIL

Web site: http://onbul.lip6.fr

1 The Project at a glance

Context and Objectives

The emergence of large-scale databases and big data has recently motivated the development of

budgeted machine learning models able to learn under operational constraints in term of memory/CPU

consumptions, data access, etc. … It involves integrating the constraints of scarce resources in the

learning process itself. In parallel, in the neuroscience community, reinforcement learning online

capabilities are understood as results of the coexistence of complementary learning systems, the choice

of which brain learning system should be activated at each moment being also based on limited

budgets (mainly in terms of computational cost). Based on the observation that the recent context of

learning under stress is highly relevant both for massive data processing and neuroscience, we aim to

study this issue in the online learning context that seems suited for these two families of problems. We

seek cross-fertilization between the two fields: 1) import the multi-model (i.e. multiple learning

systems) from neuroscience in statistical machine learning architectures as a potential solution to

budgeted data analysis and prediction; 2) update the reinforcement learning concepts in use in

neuroscience by a confrontation with modern budgeted learning frameworks.

The project is organized around scientific objectives and different concrete applications. The

scientific objectives are to develop original budgeted learning models. We will explore two families of

models: a first family where the information acquisition process will be modeled as a sequential

process, and where reinforcement learning and representation learning techniques will be used

together. The second family that is more human-inspired aims at developing model selection

approaches where, at each step of the sequential process, the system has to choose between different

concurrent decision/learning models, each model having its own prediction/learning ability, but also

its own budget. These families of models will be both explored when the budget applies in prediction

only, but also when the budget applies during learning, resulting in online budgeted learning models.

Experiments will be made on classical machine learning models involving large amount of data

(recommendation, image classification…), in robotics (robot localization), and also in the

neuroscience domain (modeling behavioral data).

2 Scientific progress and results

During the first year of the project, we have mainly focused on the first family of models i.e.

sequential budgeted models as described in Section 2.1. This has been both investigated in terms of

big data processing (LIP6) and through the set up of a new robotic paradigm in collaboration between

the two partners in order to have a robotic automatically collect information under budget constraints

for localization. The development of model selection approaches (Section 2.2) started this year. It

involved first investigations of computational principles from neuroscience which best explain how

animal behavior relies on budget constraints to perform online model selection and learning

parameters regulation (meta-learning). The second part of model selection approaches will start next

29

year and will again involve collaboration between the two partners to have a robot perform online

model selection under budget constraints during a navigation task.

2.1 Sequential Budgeted Learning

Existing machine learning approaches are based on the following strong assumptions: (i) It is

assumed that data to be processed is fully known, that is to say that needed information was previously

acquired. (ii) It is also assumed that the system is not constrained in terms of inference or learning

time, memory space,... However, these assumptions are unrealistic in the emerging eco-system: (i) The

usual paradigm which considers that the learning method has to optimize a single task-dependent

performance without taking into account the time taken to learn, the complexity of the produced

method, or even its CPU/Memory consumption is outdated. (ii) Considering that information has been

already acquired and stored in a Big Data context is unrealistic since the amount of produced data is so

huge that, in the best case, it can only be stored through very large expensive storage clusters, and in

many cases, it cannot be stored at all. The information acquisition process is thus a key element in

machine learning, which is today only done by hand. The ability of a system to automatically

determine which information has to be collected but also to realize a good balance between

performance and “operationability”. This ability is thus a key aspect of future machine learning

models that we started to develop in this project.

2.1.1 Principles

The project aims at studying the following process in a massive data context: (i) first a model has to

choose which information to use. This first phase is called information acquisition and is a critical

point. Collecting misleading or irrelevant information will both decrease the ability of the system to

solve a particular task and increase its budget i.e. collecting information can be time/memory-

consuming, or even expensive. This acquisition process occurs during both the learning phase where

one wants to build a ''good'' training set, and during the testing phase where one wants to predict

outputs. In the first case, the model has to learn which information, but also which supervision, to get,

while in the second case the model has to acquire the information that will help it to do a good

prediction. (ii) At each step of the process, the collected information has to be aggregated. This phase

called representation learning - which has seen a surge of interest during the last year with the

emergence of the representation learning community - aims at ''aggregating'' the collected data, and

extracting relevant information on which learning/inference will be done. (iii) At last, the system has

to perform a prediction. During the learning process, this last step aims at producing or updating a

model, while during the testing phase; it aims at producing an output for a given datum. Note that,

since the information acquisition will be guided by previously acquired information, step (i) and (ii)

can be inter-dependent. It can be instantiated both during learning and during inference. At last, in

order to deal with large scale datasets, these three steps have to be jointly constrained by budgeted

constraints like time spent, CPU consumption, memory usage,...

Positioning w.r.t state-of-the-art

2.1.2 Main achievements Considering the previous description of what we intend to realize, we have already proposed some

original approaches to different aspects of this proposal:

30

The development of sequential acquisition models has been handled by developing specific

approximated reinforcement learning models. The underlying idea is to model the

acquisition process by a Markov Decision Process where each action can be either an

acquisition action or a decision action allowing the model to choose if it needs to get more

information, or if it can decide what to predict. We have proposed a new reinforcement-

learning algorithm for the case where the number of acquisition steps is fixed and cannot be

exceeded. Applications on image classification, where the classification model sequentially

explore parts of the image have been proposed.

We have also proposed original representation learning algorithms for states in a partially

observed Markov Decision Process (POMDP). When facing approximated reinforcement

learning problems, the input consists in a feature vector - called observation - which is

assumed to fully characterize the current state of the process, thus allowing for an optimal

action choice. However, this assumption is unrealistic in real-life applications where the

observation is only a partial view of the current state provided by limited sensors. The model

operates in two steps. (i) First, it learns how to find good representations on a set of randomly

collected trajectories. This unsupervised operation is used to learn the system only once, and

may be used to tackle different tasks sharing the same dynamical process. (ii) The model then

infers new representations for any new trajectory, these representations being then used for

discovering an optimal policy for a particular reward function.

2.1.3 Work in progress

We are currently trying to merge the two approaches by developing (sequential) models able to

simultaneously learn a representation of the acquired information, but also how/when to acquire

information and when/how to predict. A first model – which is not sequential – is currently under

development and is showing promising results. It is based on a L1-regularization technique where the

L1-regularized weights are not applied on the parameters of the model, but on the information that can

be acquired: a zero-weight on a particular input means that this information is not needed for

computing a good prediction. A sequential extension of this model is also under development.

Submissions are planned to both ICLR 2015 and ICML 2015. Concrete experiments have been made

on different types of data: (i) image classification (ii) toys MDPs (iii) recommender systems.

2.2 Model selection approaches Work on model selection is described in more details in the PhD report joined to this report.

In the last 15 years, the theory of reinforcement learning has significantly contributed to researches in

machine learning, robotics and neuroscience. It formally specifies how an agent should choose the best

actions to perform and update this choice through learning by trial-and-error so as to maximize long-

term cumulative rewards. This theory has helped better understand the mechanisms underlying

reinforcement-based plasticity in brain circuits dedicated for action selection. In parallel, it contributed

in designing adaptive agents that can learn optimal paths to rewards in simulated discretized grid

worlds. However, the application of reinforcement learning algorithms to robotics experiments –

involving continuous noisy unpredictable environments - produced limited progresses, due to

applications to quite simple problems, with a small number of states and actions, to slowness in

learning and to systematic instability observed throughout the learning process. This led to the idea

that an online dynamic regulation of reinforcement learning algorithms was necessary to produce

efficient and robust robotics results. Such online regulation is called meta-learning and consists in two

main processes:

(i) The online regulation of reinforcement learning parameters (e.g. the exploration parameter) so

that they are not constant over time but are rather dynamically adapted to the current task

requirements and performance of the agent. For instance, if the agent detects that its

performance is getting worse, this may indicate that a change in the task has occurred and that

31

the agent should change its parameter for exploration so as to re-explore and re-learn the new

task contingencies.

(ii) The online selection between learning models through the monitoring of the agent’s

performance: each model having its own advantages and drawbacks, the agents should be able

to learn which model is the most appropriate for each subpart of the learning process. Meta-

learning algorithms have been recently applied to online learning problems both to the

Neuroscience and Robotics. However, they do not explicitly take into account the budget

constraint: how much time and computational cost the agent can use to learn the task? On the

other hand, budgeted learning methods have been proposed in machine learning, but they do

not yet work online. The objective of this work is thus to extend budgeted learning methods to

make them work online by taking inspiration from recent neuroscience data, and to apply them

to online meta-learning and model selection tasks in Robotics. The initial criterion that was

used at ISIR for online model selection consisted in learning in a meta-controller which

learning model among two was the most efficient at each moment. The two tested learning

models were a model-based system that learns progressively the possible transitions between

states in the environment; the second tested model was a model-free system that avoids

learning transitions and rather simply learns reward values associated to each possible action

in each possible state. We have previously tested such algorithm in navigation in a simple

robotic task. The robot could efficiently but slowly adapt to changes in the goal location by

adapting its model selection.

The objective of the current investigations is to explicitly take into account the computational cost

(budget) of each model so as to perform online model selection as a function of this cost. This should

reduce the learning cost and should push the algorithm to learn the task in a shorter time. On the other

hand, in uncertain situations following a change in the environment, such an algorithm should be able

to detect that a high cost is necessary to adapt to the new situations by performing computations in

more than one model and analyzing which model is the most efficient in this new situation.

3 Future Work

The sequential budgeted machine learning models will be finished during the first half of 2015 with

the development of sequential representation learning models and their applications to collaborative

filtering. The second half of the year will be devoted to both the development of learning models with

a large number of actions – which is a needed condition to large scale budgeted acquisition – and to

the development of models able to learn under budgeted constraints.

Concerning the robotics task of automatic navigation, ongoing work on budgeting sensors will be

completed during the first third of 2015. The rest of the year will be devoted to the development of the

learning online budgeted algorithm aiming at selecting between navigation strategies depending on

their respective effectiveness and computational costs.

4 Recruitment

In addition to permanent researchers, the project currently involves two PhD students:

Nassim Aklil at ISIR who mainly works on applying sequential budgeted learning models to

robot, and who is starting to explore model selection approaches to online budgeted learning

Gabriella Contardo at LIP6 (grant not provided by the LABEX) who works on sequential

budgeted learning models applied to ‘’big data’’ tasks.

A postdoc will be recruited in 2015 for the specific development of online sequential budgeted

algorithms.

32

5 Publications

Contardo G., Denoyer L., Artières T., Gallinari P. (2014) Learning States Representations

in POMDP. CoRR abs/1312.6042 (2013) and ICLR 2014 (Short paper)

Contardo G., Denoyer L., Artières T., Gallinari P. (2014) Apprentissage Sous Contraintes

Budgetisées – Application à la Recommendation – Poster CAP 2014

Contardo G., Denoyer L., Artières T., Gallinari P. (2014): Apprentissage Sous Contraintes

Budgetisées – Application à la Recommendation – Poster CAP 2014

Dulac-Arnold G., Denoyer L., Thome N., Cord M., Gallinari P. (2014) Sequentially

Generated Instance-Dependent Image Representations for Classification, Internation

Conference on Learning Representations – ICLR 2014

Aklil N., Marchand A., Fresno V., Coutureau E., Denoyer L., Girard B., Khamassi M.

(2014) Modelling rat learning behavior under uncertainty in a non-stationary multi-armed

bandit task. Fourth Symposium on Biology of Decision Making (SBDM 2014). Paris.

Denoyer L., Gallinari P. (2014) Deep Sequential Neural Network (2014) - Workshop

Deep Learning NIPS 2014

33

PhD Thesis: Navigation learning with multiple models under budgeted

constraints PhD student: Nassim Aklil

Supervisor(s): Mehdi Khamassi (ISIR) & Ludovic Denoyer (LIP6)

Laboratory: Institute of Intelligent Systems and Robotics (ISIR, UPMC-CNRS)

Doctoral School: ED3C, Cerveau Cognition Comportement (UPMC)

Period: 01/09/2013 – 31/08/2016

1 Description

In the last 15 years, the theory of reinforcement learning has significantly contributed to

researches in machine learning, robotics and neuroscience. It formally specifies how an agent should

choose the best actions to perform and update this choice through learning by trial-and-error so as to

maximize long-term cumulative rewards. This theory has helped better understand the mechanisms

underlying reinforcement-based plasticity in brain circuits dedicated for action selection. In parallel, it

contributed in designing adaptive agents that can learn optimal paths to rewards in simulated

discretized grid worlds. However, the application of reinforcement learning algorithms to robotics

experiments – involving continuous noisy unpredictable environments - produced limited progresses,

due to applications to quite simple problems, with a small number of states and actions, to slowness in

learning and to systematic instability observed throughout the learning process. This led to the idea

that an online dynamic regulation of reinforcement learning algorithms was necessary to produce

efficient and robust robotics results. Such online regulation is called meta-learning and consists in two

main processes:

1. The online regulation of reinforcement learning parameters (e.g. the exploration parameter) so

that they are not constant over time but are rather dynamically adapted to the current task

requirements and performance of the agent. For instance, if the agent detects that its

performance is getting worse, this may indicate that a change in the task has occurred and that

the agent should change its parameter for exploration so as to re-explore and re-learn the new

task contingencies.

2. The online selection between learning models through the monitoring of the agent’s

performance: each model having its own advantages and drawbacks, the agents should be able

to learn which model is the most appropriate for each subpart of the learning process.

Meta-learning algorithms have been recently applied to online learning problems both to the

Neuroscience and Robotics. However, they do not explicitly take into account the budget constraint:

how much time and computational cost the agent can use to learn the task? On the other hand,

budgeted learning methods have been proposed in machine learning, but they do not yet work online.

The objective of this PhD thesis is thus to extend budgeted learning methods to make them work

online by taking inspiration from recent neuroscience data, and to apply them to online meta-learning

tasks in Robotics.

34

2 Results

Two experimental works have been performed so far:

1. We investigated neuroscience data illustrating meta-learning processes during the online

regulation of exploration in rats having to learn a multi-arm bandit task under different levels

of uncertainty (Figure 1 Left).

2. We prepared a robotics setup to apply online budgeted learning in an initial simple navigation

task. The goal of the robot is to minimize the budget (here the number of accessed data about

the environment from its sensors) in order to determine its current position (Figure 1 Right).

FIGURE 1: (LEFT) ONLINE REGULATION OF EXPLORATION (META-LEARNING) IN A RAT MULTI-ARMED BANDIT TASK. RATS

HAVE TO CHOOSE AT EACH TRIAL BETWEEN 3 LEVELS, EACH ONE BEING ASSOCIATED WITH A DIFFERENT PROBABILITY OF

REWARD. MODEL FITTING ON RAT BEHAVIOR USING A META-LEARNING ALGORITHM REVEALED THAT THEY DYNAMICALLY

REGULATE THEIR EXPLORATION LEVEL IN ORDER TO EFFICIENTLY SOLVE THIS TASK. (RIGHT) ROBOTICS NAVIGATION SETUP

TO INVESTIGATE ONLINE BUDGETED LEARNING FOR THE DETERMINATION OF THE ROBOT’S LOCATION. THIS PART OF THE

WORK HAS JUST STARTED AND WE HAVE MADE THE SPECIFICATION AND STARTED THE ACQUISITION OF A DATASET WITH

DIFFERENT ROBOT POSITIONS, SENSING DATA AND MOVEMENTS.

3 Publication

Aklil N., Marchand A., Fresno V., Coutureau E., Denoyer L., Girard B., Khamassi M. (2014)

Modelling rat learning behavior under uncertainty in a non-stationary multi-armed bandit task.

Fourth Symposium on Biology of Decision Making (SBDM 2014). Paris.

35

SeNSE

36

SeNSE: Socio-Emotional Signals Responsible of the project: Catherine ACHARD

Partners:

ISIR (UPMC): C. Achard, K. Bailly, M. Chetouani, S. Dubuisson, O. Grynzspan

LIP6 (UPMC): P. Garda, C. Marsala, A. Pinna, M. Rifqi

LTCI (Telecom Paris-Tech): C. Clavel, S. Essid, C. Pelachaud, G. Richard

STMS (IRCAM): F. Bevilacqua, G. Assayag

Web site: http://sense.isir.upmc.fr/en/

1 The Project at a glance

Context and Objectives

The SeNSE project, which focuses on social emotional signals exchanged during an

interaction, investigates topics ranging from signal acquisition (video, audio, neurophysiological) to

interaction (virtual agent, musical interaction, people interaction) including interpretation and

modeling. This project brings together key partners of social signal processing, machine learning,

electronics and human computer interaction. The SeNSE project will break new ground for the multi-

modal analysis and synthesis of social behavior. We are particularly interested in both dynamical and

temporal aspects of interaction.

The methodology will deal the heterogeneous nature of cues from low-level information (audio, video,

EEG, ECG, EDA…) to high-level information (emotions, social attitude, user traits…). Thus, the

considered signals are multimodal with their own dynamics and they may influence each other during

social interactions. For example, for a virtual agent, understanding the dynamics of socio-emotional

signals is one of the challenges for the analysis and synthesis of realistic behaviors. In musical

interaction, analyzing and describing multi-modal signals in large group provide new paradigms for

expressive and collaborative interactions.

Analyzing such situations by social signal processing techniques requires new models and

methodologies. To tackle this challenging problem, we focus on three main aspects (1) developing

computational models of socio-emotional behaviors considering different modalities (audio, gestural,

37

physiological and brain signals...) (2) studying intermodal and temporal dependencies for both intra

and inter personal signals and (3) designing smart devices embedding socio-emotional processing.

2 Scientific progress and results

The first two main aspects of SeNSE have been studied during the first 18 months (total duration 48

months).

2.1. MULTI-MODAL MODELS FOR SOCIO-EMOTIONAL BEHAVIOR

In the literature, low-level information have been considered to study social interactions by

mainly exploiting audio, video or either physiological signals... The SeNSE project investigates these

various multi-modal signals in order to build rich computational models of socio-emotional behaviors.

The aim of physiological study is to design a smart sensor with an embedded adaptive emotion

recognition algorithm. For this purpose, a set of experimental physiological data has been recorded

during the last months. We considered a person watching a football game during the world cup soccer

in Brazil and we recorded skin conductivity, respiratory and ECG signals. The aim is to map features

extracted from physiological signals to game’s events such as goal against the supported team, of the

supported team, corner, and free kick… We expect that these events will elicit natural emotions will,

which will allow us to infer relevant features for automatically recognizing emotion from

physiological sensors. The objective is to design electronic sensors able to embed an adaptive

recognition algorithm from these cues. The rationale here is to develop systems that are able to adapt

in real-time, while ensuring privacy of data (all data are acquired and processed by the same device).

A PhD supervised by M. Rifqi, A. Pinna, P. Garda and C. Marsala started in November 2014 on

this issue.

Regarding audio-visual signals, we focused on vocal and facial features corresponding to

social attitudes. We have provided a preliminary state-of-the-art on the various definitions and theories

associated with social attitudes and studied the available annotations of the SEMAINE Database.

Prosodic cues and action units have been extracted on this database. As annotations of social stances

are not provided in SEMAINE, we have focused on the study of their correlation with emotional

valence, as a first step. Future work will focus to the development of a database dedicated to the

analysis of social stances (e.g. dominance and friendliness) in a human-agent interaction relying on

SEMAINE protocols and the analysis of intermodal dependencies with the aim to make the agent able

to express social stances through prosodic and facial expression cues. A PhD supervised by K.

Bailly, C. Clavel and G. Richard started in October 2014 on this issue.

2.2. MODELING TEMPORAL DEPENDENCIES OF INTER AND INTRA INDIVIDUAL BEHAVIOURS

We study the modeling of temporal dependencies of features and behaviors at inter and intra

personal levels through various machine learning approaches (influence models, non-negative matrix

factorization, one-class SVMs) and various interactive situations (imitation, meeting, social agent,

musical interaction...).

Influence Models (IM) were used to model turn-taking in a meeting of 4 persons. Rather than

using the IM as classifier, the influence matrix is estimated and employed to characterize meetings.

The influence matrix is then used as a feature input of an SVM classifier. Preliminary results show the

interest of this characterization for a role recognition task. A PhD supervised by C. Achard and S.

Dubuisson will start in January 2015 on this issue.

We study co-factorization of non-negative matrices for EEG-based characterization of specific

forms of interaction between two individuals, such as imitation. Efforts have been dedicated to the

estimation of appropriate NMF models, in particular considering co-factorization schemes, whereby

the NMF models for the two subjects are estimated jointly in a coupled fashion. Preliminary results are

38

encouraging because some levels of correlation between NMF activations relating to each subject

during imitation phases were measured. A PhD supervised by S. Essid and M. Chetouani started in

October 2014 on this issue.

In another context, we investigated automatic measurement of imitation in a dyad.

Participant’s gestures during an EEG hyper-scanning task are characterized with 1-class SVM models.

Then a measure of imitation is derived from the likelihood ratio between these models. The

comparison with manual indexing validates the method at both behavioral and neural levels,

demonstrating its ability to discriminate significantly the periods of imitation and non-imitation during

social interaction.

Finally, the timing of a specific non-verbal social behavior, that is, communicative gaze, has

also been investigated. One of the most crucial gaze communicative actions is gaze following, i.e.

when a social partner follows with her/his eyes the gaze of the other partner. The goal was to examine

what is the acceptable delay between the eye movements of the partners for them to be engaged in

gaze following behaviors. An experiment was conducted, where participants were asked to judge

whether the gaze of a virtual human avatar responded to their gaze or not.

All previous studies have focused on short-term social cues dependencies, but in order to adapt

the interaction according to the users, a longer-term study is needed. Thus, by addressing two different

use cases, Embodied Conversational Agents and Musical Improvisation Agents, we aimed at building

a generic adaptive interaction model that should avoid the shortcoming of using ad-hoc rules. In

particular, the long-term model should allow modulating short-term dependencies such as turn-taking

and the synchronization of non-verbal elements.

During the first 18-month of the project, we started to establish a state-of-the art of adaptive

interaction models that could fit our goals. Several disciplines have been covered from Embodied

Conversational Agents, Musical Interactions and Music improvisation or Human-Robot interaction.

From a modeling point of view, very different architectures are currently reviewed, such as rule-based

systems, probabilistic models (e.g. HMM) or autonomous agents. A PhD supervised by F.

Bevilacqua and C. Pelachaud is currently working towards the design of interaction scenarios to

compare various approaches on this issue.

3 Future Work

In 2015, we will focus on data collection. This will be performed in two steps: (1) analysis of

the state-of-the-art on databases for social interactions investigations (meetings, presentations, data

shared during challenges…), (2) identification of proof-of-principle situations for the PhD thesis (e.g.,

inter-brain synchrony characterization, social stances modeling, and musical interactions…).

Regarding the computational models, we will develop and evaluate models dealing with (1)

emotional behaviors and (2) temporal dependencies of intra and inter individual behaviors. These will

be performed for analysis, modeling and synthesis phases in human-human and human-virtual agent

situations.

39

4 Recruitment

INTERNSHIPS

Study of multimodal synchrony during social interaction, F. Aujoux, M2 Internship (march-

july 2014). Supervisors: C. Achard (ISIR), S. Dubuisson (ISIR)

Modeling the temporality of multimodal social cues exchanged during natural interactions,

S. Fang, M2 Internship (july-december 2014). Supervisors: C. Achard (ISIR), S. Dubuisson (ISIR)

Multimodal analysis and recognition of social signals, L. Chen, M2 Internship (march-august

2014). Supervisors: C. Clavel (LTCI-CNRS) and K. Bailly (ISIR)

Study of the judgment of agency in the gaze modality, S. Recht, M1 Internship (March-June

2014). Supervisor: O. Grynszpan (ISIR)

Modeling of neurophysiological activity related to the dynamics of an interaction with latent

variables analysis, A. Hajlaoui, M2 Internship (april-august 2014). Supervisor: S. Essid (LTCI-

CNRS), M. Chetouani (ISIR)

PHD THESIS

● Embedded architecture and physiological sensors, W. Yang, 2014-2017. Supervisors: C.

Marsala (LIP6), M. Rifqi (LIP6) and A. Pinna (LIP6)

● Multimodal analysis and recognition of social signals: application to social stance generation

in virtual agents, T. Janssoone, 2015-2018. Supervisors: G. Richard (LTCI-CNRS), C. Clavel

(LTCI-CNRS) and K. Bailly (ISIR)

● Study of social cues exchanged during natural interactions, S. Fang, 2015-2018, Supervisors:

C. Achard (ISIR) and S. Dubuisson (ISIR)

● Temporal Adaptation of Interaction, Kevin Sanlaville, 2013-16. Supervisors: C Pelachaud

(LTCI-CNRS), F. Bevilacqua (STMS) , G. Assayag (STMS)

● Modeling interactional neurophysiological activity using latent variables, A. Hajlaoui, 2014-

2017, Supervisors: M. Chetouani (ISIR) and S. Essid (LTCI-CNRS)

5 Publications

S. Buisine, M. Courgeon, A. Charles, C. Clavel, J.C. Martin, N. Tan, O. Grynszpan, The Role

of Body Postures in the Recognition of Emotions in Contextually Rich Scenarios,

International Journal of Human-Computer Interaction, 30 (1), 2014

S. Campano, J. Durand, C. Clavel, Comparative analysis of verbal alignment in human-human

and human-agent interactions, In Proceedings of LREC 2014, Reykjavik

S. Campano, N. Glas, C. Clavel, C. Pelachaud, Production d'Hetero-Répétition chez un ACA,

In Proc. Workshop Affect, Compagnon Artificiel, Interaction, 2014

M. Chetouani, Role of Inter-Personal Synchrony in Extracting Social Signatures: Some Case

Studies, International Workshop on Roadmapping the Future of Multimodal Research, in

conjunction with the ACM International Conference on Multimodal Interaction

(ICMI'14), Istanbul, Turkey, November 2014.

M. Courgeon, G. Rautureau, J.C. Martin, O. Grynszpan, Joint Attention Stimulation using Eye-

Tracking and Virtual Humans, IEEE Transactions on Affective Computing, July 2014

E. Delaherche, G. Dumas, J. Nadel, M. Chetouani, Automatic measure of imitation during

social interaction: a behavioral and hyperscanning-EEG benchmark, Pattern Recognition

Letters, to appear

C. Langlet and C. Clavel, Modélisation des questions de l’agent pour l’analyse des affects,

jugements et appréciations de l’utilisateur dans les interactions humain-agent, In Actes de

TALN 2014, Marseille

C. Langlet, C. Clavel, Modelling user's attitudinal reactions to the agent utterances: focus on

the verbal content, LREC Workshop on Emotion, social signals, sentiment & linked open

data, May 2014

40

S. Michelet, C. Achard, M. Chetouani, Evaluation automatique de l'imitation dans l'interaction,

Reconnaissance de Formes et Intelligence Artificielle (RFIA 2014).

K. Sanlaville, F. Bevilacqua, C. Pelachaud, G. Assayag, Adaptation in an Interactive Model

designed for Human Conversation and Music Improvisation: a preparatory outline, Workshop

Affect, Compagnon Artificiel Interaction (WACAI’1), 2014, Rouen

6 Events

SEMINARS

Automatic Recognition of Affective and Social Signals

Hatice Gunes, from Queen Mary University, Londres,

Télécom-ParisTech, the 10/09/14

Understanding conversational social video

Daniel Gatica-Perez, from IDIAP, EPFL

ISIR, UPMC, the 09/10/2013

L'interaction spontanée chez l'homme: neuroimagerie et modèles computationelles

Guillaume Dumas, from Institut du cerveau et de la moelle épiniaire

ISIR, UPMC, the 02/10/2013

The MEI Robot: Towards Using Motherese to Develop Multimodal Emotional Intelligence

Angelica Lim, from Okuno Speech Media Processing Lab

ISIR, UPMC, the 27/09/2013

SPECIAL SESSION

Special session on “Behavior Imaging” at the IEEE International Conference on Image

Processing (ICIP), October 2014.

Organizers: Séverine Dubuisson (ISIR), Jean-Marc Odobez (IDIAP), Mohamed Chetouani (ISIR)

Human behavior understanding using both computer vision and signal processing has become of

major interest since the emergence of numerous applications in various domains, such as social signal

processing, affective computing or human-computer interaction. Recent advances in computer vision,

signal processing and pattern recognition now make it possible to consider the development of tools or

systems for human-human interaction or human-computer interaction analysis.

In line with these current efforts, behavior imaging was first introduced in the context of

behavioral disorder monitoring (e.g. autism). The key concept of behavior imaging is to improve,

through interdisciplinary approaches, automatic computing methods with the long-term goal of

enhancing human behavior analysis.

41

Phd Thesis: Study of social cues exchanged during natural interactions PhD student: Sheng FANG

Supervisor(s): Catherine Achard and Séverine Dubuisson

Laboratory: ISIR

Doctoral School: SMAER

Period: 01/01/2015 – 31/12/2017

1 Description

Recent researches in cognitive science define social intelligence as the ability to express and

recognize social signals produced during natural social interactions such as politeness, empathy,

kindness, conflict, etc., coupled with the ability to engage and maintain an interaction. A new

discipline is emerging: social signal processing, which aims to understand and model social

interactions (for human sciences, social sciences, and communication sciences), and provides similar

capabilities to computers (for human/computer interaction, animation of avatars...)

This PhD thesis investigates social signals exchanged during natural interactions by

developing a computational model of interaction able to estimate its quality, its strength and its

weakness. This multimodal (speech, face, gestures, posture...), dynamical (evolving over time) and

hierarchical (different levels of characterization: gestures, facial orientation, involvement, synchrony,

different time levels) model should both consider inter- and intra-personal temporality. It will assess

both Human / Computer interaction (HCI) and exchanges in dyads for applications such as interactive

robotic, assistance to people in a medical context, modeling and objectification in cognitive science,

including psychopathology, communication science...

The first problem will be to identify relevant social cues to develop this model. If a first

intuition is to consider that gestures and speech are important, further reflection leads us to consider

how representing gesture and speech signals. Even if the Kinect provides now 3D coordinates of each

joint of the skeleton, how using all of this information to obtain an exploitable measure? Should we go

back to the hands quantity of motion? To the respective body orientations? To head nods? To facial

expressions...? And it is the same for speech where many descriptors exist (turn taking, pitch, speed,

rhythm...).

We then propose to build a high level model of social interaction in which the temporal

dynamics play a very important role. Thus, the proposed model must take into account both the inter-

personal and intra-personal temporality between social signals and several time scales. Moreover, this

model can be directly constructed from low level signals (orientation of the head or torso, gesture, turn

42

taking, prosody), or be built from intermediate signals such as involvement, synchrony ... If these

concepts are regularly used in the literature, their definition remains unclear and often uses a small

amount of information, coupled with low level models often heuristics. Some researchers measure for

example synchrony by a simple correlation between descriptors [Barbosa, 2010]. Others define

involvement from static data, without consideration of temporality [Hernandez 2013]. However, we

believe that it is not enough. Indeed, synchrony involves more complex parameters as soon as we want

to take into account several modalities, such as audio and video, for which synchrony is not

characterized at the same moments, or with the same parameters [Delaherche et al. 2012].

Dialogs with experts in psychology show that it is very difficult to define a priori what are the

relevant signals and why. Similarly, the concepts used by experts to define the interaction quality are

very subjective and difficult to quantify. We propose in this thesis, to learn a model of social

interaction from a database, where participants will interact through well-known interactive scenarios

defined by psychologists (e.g., survival task...) and where high level annotations (leadership,

dominance...) will be provided by experts.

This model must be adaptive in order to contextualize its definition according to partners,

environment or situation and may, for a given social exchange, estimates its quality and drawbacks:

lack of dialogue, unsuitable posture, lack of involvement of a partner, gaze avoidance, gestures

dyssynchrony...

This assessment can then be used in several ways. First, it will help automatic evaluation of

interactive systems involving humans, robots/avatar as well as complex situations gestures, music... It

may also help patient screening; such as identifying early signs of lack of synchrony during parent-

infant as in autism. Another application could be assessment of communicative skills where

individuals can be found automatically and objectively evaluate.

[Barbosa 2010] A. V. Barbosa, E. Vatikiotis-Bateson, M. Oberg, and R.-M. Déchaine. An instantaneous

correlation algorithm for assessing intra and inter subject coordination during communicative behavior,

Modeling Human Communication Dynamics,NIPS Workshop 2010.

[Delaherche et al. 2012] E. Delaherche, M. Chetouani, A. Mahdhaoui, C. Saint-Georges, S. Viaux, D. Cohen,

“Interpersonal synchrony: a survey of evaluation methods across disciplines”, IEEE Transactions on Affective

Computing, 3 (3): 349-365, 2012.

[Hernandez 2013] J. Hernandez, Z. Liu, Geoff Hulten, D. DeBarr, K. Krum, Z. Zhang Measuring? The

Engagement Level of TV Viewers, 10th IEEE International Conference on Automatic Face and Gesture

Recognition, Shanghai, China, April 22-26, 2013.

43

Phd Thesis:

Modeling interactional neurophysiological activity using latent

variables PhD student: Ayoub HAJLAOUI

Supervisor(s): Slim ESSID (LTCI-Telecom ParisTech), Mohamed CHETOUANI (ISIR-UPMC)

Laboratories: LTCI Telecom ParisTech, ISIR - UPMC

Doctoral School: SMAER

Period: October 2014 – September 2017

1 Description

Being able to automatically analyze, model and predict social signals and social behaviors is

one of the major challenges of the social signal processing and affective computing communities. Up-

to-date, these research communities have focused on non-verbal behaviors such as facial expressions

or gesture and very few has been done on analyzing physiological signals of interacting persons

(imitation, synchrony). At the same time, investigations in neuroscience and psychology have moved

from the analysis of isolated individuals to the study of interactive contexts. This paradigm-shift is the

basis of the “two-body neuroscience” approach that has already highlighted interesting phenomena,

such as strong inter-brain synchrony during behavior imitation.

This thesis is concerned with the development of new computational models for modeling

such phenomena. It is built on the idea that there exist hidden variables that characterize inter-brain

synchrony. We propose to analyze and model them through latent variable techniques. A first line of

work will consider Non-negative Matrix Factorization (NMF) and multichannel extensions (e.g. tensor

factorization and co-factorisation schemes), as the latter hold the potential to allow for jointly

modeling signals relating to the interacting individuals, hence explicitly capturing cues relating to the

social interaction. In addition, these models should be able to grasp the temporal dynamics of the

interaction.

While the first phase of the thesis is concerned with EEG signals, multimodal extensions will

be also investigated such as the analysis of gestures of partners and/or external sounds for

synchronization.

Further, an important contribution of this thesis will be to confront the results, which will be

obtained by the proposed computational models with results obtained in neuroscience. We will not

only focus on the detection of events, but also we aim at improving the “explainability” or

interpretability of the developed models. Models such as Non-negative Matrix Factorization offer the

possibility to work on compact and, in some way, interpretable representations.

The challenge here is to develop models that capture not only the individual characteristics but

also the dyadic ones.

44

Figure 3 Non-negative Matrix Factorization applied to EEG signals

Various applications are envisaged such as imitation characterization in psycho-pathologies

(e.g. autism), emotion contagion phenomena during interactions and musical interactions. This thesis

being part of the SeNSE SMART project, these applications will be investigated consistently with the

rest of the project.

2 Results

The thesis started in October 2014. A first effort has been dedicated to the definition of a

protocol. We take inspiration from experimental protocols usually employed in neuroscience. Here,

the difficulties are numerous since we use wireless EEG sensors as well as free gestures.

Figure 4 Modeling inter-personal EEG dynamics by NMF

The global architecture is described figure 4. The key idea is to go beyond traditional methods

that compare low-level EEG data (e.g., phase of signals). We propose to compare models of brain

activities obtained by NMF. Two optimization schemes have been investigated so far: (1) comparing

individual NMF models (dictionaries and activations), (2) a co-factorization scheme allowing, during

learning, to explicitly take into account the other partners.

The current efforts are dedicated to the definition of several evaluation metrics ranging from

traditional ones in neuroscience (e.g., phase derived metrics) to machine learning (e.g., classification

rates and representation derived metrics).

FIGURE 5: Application of the NMF method in the EEG-related case

But how to determine W and H starting from the matrix V ? We will not be able

to write exactly V = WH , hence the necessity of agood approximation.

3.3 Which divergence to choose?

In order to chooseaproper approximation WH for V , wemust choosea"distance"

D (V ||WH ) that the couple (W, H ) minimizes. However, the name "distance" can be

misleading, since the chosen functions D (.|.) are not necessarily symmetrical. There-

fore, let us rather call them more generally cost functions. Given a matrix V 2 RF ⇥N+

and a number of words K , the factorization problem is equivalent to the minimization

problem :

minW 2 R

F ⇥ K+ ,H 2 R

K ⇥ N+

D (V |WH )

D being acost function such that

D (V |WH ) =

FX

f = 1

NX

n = 1

d([V ]f n |[WH ]f n ),

with d being ascalar cost function. Let us introduce three examples of d :

– The euclidean distance dE U C (x|y) =1

2(x − y)2

– The Kullback-Leibler divergence dK L (x|y) = x logx

y− x + y

– The Itakura-Saito divergence dI S (x|y) =x

y− log

x

y− 1

A classical way of evaluating the precision of the approximation is to calculate the

euclidean distance between V and WH . However, minimizing such a distance may

7

Non-Negative Matrix

Factorization A

Non-Negative Matrix

Factorization B

Co-factorization

Metrics in the space model

45

PhD Thesis Multimodal analysis and recognition of social signals: application to social stance generation in virtual agents PhD student: Janssoone Thomas

Supervisors: Richard Gael (LTCI), Clavel Chloé (LTCI), Bailly Kevin (ISIR)

Laboratories: LTCI Telecom ParisTech, ISIR - UPMC

Doctoral School: Edite

Period: Oct. 2014 - Sept. 2017

1 Description

Context - Objectives The aim of this thesis is to carry out research on the analysis of visual features (facial

expression and head movements, see Nicolle & Bailly, 2012) and audio features (linguistic and

prosodic, see Clavel & Richard, 2011) characterizing social stances, such as dominance (Burgoon,

1999) (Ravenet, Ochs & Pelachaud, 2013) . In particular, the PhD will study the various timing and

sequencing of the features coming from the different modalities. The long run goal is to integrate these

features in a model for the production of social stances in an Embodied Conversational Agents (ECA).

The PhD focuses on the analysis of socio-emotional signals (audio, video modalities). The

model developed through the PhD will be evaluated through realistic scenarios.

References:

Judee K Burgoon and Beth A Le Poire. Nonverbal cues and interpersonal judgments: Participant and observer

perceptions of intimacy, dominance, composure, and formality. Communication Monographs, 66(2):105–124,

1999.

Clavel, C., & Richard, G. (2011). Recognition of acoustic emotion. Emotion-Oriented Systems, 139-167.

Nicolle, J., Rapp, V., Bailly, K., Prevost, L., & Chetouani, M. (2012, October). Robust continuous prediction of

human emotions using multiscale dynamic cues. In Proceedings of the 14th ACM international conference on

Multimodal interaction (pp. 501-508). ACM.

B Ravenet, M Ochs, and C Pelachaud. From a User-Created Corpus of Virtual Agent’s Non-Verbal Behavior to

a Computational Model of Interpersonal Attitudes. In To appear in the proceedings of the Intelligent Virtual

Agents (IVA) conference, 2013.

R. Niewiadomski, E. Bevacqua, M. Mancini, and C. Pelachaud. Greta: an interactive expressive ECA system. In

Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2,

AAMAS ’09, 2009

46

PhD Thesis

Temporal adaptation of interaction PhD student: Kevin Sanlaville

Supervisors: Catherine Pelachaud (LTCI), Frédéric Bevilacqua (STMS), Gérard Assayag (LTCI)

Laboratory: LTCI and STMS

Doctoral School: EDITE

Period: 1/11/2013-31/10/2016

1 Description

Context - Objectives

In HCI, most interactions models deal with static architectures: the models are set by either a

fixed set of rules or through learning mechanisms in a fixed environment. Nevertheless, the interaction

occurring for example in a conversation features strong aspects of adaptation. The participants

constantly react and adapt their behaviors at short and long time scales, and can react rapidly to new

situations, using both verbal and non-verbal signals. Such phenomena are still poorly modeled by

current interaction models.

In this PhD project, the general aim is to develop a framework to model adaptive interaction at

different time scales. The ambition is to take into account phenomena such as synchronization

between participants’ behaviors as well as the emergence of new behaviors in reaction to original

situations. Two specific use cases will be considered 1) Embodied Conversational Agents (ECA)

interacting among themselves and with human users and 2) Musical Improvisation between computer

agents and musicians. We hypothesize that considering such different use cases could lead us to

formalize an interaction model that focuses on adaptation and synchronization mechanisms and will be

generic enough to be used in different settings.

2 Results

The first year has been essentially devoted to establishing a state of the art, covering several

areas of research in interaction: Embodied Conversational Agent, Musical Interaction and

improvisation, Interaction human-robots, turn-taking model, emergent model. From a modeling point

of view, different architectures have been reviewed such rule-based systems, probabilistic models or

autonomous agents.

The analysis of two particular systems has been initiated: the OMAx system that allows a

musician to improvise with an improvisational agent, and the GRETA systems that allows a user to

interact with an Embodied Conversational Agent. As shown below, both systems reveal interesting

similar architectures confirming that both related use cases could be treated with a generic adaptive

interaction model.

47

3 Publication

Sanlaville, F. Bevilacqua, C. Pelachaud,G. Assayag, Adaptation in an Interactive Model designed

for Human Conversation and Music Improvisation: a preparatory outline, Workshop Affect,

Compagnon Artificiel Interaction (WACAI’1), 2014, Rouen

Fig. 1 General ideal scheme of OMAx

Fig. 2 General Architecture of GRETA

Using these loops, the interacting agent is able to learn

not only from its interlocutor or fellow improviser but

also from its own abilities, leading it to amend its

representation in a more believable way, since it takes

into account both interactants.

Since these models are similar in their

conceptualization, we propose that our model be

conceived in a top-down approach; by reifying the

concepts at stakes in both models and their embodiment

in each field of research. Trough this approach will we

be able to model our goal of adaption of interaction.

6 Conclusion To perform the cooperative and communicative

interactions that are multimodal conversation and

musical improvisation, these interactions need to reach a

certain level of synchronization. To provide a believable

interaction, we aim to use emergent properties of the

interaction between our agents.

In this paper, we have defined what aspects of

adaptation that are relevant in our context. We showed

through use cases how we intend to exploit this

closeness in two specific contexts.

Acknowledgments

This work was performed within the Labex SMART

(ANR-11-LABX-65) supported by French state funds

managed by the ANR within the Investissements

d'Avenir programme under reference ANR-11-IDEX-

0004-02.

Bibliographie

[1]Allauzen, Cyril, Maxime Crochemore, and Mathieu

Raffinot. "Factor oracle: A new structure for pattern

matching." SOFSEM’99: Theory and Practice of

Informatics. Springer Berlin Heidelberg, 1999.

[2] Allwood, Jens. "Linguistic communication as action

and cooperation." Gothenburg monographs in

linguistics 2 (1976): 637-663.

[3] Assayag, Gérard, and Shlomo Dubnov. "Using

factor oracles for machine improvisation." Soft

Computing 8.9 (2004): 604-610. [4]Assayag, Gérard, and Georges Bloch. "Navigating

the oracle: A heuristic approach." International

Computer Music Conference. Vol. 7. 2007. [5] Baron-Cohen, Simon. "The evolution of a theory of

mind." The descent of mind: Psychological

perspectives on hominid evolution (1999): 261-277.

[6] Clair, Gael, Frédéric Armetta, and Salima Hassas.

"Self-adaptive tuning of dynamic changing problem

solving: a first step to endogenous control in Multi-

Agents Based Problem Solvers." ICAS 2011, The

Seventh International Conference on Autonomic and

Autonomous Systems. 2011.

[7]Donnay, Gabriel F., et al. "Neural Substrates of

Interactive Musical Improvisation: An fMRI Study

of ‘Trading Fours’ in Jazz." PloS one 9.2 (2014):

e88665.

[8] Fujie, Shinya, et al. "Spoken dialogue system using

prosody as para-linguistic information." Speech

Prosody 2004, International Conference. 2004. [9] Heylen, D., et al. "Why conversational agents do

what they do? Functional representations for

generating conversational agent behavior." The First

Functional Markup Language Workshop. Estoril,

Portugal. 2008.

[10]Ishi, Carlos Toshinori, Hiroshi Ishiguro, and

Norihiro Hagita. "Analysis of relationship between

head motion events and speech in dialogue

conversations." Speech Communication 57 (2014):

233-243. [11]Knapp, Mark, Judith Hall, and Terrence Horgan.

Nonverbal communication in human interaction.

Cengage Learning, 2013.

[12]Kopp, Stefan, et al. "Towards a common framework

for multimodal generation: The behavior markup

language." Intelligent virtual agents. Springer Berlin

Heidelberg, 2006.

[13] , Radosław, et al. "Cross-media agent platform."

Proceedings of the 16th International Conference on

3D Web Technology. ACM, 2011.

[14]Novielli, Nicole, Fiorella de Rosis, and Irene

Mazzotta. "User attitude towards an embodied

conversational agent: Effects of the interaction

136

Fig. 1 General ideal scheme of OMAx

Fig. 2 General Architecture of GRETA

Using these loops, the interacting agent is able to learn

not only from its interlocutor or fellow improviser but

also from its own abilities, leading it to amend its

representation in a more believable way, since it takes

into account both interactants.

Since these models are similar in their

conceptualization, we propose that our model be

conceived in a top-down approach; by reifying the

concepts at stakes in both models and their embodiment

in each field of research. Trough this approach will we

be able to model our goal of adaption of interaction.

6 Conclusion To perform the cooperative and communicative

interactions that are multimodal conversation and

musical improvisation, these interactions need to reach a

certain level of synchronization. To provide a believable

interaction, we aim to use emergent properties of the

interaction between our agents.

In this paper, we have defined what aspects of

adaptation that are relevant in our context. We showed

through use cases how we intend to exploit this

closeness in two specific contexts.

Acknowledgments

This work was performed within the Labex SMART

(ANR-11-LABX-65) supported by French state funds

managed by the ANR within the Investissements

d'Avenir programme under reference ANR-11-IDEX-

0004-02.

Bibliographie

[1]Allauzen, Cyril, Maxime Crochemore, and Mathieu

Raffinot. "Factor oracle: A new structure for pattern

matching." SOFSEM’99: Theory and Practice of

Informatics. Springer Berlin Heidelberg, 1999.

[2] Allwood, Jens. "Linguistic communication as action

and cooperation." Gothenburg monographs in

linguistics 2 (1976): 637-663.

[3] Assayag, Gérard, and Shlomo Dubnov. "Using

factor oracles for machine improvisation." Soft

Computing 8.9 (2004): 604-610. [4]Assayag, Gérard, and Georges Bloch. "Navigating

the oracle: A heuristic approach." International

Computer Music Conference. Vol. 7. 2007. [5] Baron-Cohen, Simon. "The evolution of a theory of

mind." The descent of mind: Psychological

perspectives on hominid evolution (1999): 261-277.

[6] Clair, Gael, Frédéric Armetta, and Salima Hassas.

"Self-adaptive tuning of dynamic changing problem

solving: a first step to endogenous control in Multi-

Agents Based Problem Solvers." ICAS 2011, The

Seventh International Conference on Autonomic and

Autonomous Systems. 2011.

[7]Donnay, Gabriel F., et al. "Neural Substrates of

Interactive Musical Improvisation: An fMRI Study

of ‘Trading Fours’ in Jazz." PloS one 9.2 (2014):

e88665.

[8] Fujie, Shinya, et al. "Spoken dialogue system using

prosody as para-linguistic information." Speech

Prosody 2004, International Conference. 2004. [9] Heylen, D., et al. "Why conversational agents do

what they do? Functional representations for

generating conversational agent behavior." The First

Functional Markup Language Workshop. Estoril,

Portugal. 2008.

[10]Ishi, Carlos Toshinori, Hiroshi Ishiguro, and

Norihiro Hagita. "Analysis of relationship between

head motion events and speech in dialogue

conversations." Speech Communication 57 (2014):

233-243. [11]Knapp, Mark, Judith Hall, and Terrence Horgan.

Nonverbal communication in human interaction.

Cengage Learning, 2013.

[12]Kopp, Stefan, et al. "Towards a common framework

for multimodal generation: The behavior markup

language." Intelligent virtual agents. Springer Berlin

Heidelberg, 2006.

[13] , Radosław, et al. "Cross-media agent platform."

Proceedings of the 16th International Conference on

3D Web Technology. ACM, 2011.

[14]Novielli, Nicole, Fiorella de Rosis, and Irene

Mazzotta. "User attitude towards an embodied

conversational agent: Effects of the interaction

136

48

PhD Thesis:

Embedded architecture and physiological sensors PhD student: Wenlu YANG

Supervisors: Maria Rifqi (LIP6), Christophe Marsala (LIP6), Andrea Pinna (LIP6)

Laboratory: LIP6

Doctoral School: EDITE (Ecole doctorale de l’UPMC)

Period: November 1st, 2014 – October 31st 2017

1 Description

The purpose of this thesis is to study emotions and learning in human-virtual agent

interactions. Extraction and characterization of social signals as well as adaptation over time are

employed to determine the emotional impact of sequences of video games. We particularly investigate

coaching sessions.

The originality of this thesis is the design of an embedded architecture implementing dynamic

machine learning techniques on physiological signals (EDA, EEG, ECG, EMG...) to automatically

recognize emotions. The objective is to obtain an architecture that reacts as closer as possible to a

particular person. For this, the machine-learning algorithm must automatically adapt to new

physiological data it acquires.

The automatic adaptation of the learning algorithm to these changes is an emerging problem

and challenges of the thesis are to design algorithms and architectures: embedded, effective execution,

speed and memory space; capable of integrating new descriptors, as well as new classes (new mental

states); capable of detecting abrupt changes or breaks without confusing them with noise; able to

follow developments and remaining robust and therefore knowing control oblivion.

The final aim of this thesis is to design a smart sensor with an embedded adaptive emotion

recognition algorithm.

In the future, we will exploit experimental data to tune our recognition method. The search of

relevant features will be also investigated but with another point of view: we will study the impact of

considering a symbolic representation of the signals. Finally, the electronic side of our project will be

developed. We will design an electronic sensor able to embed an adaptive recognition algorithm.

These challenges will be validated in several scenarios, such as video games, coaching,

professional training and events.

49

SMART-BAN

50

SMART-BAN Self-organizing, Mobility Aware, Reliable and Timely Body-Area-Networks

Responsible of the project: Julien Sarrazin

Partners:

L2E: A. Benlarbi-Delaï, J. Sarrazin

LIP6: S. Tixeuil

LTCI: C. Chaudet

Web site: http://www.smart-labex.fr/index.php?perma=SMART-BAN

1 The Project at a glance

The SMART-BAN project aims to optimize the energy consumption of wireless Body Area

Networks for medical applications while minimizing its impact on the human body. The project

gathers researchers from different fields ranging from

electromagnetics and communication theory to computer

science. By undertaking a trans-disciplinary approach,

fundamental energy limitations will be drawn and optimal

communication strategies for reliably routing and

aggregating data will be developed.

Context and Objectives

To improve the efficiency of the medical sector,

recent years have witnessed the emergence of wireless

Body Area Networks (BANs). BANs are sensor networks

that are embedded on the human body and provide useful

healthcare monitoring such as EEG, ECG, blood pressure...

The use of wireless technology to interconnect sensors

enables practical and seamless means to monitor patients

and thus can lead to more efficient management in hospital

or during mass-casualty disasters. As sensors become more and more miniaturized, BAN could be

worn permanently by people, thereby enabling continuous monitoring. One could then dream of

preventing striking decease. Continuous monitoring also represents a comfortable and effective

economic way of taking care of age-related illnesses.

However, to consider such a future, BAN will need to have a huge autonomy. That is why the

main goal of the SMART-BAN project is to find a way to reach the fundamental lower limit of power

consumption in BAN. This will be achieved by jointly taking into account the physics involved in the

wireless propagation around the human body and the dynamic distributed topology of the BAN sensor

network. This approach will enable the greatest autonomy as well as reducing the human exposure to

electromagnetic waves.

51

2 Scientific progress and results

A Body Area Network (BAN) is a challenging network in the sense that its topology is

dynamically changing over the time. In addition, constraints in terms of energy are drastic, taking into

account the limited room available on the sensors for the battery. Consequently, in order to one day

consider medical applications such as post-surgery monitoring for instance (lasting typically 2-3

weeks), it is important to decrease as much as possible the energy consumed by the BAN.

To do so, SMART-BAN is undertaking a transversal approach where the physical layer as

well as MAC and network layers are considered. In its initial phase, SMART-BAN is developing the

tools required to evaluate the influence of the variability of the BAN regarding the energy

consumption.

Regarding the physical layer, since sensors are

wirelessly communicating to each others, it is

necessary to understand the complex propagation

mechanism of electromagnetic waves around the

body. This task consists in both electromagnetics

modeling and radiofrequency measurements, and has

led to developing channel models that describe the

dynamic attenuation faced by a signal between a

transmitter and a receiver. These models are then

used in protocol layer software to assess different

routing algorithms and scenarios. From the

performed measurements, we can also assess the

performance in terms of data rate, QoS, and energy

consumption, of the physical layer of the IEEE

802.15.6 standard, which is a BAN-dedicated

standard for wireless communications. In particular,

we showed that using higher data rates in BAN,

which is typically more power consuming, can significantly reduce the amount of energy

required to transmit a given data.

Concerning the network layer, we began by looking at how to provide an efficient broadcast

given the WBAN specificities, in particular their mobility and channel characteristics:

WBANs use a radio medium for communication and alternate connection and disconnection

periods. This type of scenario is common in the delay tolerant networks environment and a

few algorithms for broadcast have been proposed in this context. In his master internship,

Federico Petruzzi modified the Omnet++ network simulator to evaluate the performance of

these algorithms and to propose relevant adaptations. He implemented the most relevant

proposals found in the literature and included to the simulator a dynamic channel model that

corresponds to the WBAN environment. Under this realistic simulation environment, he was

able to evaluate the success probability, the delay required to flood the whole network and the

required number of packets emissions. He also managed to extract some general results (e.g.

almost half of the time is spent reaching end nodes in the BAN) and to evoke algorithms

combinations to improve performance. On the theoretical side, Federico worked on time

varying graphs and on the adaptation of this formalism to the unreliability resulting from the

WBAN radio channel.

In Guy Landry Djatche Simo's master internship, we examined the implementation of auto-

configuration algorithms in an experimental platform composed of ST microeletronics'

Greennet nodes. Greennet nodes are narrowband sensors that are compact enough to be worn

by humans and could represent a good way to evaluate our algorithms in a real environment.

Guy Landry managed to set up the platform and to evaluate the performance of the dynamic

routing algorithm provided by ST microelectronics. These algorithms were designed and

Figure 5 - Wireless BAN communication measurements

52

tuned for a static sensor network scenario, and could require adaptation to work in a WBAN

environment. He therefore proposed a few adaptations to the algorithms to improve the

network reparation phase that occurs when wireless links are broken, for example due to

mobility.

Since the key feature of body area networks is to aggregate data collected by individual

sensors in an energy efficient manner, we studied the data aggregation problem from a

theoretical point of view. In fact, we can represent communication links in a body area

network as a sequence of static graphs, where edge appearance and disappearance represents

the connection and the disconnection between two sensor nodes. In this simple model, we

were able to define the data aggregation problem as the most energy efficient way to retrieve

the data from the sensor nodes to the coordinator.

Figure 6 - An example of data aggregation in a dynamic graph

In this context, we studied the impact of those constraints in the duration of data retrieval.

Firstly, we proved the first lower and upper bound on the duration of the data aggregation.

Secondly, we proved that the problem is NP-Hard i.e., intrinsically hard to resolve, even with

a very powerful computer. Finally, we gave the first approximation algorithm for the data

aggregation problem.

3 Future work

In 2014, SMART-BAN mainly focused on developing the required tools to evaluate the

influence of the variability of Body Area Networks in terms of energy consumption, from physical to

network layers.

In 2015, SMART-BAN will refine these tools as well as use them in order to assess a number

of different classic and original operating scenarios. By taking into account both hard and soft aspects,

both physics and network levels, we will truly be able to declare that a given strategy is less energy-

demanding than another. In fact, some techniques that are energy efficient at the scale of a single

wireless link can actually decrease the total efficiency of a whole network, hence the need of a global

assessment strategy.

Assessed techniques in 2015 will include data aggregation, multi-hop routing, dedicated

antenna design… These strategies will use the hypothesis found to be valid thanks to the work

conducted in 2014. This knowledge is indeed very important when it comes to global optimization.

4 Recruitment

- 1 PhD:

- Quentin BRAMAS (LIP6), “Self-organizing, Mobility aware, Reliable and Timely Body Area

Networks”, at LIP6, started in October 2013

- 1 Postdoc:

- Zhongkun MA (L2E), “Channel modeling in Body Area Networks (BAN)”, started in June

2014

- 3 Masters:

53

- Huiliang LIU (Tsinghua University, China), “Wireless Communications in Body Area

Networks”, February-July 2014

- Guy Landry DJATCHE SIMO (UPMC), “Gestion d’autonomie et churn dans GreenNet”,

March-August 2014

- Frederico PETRUZZI (Politecnico di Torino), “Models and Design of communication

protocols for WBAN and simulation on GreenNet platform”, April-September 2014

5 Publications

Journal paper:

Luca Petrillo, Theodoros Mavridis, Julien Sarrazin, Aziz Benlarbi-Delai, Philippe De

Doncker, “Statistical On-Body Measurement Results at 60 GHz”, IEEE Transactions on

Antennas and Propagation, DOI: 10.1109/TAP.2013.2287524, 2014

International conference:

Solofo Razafimahatratra, Julien Sarrazin, Philippe De Doncker, Aziz Benlarbi-Delai, "Horn

antenna design for BAN millimeter wave onbody communication", IEEE Antennas and

Propagation Symposium (APS), Memphis (USA), July 2014

Luca Petrillo, Theodoros Mavridis, Julien Sarrazin, Aziz Benlarbi-Delai, Philippe De

Doncker, “Experimental On-Body Shadowing on Torso at 60 GHz”, International

Conference on Body Area Networks (BodyNets), London, September 29 – October 1, 2014

Other conference:

Julien Sarrazin, Theodoros Mavridis, Luca Petrillo, Philippe De Doncker, Aziz Benlarbi-

Delai, “Antenna efficiency influence in Body Area Networks (BAN)”, International

Conference on Communication Systems (ICCS-2013), Pilani (India), 18-20 October 2013

(Invited talk)

54

6 Events

International Workshop on Green Solutions for Body Area Networks - GreenBAN

6-7 November 2014 at UPMC

The objective of the International Workshop on Green Solutions for Body Area Networks is to present

the latest developments in energy efficient Wireless Body Area Networks. It aims to bring researchers

working in the field of BAN with a focus on energy as well as in the field of power supply for such

networks. http://www.greenban2014.upmc.fr

56 participants, 25 speakers

During the workshop, SMART-BAN partners presented the following topics:

H. Liu, J. Sarrazin, F. Deshours, A. Benlarbi-Delaï, P. De Doncker, Z. Liu, “Performance

Evaluation on IR-UWB BAN with OOK Modulation”

Q. Bramas, S. Tixeuil, “The Complexity of Data Aggregation in Body Area Networks”

T. Mavridis, L. Petrillo, J. Sarrazin, A. Benlarbi-Delaï, P. De Doncker, “Polarization Impact

on 60 GHz Indoor Off-Body Communications”

C. Chaudet, F. Petruzzi, M. Potop-Butucaru, “Analyzing Various Broadcast Strategies in

WBAN”

Z. Ma, J. Sarrazin, L. Petrillo, T. Mavridis, P. De Doncker, A. Benlarbi-Delaï, “Antenna

Characterization for On-Body Communication Channel Using Creeping Wave Theory”

55

Phd Thesis: Self-organizing, Mobility Aware, Reliable and Timely Body-Area-Networks PhD student: Quentin Bramas

Supervisor(s): Sebastien Tixeuil

Laboratory: LIP6

Doctoral School: EDITE

Period: 01/10/2013 to 30/09/2016

1 Description

This thesis takes place within the SMART-BAN project, which aims to optimize the energy

consumption of wireless Body Area Networks for medical applications while minimizing its impact on

the human body. To do so, a trans-disciplinary approach will be undertaken. The impact of the

physical layer will be taken into account in MAC and Network layers to draw fundamental energy

limitations and to develop optimal communication strategies for reliably routing and aggregating data

in medical Body-Area-Networks.

The goal, for the Phd student, is to propose models of

medical Body-Area-Networks and strategies for routing and

information diffusion in Body-Area-Networks.

SYSTEM MODELING

Several measurement campaigns have been conducted in

various BAN projects, in order to evaluate the channel behavior

and evolution when an equipped user walks, runs, falls, etc.

These measurements are often realized on a point-to-point link

in a single scenario and the approach may fail in giving

sufficient insights related to what could be obtained through

multi-sensors on the same body… Our goal, in this task, is to provide a model of the network

topology and of its dynamicity with the wearer movement.

Based on the individual measurement campaigns realized in

partner projects, our aim is to create a generic and configurable

dynamic graph model that complies with all measurements and

represents the following dynamic aspects: α. dynamic/flexible topology β. temporarily unavailable links χ. links with variable reliability Two theoretical models for dynamic networks are natural candidates: a. Time-varying graphs b. Temporal reachability graphs

CASE STUDIES AND EVALUATION

In the evaluation part of our project we plan to consider two different scenarios. First, the remote

patient monitoring aims at collecting statistical. The pace is supposed to be low, and the main issue is

the saving of energy and the inference of patient activities without being intrusive. The second one

relies on a hospital infrastructure and is expected to trigger with extreme velocity life-threatening

alerts. The pace is supposed to be high, and the main issue is timeliness of alert reporting.

Figure 1. A body area network

56

2 Results

Since the key feature of body area networks is to aggregate data collected by individual

sensors in an energy efficient manner, we studied the data aggregation problem from a theoretical

point of view. In fact, we can represent communication links in a body area network as a sequence of

static graphs, where edge appearance and disappearance represents the connection and the

disconnection between two sensor nodes. In this simple model, we were able to define the data

aggregation problem as the most energy efficient way to retrieve the data from the sensor nodes to the

coordinator.

In this context, we

studied the impact of those

constraints in the duration of data

retrieval. Firstly, we proved the

first lower and upper bound on

the duration of the data

aggregation. Secondly, we

proved that the problem is NP-Hard i.e., intrinsically hard to resolve, even with a very powerful

computer. Finally, we gave the first approximation algorithm for the data aggregation problem.

3 Publications

The complexity of data aggregation in BANs, Quentin Bramas and Sébastien Tixeuil,

GreenBAN2014, 2014.

The complexity of data aggregation in static and dynamic wireless sensor networks, submitted

to Symposium on Theoretical Aspects of Computer Science (STACS)

Figure 2. An example of data aggregation in a dynamic graph

57

Post-Doc:

Channel modeling for body area networks Name: Zhongkun MA

Supervisor: Julien Sarrazin

Laboratory: L2E

Period: June 2014 – May 2015

1 Description

This post-doc takes place within the SMART-BAN project, which aims to optimize the energy

consumption of wireless Body Area Networks for medical applications while minimizing its impact on

the human body. The project gathers researchers from different fields ranging from electromagnetics

and communication theory to computer science. By undertaking a trans-disciplinary approach,

fundamental energy limitations will be drawn and optimal communication strategies for reliably

routing and aggregating data will be developed.

The goal of this post-doc is to analyze, characterize and model wireless communications

around the human body in the framework of Body Area Networks (BAN). In particular, on-Body

communications for medical applications are investigated. The job is to study the propagation channel

and the antenna’s influence when people are still or in motion. Developed models are then used in

network simulation software in order to determine consumed energy limitations and to develop

optimal communication strategies so that to increase BAN’s autonomy.

2 Results

Creeping wave theory was originally intended to be applied to a Hertzian dipole radiation around

the bending earth surface as demonstrated in Fig.1. It is re-visited for BAN (Body Area Network)

channel modeling, where the human body was modeled as cylinder (see Fig.2).

The formulation includes both the characteristics of antenna and human tissues, in which the field

density at a distance can be directly obtained by input power and antenna gain over infinitely large

PEC (perfect electric conductor) plane A drawback of this formulation approach is that it requires on-

body antenna gain measurement, which is very difficult in practice. To overcome this difficulty, we

successfully demonstrate the difficult and complicated on-body antenna gain measurement can be

substituted by measuring antenna gain above PEC plane to determine field density by simulation,

Fig.1. Creeping wave theory to model dipole radiation around the bending earth surface.

Fig.2. Model human body as cylinder.

58

which is much easier to be employed in practice. Taking advantage of the time gating technique, the

PEC plane employed in simulation and measurement does not have to be infinitely large. The obtained

results are plotted in Fig. 3 and Fig. 4 for PEC and dielectric cylinder cases, respectively. The

experiment validation is still under going to prove the above concept.

3 Publication

Z. Ma, J. Sarrazin, A. Benlarbi-Delaï, L. Petrillo, T. Mavridis and P. D. Doncker, “Antenna Radiation

Characterization for On-Body Communication Channel Using Creeping Wave Theory”, submitted to

European Conference on Antennas and Propagation (EUCAP 2015).

Fig.3. Analytical and simulated E-Field radiated by

monopole antenna around PEC cylinder

Fig.4. Analytical and simulated E-Field radiated by

monopole antenna around dielectric cylinder (relative permittivity 50 and conductivity 1.7 S/m)

59

SpinalCOM

60

SpinalCOM

Spinal Cord Imaging Responsible of the project: S. Feruglio (LIP6)

Partners:

LIP6: A. Alexandre, S. Feruglio, A. Pinna, F. Valette

LIB: H. Benali

Web site: http://www.smart-labex.fr/index.php?perma=SPINALCOM

1 The Project at a glance

Context and Objectives

The Spinal Cord (SC) is the input of sensory information and the output of the motor

commands of the limbs and trunk. Its damage by trauma, diseases of an inflammatory nature or

neurodegenerative can have major consequences, affecting the life quality and the life expectancy of

patients. With 28k new cases per year for the 49 countries of Europe and 12k in US1, it is a public

health problem.

Actual monitoring techniques (Magnetic Resonance Imaging - MRI or scanner) only provide structural

information of the spinal cord integrity (and no functional data). At the research level, functional MRI

(fMRI) can potentially provide information. However, it will be punctual (during the total

immobilization of the subject) and not chronic, under normal condition of life. Moreover, the temporal

resolution fMRI is modest and the inhomogeneous magnetic properties of the spine affect the

reliability and the repeatability of the measurement. Furthermore, the mechanical movements of SC

within the vertebra, related to the patient's breathing cycles, remain a source of noise and imprecision.

In addition to the fMRI approach, it is particularly appropriate to propose a system for the ambulatory

collection of the both metabolic (blood oxygenation) and electrophysiological (spinal nerves activity)

parameters. Such a development will strengthen the resources of the medical community, available

for:

1. Accurate identification of injured centers and tracking changes to make more effective the

regeneration process and/or the rehabilitation effort.

2. Understanding the role of SC, functioning in interaction with the brain and its degree of

independence in some motor "decisions" or sensory interpretation.

The monitoring of the SC blood supply and the measurement of the oxygenation of

Haemoglobin (Hb) are used to see these phenomena, to analyze their evolution, and to fight (from

pharmacological injections, surgery or appropriate rehabilitation) more effectively against the

degradation of SC. Monitoring of the nerve activity by electrodes implanted in SC also offers

interesting prospects.

The SpinalCOM project aims to investigate a new approach for the chronic imaging of SC

through the realization and the implementation of a multimodal telecommunicating implant (see Fig.

1) and its modelling. This approach will determine the extent of the activity of the spine, enabling the

production of a functional map of SC and the development of complementary tools to fMRI. As

shown in Fig. 2, it’s a multi-domains project of research, located at the intersection of various topics.

The implant will use the Diffuse Optical Imaging (DOI) principle and will be implemented using a

PhotoDetector (PD) and a minimum of 2 pulsed light sources at different wavelengths, with associated

electronic, for acquisition of information about the two forms of Hb (deoxy-haemoglobin and oxy-

haemoglobin) in real-time. This first embedded system could be coupled with a tailor-made

1 NIDRR, Office of Special Education and Rehabilitative Services, U.S. Department of Education, Washington, DC.

61

Fig. 3 – Methodology.

instrumentation to acquire bio-potentials, due to the SC electrical activity (ESG - ElectroSpinoGram).

After processing, obtained data will be transmitted by radiofrequency outwardly of the body to be

operated with particular developed neurological models.

To establish the proof of concept for the chronic imaging of SC, 4 main technical challenges are

addressed by this project:

1. Reduction of Power Consumption (PC) for an acceptable Signal-to-Noise Ratio (SNR) and a

minimal size for the implant.

2. Extraction of data relative to background noise and artifacts.

3. Minimize the harmful effects of biocompatible encapsulation and the implant connectivity on data

recovery and exploitation.

4. Exploitation of the data in order to correlate them with those obtained by fMRI.

Fig. 1 – Illustration of the SpinalCOM project. Fig. 2 – Scientific fields associated to the

project.

These points will be solved, by the study of the best tradeoff between PC, congestion, algorithm and

architecture of the application in this harsh environment. In Fig. 3,

the employed methodology is presented, where we have a high

interdependence between modeling, prototyping and measurement.

The multi-physics modeling of the system in its environment is

mainly developed to minimize the animal experiments, but also to

optimize the system performances by its virtual prototyping.

In this kind of applied research, in vivo experimentations are

inevitable to valid, in real conditions, various aspects related to

mechanical, anatomical and physiological constraints, artifact, etc.

and also to obtain data that are not available in literature. The pig has

been chosen for our in vivo experiences, because this is one of the

animals nearest human. These experiments are subcontracted by the

highly experienced veterinarians of the XP-MED society (INRA,

Jouy-en-Josas) with the active participation of the consortium

members.

62

Table 2 – Gantt diagram of the project.

2 Scientific progress and results

SpinalCOM is a two years project. The updated Gantt diagram is shown in Table 2.

The project began with a state of art on this kind of research and the related topics, to impose

specifications of the

implant (task 1).

Concerning the task

2, we have focused on

the opto-electronic

part. A study has been

done to find the best

tradeoff between

SNR, response time

and PC. In parallel,

various new

prototypes have been

developed, from

discrete components

(to reduce cost and

development time),

and are still in

progress. It may be

noted that new king of optical sources, attractive for very low PC, have been found, for example.

Moreover, the use of an original multispectral PD (see Fig 4), rather than conventional photodiode,

seems to be the best approach. In Fig. 5, some prototypes are presented. Study and realization of

analog and digital signal processing to improve SNR with very low PC has been also done and a

Labview interface software has been realized for command and visualization of results in real-time

(see Fig. 6). ESG has been set-aside for the moment. Indeed, the impact factor of the DOI RF system

only on SC is already strong for the community and need time for good prototyping. Moreover, good

ESG acquisition needs of particular electrodes. Thus, this point will be treated at the end of the project.

Furthermore, an ANR project has been submitted in October 2014, where one of partners, (ESYCOM

laboratory, ESIEE, Univ. Paris Est) will develop tailor-made electrodes.

Task 3 concerns the modeling of the heterogeneous system. After cutting of the system in elementary

blocks, a model of each of them has been produced. For PDs and light sources, it is an opto-electrical

behavior model that has been realized in VHDL-AMS language. Concerning SC and the vertebrae

bone, a model, based on the Beer-Lambert law, has been implemented. For purely electrical

components, SPICE-like model is employed if possible or otherwise created. Then, these blocks are

associated to realize the system architecture. Some improvements are actually in progress.

Many experiments (task 4) have been done. Firstly, all prototypes are characterized in the usual

conditions, then in the case of digital pulsed oximeter and with an elementary in vitro tester. For this

a)

b)

c)

Fig. 4 – BQJ PD spectral responses versus

wavelength.

Fig. 5 – Examples of prototypes: a) Encapsulated receiver, b) Board

for commands and signal processing, c) RF DOI prototype.

63

latter, choice and purchase of specific equipment has been done and a mixture to emulate the optical

properties of the SC has been chosen. Concerning the in vivo experiments, protocols has been written

and some experiences have been realized, including the optical characterization of vertebra bones of

pig and the optical study of in vivo pig vertebra with its SC (see Fig. 7) and began to be treated (first

step of task 5).

Besides discussions for the ANR project submissions with other academic laboratories,

doctors and veterinarians, we are in contact with companies and SATT-LUTECH (Structure

specialized in transfer and commercialization of innovative technologies).

Fig. 6 – Software interface.

3 Future Work

In 2015, the SpinalCOM partners will focus on (1) the finalization of optoelectronic

prototypes and (2) in vitro and in vivo experiments, in order to retrieve information about the

hemodynamic changes in the spinal cord. In addition, modeling of the biological environment will be

performed. A master-2 trainee, mainly recruited by LIB and he/she will focus on the modeling phase

combined with models already developed at LIP6. Olivier Tsiakaka, a new Phd student newly

recruited in October 2014 (PhD research grant EDITE of Paris, MENRT) will cover part of those 2

points.

In addition, a national project (ANR: sub-priority 13 Health Technologies, challenge of 4 -

Life, Health and well-being) has been submitted. SpinalCOM partners (LIB, LIP6) compose the

consortium as well as ETIS (UMR8051), ESICOM (EA2552), INL (UMR5270, in association with

LN2 Sherbrooke UMI-3463), and the orthopedics department at Pitié Salpêtrière (AP-HP).

If accepted, we can go further in the study of the spinal cord by the investigating a potential transfer to

the human beings. We will also develop integrated circuits and electrodes dedicated for our

application. Finally, we plan to participate to conferences and submit journal papers as well as

a)

b)

c) Fig. 7 – Examples of in vitro and in vivo experiments: a) Optical transmittance measurement of vertebra bone of pig, b) 3D

imaging of the pig SC and c) Normalized optical response versus wavelength of various biological materials.

64

finalizing discussions with industrial contacts.

4 Recruitment

1 thesis: « Design of telecommunicating neurological implant », O. Tsiakaka, Doctoral school

PhD scholarship (PhD research grant EDITE de Paris), from October 2014.

1 study’s Engineer: « Electronic Instrumentation Engineer for Biomedical Application », M.

Feher (April – Sept. 2014).

6 trainees:

1. « Contribution to the modelling of an implant for the chronic imaging of the spinal cord », R.

Ghanem, Master 2 SESI, UPMC-Paris 6 (April – Sept. 2014).

2. « Contribution to the realization of an implant for the chronic spinal cord imaging », O. Tsiakaka,

Master 2 SESI, UPMC-Paris 6 (April – Sept. 2014).

3. « Digital implant for the functional imaging of the spinal Cord », H. Saadi, Master 2 SESI, UPMC-

Paris 6 (April – Sept. 2014).

4. « Design of multi-wavelength source », M. Vallée, Master 1 SESI, UPMC-Paris 6 (April 2014).

5. « Modelling of a communicating neurological implant in its environment to the multimodal

imaging of the spinal cord », D. El Azzi, Master 2 SDI, UPMC-Paris 6 (April – Sept. 2013).

6. « Optical modelling of CMOS Photodetector », A. Karami, Master 1 SESI, UPMC-Paris 6 (July

2013).

5 Publications

S. Feruglio, T. Courcier, A. Karami, A. Alexandre-Gauthier, O. Romain, V. Aimez, P.G. Charette,

P. Pittet, G.N. Lu.: « Opto-electrical Modeling of CMOS Buried Quad Junction Photodetector »,

IC-MAST, Prague, Czech Republic, 15-17 September 2013. Proc. published in Key Engineering

Materials, Trans Tech Publications, Switzerland, Vol. 605, pp. 470-473, 2014.

6 Events

“Fêtes de la Science” 2013 and 2014

4 seminars in Master 2 SESI in relation with the project

65

SMART Actions

66

67

PhD Theses

68

69

Phd Thesis

Complex Networks Dynamics for the Study of the Structure-

Function Relationship in the Human Brain.

PhD student: GARNIER Aurélie

Supervisor(s): BENALI Habib (LIB), VIDAL Alexandre (LIB), FRANÇOISE Jean-Pierre

(Laboratoire Jacques-Louis Lions)

Laboratory: Laboratoire d’Imagerie Biomédicale (LIB), Sorbonne Universités, UPMC Univ Paris 06,

INSERM U1146, CNRS UMR 7371

Doctoral School: EDITE (École Doctorale Informatique Télécommunication et Électronique)

Period: 01/12/2012-30/11/2015

1 Description

Context - Objectives

One of the current objectives in neurosciences is to elaborate complex networks integrated models

from multi-scale data to study the relationship between structure and function in the human brain.

Classically, these models are of high dimension, complex and with many parameters and therefore

hard to use. In this work we are interested on a network model developed for fMRI data decryption.

We focus on this network model theoretical and numerical mathematical analysis at different scales.

Usually, specific mathematical tools are used on such analysis such as bifurcation theory, network

dynamics or singular perturbation theory [1]. This analysis could help understand dynamics

underlying the cortical system, estimate biological parameters hard to quantify experimentally and

infer experimentally non-identified behaviors.

Till now we have considered the model at voxel scale (network node) to understand the local model

generated dynamics and possibly to improve the control of its outputs by reducing its dimension.

These analyses will help facilitating the network model analysis. This analysis aims to determine

whether networks functionally separated are also anatomically separated and conversely.

[1]: L. Arnold, C.K.R.T. Jones, K. Mischaikow and G. Raugel. Dynamical systems. Springer Berlin

Heidelberg, 1995.

2 Results

We first consider the voxel scale with a neuro-glio-vascular model representing interactions between

neurons, neurotransmitters and hemodynamic variables (such as blood flow or BOLD signal) and

generating the corresponding activities

Till now, the work focuses on the neural compartment. We extend a well-known neural mass model

(Jansen-Rit model [2]) by adding a direct feedback on the pyramidal cells population (Figure 1). We

provide an exhaustive co-dimension 2-bifurcation analysis (according to two specific parameters: the

main model input and the main coupling gain parameters) of this extended model providing a glossary

of the time series the model is able to generate. We also study the impact of the balance between the

main population direct and indirect feedbacks by varying suitable parameters (direct and indirect

coupling gain parameters). We compute a partition of this parameter space based on the co-dimension

2 bifurcation diagrams (Figure 2). This partition also represents the distribution of the five time series

70

the model is able to generate allowing to estimate which direct and indirect coupling gain parameters

range is needed to generate a specific time series.

We applied this work by generating a time series representing an experimental Local Field Potential

(LFP) from epileptic mouse. The partition was useful for this application. Indeed, the LFP displayed a

change in its regime corresponding to a switch between to bifurcation diagrams. Thus we used the

partition to establish the way to change parameter values to generate a time series comparable to the

experimental one.

Figure 1 [3] Figure 2 [3]

[2]: B.H. Jansen and V.G. Rit. Electroencephalogram and visual evoked potential generation in a

mathematical model of coupled cortical columns. Biological Cybernetics, 73(4): 357-366, 1995

[3]: A. Garnier, A. Vidal, C. Huneau, H. Benali. A neural mass model with direct and indirect

excitatory feedback loops: identification of bifurcations and temporal dynamics. To be published in

Neural Computation.

3 Publications

Garnier, C. Huneau, A. Vidal, F. Wendling, H. Benali. Identification of dynamical behaviors

in epileptic discharges using a neural mass model with double excitatory feedbacks.

Proceedings of ICCSA 2014: Normandie University, Le Havre, France, 205-210.

Garnier, A. Vidal, C. Huneau, H. Benali. A neural mass model with direct and indirect

excitatory feedback loops: identification of bifurcations and temporal dynamics. To be

published in Neural Computation.

71

PhD Thesis Autonomic Distributed Environments for Mobility PhD student: Mohamed Hamza Kaaouachi

Supervisor(s): Franck Petit (LIP6) and François Jouen (Chart-Lutin)

Laboratory: LIP6, Chart-LUTIN

Doctoral School: EDITE

Period: 01/10/2012 to 30/09/2015

1 Description

The explosion of the number of communicating objects leads the distributed computing community to

consider new models of distributed systems that must take in account, on the one hand, the scale of

these new networks, and on the other hand, their high dynamic. Dynamic means that the network may

be always disconnected in the worst case since end-to-end path between two processes does not

necessarily exist at any time. However communication may be possible over time in such networks.

Several approaches to handle such dynamic have been proposed in the last two decades. The authors in

[CFQS11] propose a model that aims at gathering most of the existing models, so called time varying

graph (TVG).

Network protocols often rely on the construction of distributed spanning structures such as trees,

coloring, matching, minimal dominating sets, etc. In a (static) graph, a dominating set is a subset of its

vertices such that each vertex of the graph is either in the dominating set or has a neighbor in the

dominating set. A minimal dominating set (MDS) is a dominating set such that none of its strict subset

is also a dominating set. One of classical application of MDS in network protocols is hierarchical

routing. Indeed, an MDS can be used to split the network processes into two sets. Dominating

processes are designated to be cluster-head and act as routers. Other processes communicate

exclusively with their dominating neighbors. This hierarchy allows reducing communication costs.

The MDS problem was well studied in the literature. We can observe that the definition of dominating

set was extended in different ways when considering dynamic environments. For instance, some

authors [WddACG12] propose the evolving dominating set, where each change in the dynamic graph

leads to a new independent (static) graph for which a new dominating set must be computed. This

approach makes sense only when topological change frequency is low since each topological change

implies a new dominating set computation.

In this thesis, we study the MDS problem in highly dynamic networks. In such networks, there does

not exist necessarily an end-to-end path between each couple of processes at any time. Therefore,

communications mainly occur over time. This model is the more general (when the lifetime of the

network is infinite) among the ones proposed in the hierarchy of TVGs in [CFQS11] since it only

requires the dynamic graph to be connected over time. This class of TVGs is denoted C5 in [CFQS11]. More precisely, our goal is to characterize under which conditions it is impossible to cope

autonomously with unpredictable topological changes to solve the problem of MDS in C5. These

conditions must be proved necessary and sufficient (that is, we must provide an algorithm solving the

problem when the conditions are not satisfied). Our second goal is to perform a similar study for fault-

tolerant algorithms in C5.

72

2 Results

Since none of the definition of dominating set that we found in the literature is suitable to our

assumptions, our first contribution in this thesis is to extend the specification of the MDS problem for

TVGs. Then, we define a dominating set over time of a TVG as a subset of its vertices that infinitely

often dominate other vertices.

In previous work, the complexity measures in the TVG model are restricted to message complexity or

to time complexity in specific class of TVGs in which a classical notion of complexity naturally

extends. However, we need a time complexity measure that makes sense in any TVG class. In

particular, this measure must capture the quality of an algorithm independently of delays introduced

by asynchronous communications but also by topological changes. A typical example of such a delay

is the waiting after next apparition of an incident edge to a disconnected process that may introduce a

long delay that is not imputable to the algorithm but only to the dynamicity of the system. To perform

this goal, we propose in this thesis a new complexity measure that extends the classical notion of time

complexity in asynchronous message passing (static) systems.

Our study on the feasibility of a MDS over time construction in C5 leads us to prove that each

computable MDS has the specific property to be a MDS of all connected spanning subgraph of the

underlying graph of the TVG (the static graph that gathers all edges that appear at least once during

the lifetime of the TVG). We say that such MDSs are strong. The next contribution of this thesis is to

prove that there exists no algorithm that constructs a MDS over time in C5 whose underlying graph

does not admit at least one strong MDS. In the contrary case, we provide an algorithm that computes

an MDS over time. This algorithm is proved to be time optimal using our measure of complexity

introduced above.

Finally, we interest in a fault-tolerant solution for MDS over time construction. We consider here self-

stabilization (i.e. transient fault tolerance). For the first phase of this study, we restrict ourselves to

specific topologies. We propose a self-stabilizing solution to our problem for any TGV in C5 whose

underlying graph is a bipartite graph.

References

[CFQS11] Arnaud Casteigts, Paola Flocchini, Walter Quattrociocchi, and Nicola Santoro. Time-varying graphs and dynamic networks. In Ad-hoc, Mobile, and Wireless Networks, pages 346–359. Springer, 2011.

[WDdACG12] John Whitbeck, Marcelo Dias de Amorim, Vania Conan, and Jean-Loup Guillaume. Temporal reach- ability graphs. In Proceedings of the 18th annual international conference on Mobile computing and networking, pages 377–388. ACM, 2012.

3 Publications

N. Braud-Santoni, S. Dubois, M. Kaaouachi, F. Petit. The next 700 impossibility results in

TVG. To be submitted.

S. Dubois, M. Kaaouachi, F. Petit. Minimal dominating set for time-varying graphs. To be

submitted.

73

PhD Thesis

Automatic acquisition of sensori-motor concepts for a robot

PhD student: Carlos MAESTRE

Supervisor(s): Stéphane Doncieux (ISIR), Christophe Gonzales (LIP6)

Laboratory: ISIR, LIP6

Doctoral School: EDITE

Period: 2014-2017

1 Description

Context

In our daily life we perform simple tasks that a robot could execute: to tidy, to clean, to buy groceries,

etc. Some of these tasks can be clearly defined without any ambiguity, as to vacuum up some dirt in an

uncrowded space. There is a big gap between the autonomous robots, dedicated to a specific task, and

the assistance robots, able to tidy up and clean an apartment, to use a dishwasher, to fold the laundry,

etc. They must have much more complex sensori-motor skills that the vacuum cleaner robots. Their

behavioral directory must be rich, what raises many behavioral questions, such as how to build this

complex directory. But it also raises questions related to the human-robot interactions, such as how to

provide a task to the robot in order to be sure that it can accomplish it, or how to help the robot to

perform the task if needed.

Therefore, the assistance robots must execute complex tasks in a continuous space of big

dimensionality, gathering information from their sensors and defining concepts (perceptions, actions,

behaviors, but also more abstract concepts) to feed their planning system and task solver. In order to

make this possible, it is necessary to reduce the dimensionality of the space working in discretized

spaces, to be able to define an affordable planning for a specific task.

The main goal of this thesis is the autonomous creation by a robot of a minimal directory of concepts

related to its morphology, its environment, and the accomplished tasks executed. The robot must be

able to build and update its internal world model through interactions with its environment (cognitive

bootstrapping). This work follows the approach proposed in developmental robotics, inspired by the

development of the infants, where abstract concepts are created progressively based on the sensori-

motor capabilities of an agent. Executed in an iterative loop, a dataset is created by the robot through

the babbling of its environment; some candidate world models are defined based on the data gathered;

and a new dataset is created, in order to discriminate these models, improve them and generate new

simpler ones.

Objectives

1. Generation of datasets containing information about the functioning of the agent's

environment. This data is gathered by the agent through the babbling of its environment using

its sensori-motor schema. The exploration is driven by the intrinsic motivations of the agent in

order to improve its model of the world. These motivations are based on the search for novel

behaviors, which relies on evolutionary computation algorithms. An external agent (a human

being) can also help to guide the exploration.

74

2. Creation of a model of the world. Once the datasets are created these are analyzed to build,

using statistical learning, non-stationary dynamic Bayesian networks. These networks provide

predictions about the possible actions to be performed by the robot.

3. Abstraction of the learned concepts. The previously learned Bayesian networks will be

transformed into Object Oriented Probabilistic Relational Models (OOPRM), enabling to

model “complex” worlds by generalizing that was learnt on simpler ones, similar to the

abstraction of concepts.

75

PhD Thesis Flexible Queries for Smart Information Extraction

PhD student: Sébastien Lefort

Supervisor(s): Marie-Jeanne Lesot (LIP6, UPMC), Elisabetta Zibetti (CHArt, Paris 8)

Laboratory: LIP6

Doctoral School: EDITE

Period: October 2014 – September 2017

1 Description

The aim of the thesis is to define natural interaction methods between humans and intelligent systems:

it combines issues from cognitive psychology, machine learning and data base querying, so as to

produce a triple conceptualization of natural language use to express queries: user personalization,

adaptation to the data base content and cognitive adjustment. The thesis will formalize the notions of

interpretability and linguistic imprecision, in particular for temporal and spatial terms, and propose

methods to build formal representations of vocabularies, taking into account the formalized constraints

and considering the use of fuzzy approaches. Methods to automatically adapt a vocabulary will be

proposed and exploited in the applicative context of collaborative querying, to process issues related to

plethoric or empty answers.

2 Results

In the two months since the beginning of the thesis, a bibliographic study of the cognitive

representations of space, time and quantities, and their relationships in human cognition has been

started. This work is currently being extended in the directions of mental scales used to represent

quantities and methods to affect an interval of possible values denoted by a numerical expression

according to its precision (e.g., 2 hours: [1h40; 2h20]).

76

77

Post-Doctoral Program

78

79

Post-Doc

New approaches for the evaluation and the treatment of emotional

disorders –

Virtual Reality, multisensory integration and affect Name: Marine TAFFOU

Supervisor: Isabelle VIAUD-DELMON (STMS)

Laboratory:

Sciences et Technologies de la Musique et du Son, STMS UMR9912 Institut de Recherche et

Coordination Acoustique/Musique

Period: 01/01/2015 – 31/12/2015

1 Description

Phobic disorders are emotional disorders, which are characterized by an intense and uncontrollable

fear in response to a specific object or situation. Phobic individuals tend to avoid confrontation with

the fear object. This is often challenging in daily life and could lead to social isolation in the long

term. Exposure therapy is an efficient treatment for phobias. This therapy consists in a progressive

confrontation of patients to the situation they fear with the aim of developing habituation. Yet, the

exposition of patients to real situations, in vivo, is not completely controllable and thus not much

reassuring. Over the past twenty years, a new type of exposure therapies has emerged: exposure

therapies with virtual reality (VR).

Exposure in VR offers many advantages for the treatment of emotional disorders (North, North, &

Coble, 1998). VR allows for the exposure of patients to feared stimuli, which are complex, dynamic,

interactive and in 3D. The feared stimuli or situations are totally controlled, preventing unpredicted

events from interfering with treatment. Situations can also be repeated and the exposure intensity

manipulated, enabling the establishment of a treatment plan and its realization in total safety for the

patient. Successful outcome of exposure therapy in VR has been found for several specific phobias as,

for example, arachnophobia (e.g. Carlin, Hoffman, & Weghorst, 1997; Garcia-Palacios, Hoffman,

Carlin, Furness, & Botella, 2002 - see Figure 1 for an example of a virtual environment for the

treatment of arachnophobia).

In a natural environment, emotional information is perceived via multiple senses (vision,

audition…). Yet, the effect of VR involving multisensory stimulation on exposure therapy remains

undiscovered and VR applications generally underexploit the auditory sensory modality. Information

Figure 1: Example of a virtual

environment for the treatment

of arachnophobia

80

coming from each sensory modality can influence emotional perception, feeling and behaviors.

Auditory stimulations increase immersion and the feeling of presence in a virtual environment (VE)

(Hendrix & Barfield, 1996). Moreover, in many phobias, the auditory component of the feared

situations conveys emotional information. The use of multisensory stimulation allows understanding

how information coming from the different sensory modalities combines to induce affective reactions

in a VE.

While affective processing has mostly been studied in one sensory modality at a time, studies have

recently started to explore the processing of multisensory emotional stimuli. VR techniques represent

an ideal tool to investigate this question given that they allow the experimental manipulation and

control of the different sensory inputs delivered to the subject as well as the presentation of stimuli in a

more ecological manner, embedded within a natural and significant context. Recently, a study has

used VR to present emotional stimuli with the objective of investigating the influence of multisensory

stimuli on emotional feeling (Taffou, Guerchouche, Drettakis, & Viaud-Delmon, 2013). They used

virtual environments involving both visual and auditory stimulations of high quality and in 3D and

showed that auditory-visual aversive stimuli induce a more intense fear than only-visual or only-

auditory stimuli. This finding suggests that sensory presentation of emotional stimuli influences

emotional feeling and has thus implications for the field of exposure therapy in VR. Manipulating the

sensory characteristics of feared stimuli presented in an auditory-visual VE could allow modulating

the affective reactions induced in the patients and thus enable the control of exposure progression

during treatment in VR.

The neural basis of integration of emotional information coming from different sensory modalities

can be studied with dynamic brain imagery techniques such as the electro-encephalography (EEG) and

the magneto-encephalography (MEG). Whereas several studies have started to explore the time-course

of the processing of multisensory emotional stimuli (De Gelder, Böcker, Tuomainen, Hensen, &

Vroomen, 1999; Hagan et al., 2009; Pourtois, De Gelder, Vroomen, Rossion, & Crommelinck, 2000;

Pourtois, Debatisse, Despland, & De Gelder, 2002), the stimuli used have remained mostly static and

not much ecological (pictures of faces coupled with sound of voices). The findings of these studies

suggest an early combination of visual and auditory emotional information (from 110ms post-

stimulus). But, what happens at the cerebral level when exposed to more ecological multisensory

stimuli, such as the stimuli presented during an immersion in VR?

This project aims at investigating, with virtual reality, the influence of the sensory presentation of

emotional stimuli on affect, from early stages of affective processing to later stages of emotional

feeling. The findings will provide important information for the development of virtual environments

that will allow the mastering of the user’s affective reactions in terms of sensory presentation

parameters and thus could help refine virtual reality-based therapies for emotional disorders.

References Carlin, A. S., Hoffman, H. G., & Weghorst, S. (1997). Virtual reality and tactile augmentation in the treatment of spider phobia: a case report. Behavior Research

and Therapy, 35, 153-158.

De Gelder, B., Böcker, K. B., Tuomainen, J., Hensen, M., & Vroomen, J. (1999). The combined perception of emotion from voice and face: early interaction

revealed by human electric brain responses. Neuroscience letters, 260(2), 133–6.

García-Palacios, A., Hoffman, H. G., Carlin, A., Furness, T., & Botella, C. (2002). Virtual reality in the treatment of spider phobia: A controlled study.

Behaviour Research and Therapy, 40, 983–993.

Hendrix, C. & Barfield, W. (1996). The sense of presence within auditory virtual environments, Presence: Teleoperators and Virtual Environments, 3, 290-301.

Hagan, C. C., Woods, W., Johnson, S., Calder, A., Green, G. G. R., & Young, A. W. (2009). MEG demonstrates a supra-additive response to facial and vocal

emotion in the right superior temporal sulcus. Proceedings of the National Academy of Sciences of the United States of America, 106(47), 20010–5.

doi:10.1073/pnas.0905792106

North, M. M., North, S. M., & Coble, J. R. (1998). Virtual reality therapy: An effective treatment for phobias. Studies in health technology and informatics, 58,

112-119.

Pourtois, G., De Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective

information. Neuroreport, 11(6), 1329–33.

Pourtois, G., Debatisse, D., Despland, P.-A., & De Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. Brain

research. Cognitive brain research, 14(1), 99–105.

Taffou, M., Guerchouche, R., Drettakis, G., & Viaud-Delmon, I. (2013). Auditory–Visual Aversive Stimuli Modulate the Conscious Experience of Fear.

Multisensory Research, 26, 347–370. doi:10.1163/22134808-00002424

81

Visiting Professors Program

82

Jan Babic Electrical Engineering Department of Ljubljana University in Slovenia

Jan Babic is an associate professor at the Jožef Stefan Institute and a member of the

Electrical Engineering Department of Ljubljana University in Slovenia. His research

deals with human motor control. He is interested in understanding sensori-motor

learning and adaptation mechanisms in whole-body activities, both from the

Neuroscience and Robotics point of view. These research topics are very directly

related to the SMART Labex research programs "Modeling humans", "Interfaces and

Interaction with humans" and "Human autonomy and e-health". His coming at UPMC and ISIR is a

very good opportunity to build up collaboration and synergies in these domains."

Personal web page: http://www.ijs.si/~jbabic/

Contact: Vincent Padois (ISIR)

Period: November 2014

Nathan Ida (University of Akron)

Nathan Ida is currently a Distinguished Professor of Electrical and Computer

Engineering at the University of Akron, OH. He teaches electromagnetics,

antenna theory, electromagnetic compatibility, sensing and actuation, and

computational methods and algorithms. His current research interests include

numerical modeling of electromagnetic fields, electromagnetic wave

propagation, theoretical issues in computation, nondestructive testing of

materials at low and microwave frequencies as well as in communications,

especially in low-power remote control and wireless sensing. He has

published extensively on electromagnetic field computation, parallel, and

vector algorithms and computation, nondestructive testing of materials,

surface impedance boundary conditions, sensors and others. He is the author or coauthor of six books.

Dr. Ida is a Fellow of the Institute of Electrical and Electronics Engineers (IEEE), of the Institute of

Engineering and Technology (IET), of the American Society of Nondestructive Testing (ASNT) and

of the Applied Computational Electromagnetics Society (ACES).

Personal web page: http://ee.ascs3.uakron.edu/ida/

Contact: Aziz Benlarbi-Delai et Zhuoxiang Ren (L2E)

Period: November-December 2014

83

Hannah Michalska (McGill University)

Hannah Michalska is Associate Professor in the Department of Electrical and

Computer Engineering at McGill University, Montréal, Canada. Prof.

Michalska current interests include the feedback control design for strongly

nonlinear systems, differential geometric control, Hamiltonian systems and

symplectic integration, nonlinear control of robotic systems, identification and

control of time-delayed systems, and nonlinear control and identification in

biological systems. Prof. Michalska is collaborating with Vincent Hayward and

Alain Berthoz on the nonlinear observation from idiothetic measurements of

locomoting systems in non-inertial frames and on multi-variate models of human perception. Prof.

Michalska’s well-known contributions include Robust Receding Horizon Control and Lie Algebraic

time-varying stabilizing controls for nonlinear systems.

Personal web page: http://www.cim.mcgill.ca/~michalsk/

Contact: Vincent Hayward (ISIR)

Period: August 2014

84

85

SMART Perspectives

86

87

1 Perspectives of current SMART actions

SMART Labex is a long-term project (2012-2019). The SMART agenda is organized through calls for

internships, PhD thesis, Post-Docs, Invited Visiting Professors and Projects.

Six projects were selected during the first call in 2013 (T0: Sept. 2013). EDHHI (12 months) ended in

September 2014. The other projects are still running: ISMES (48 months), Onbul (42 months), SeNSE

(48 months), SMART-BAN (18 months project), and SpinalCOM (24 months). All these projects have

recruited: 11 PhD students of which 7 were directly financed by SMART, 3 post-docs of which 2 were

directly financed by SMART, 2 engineers and a physiotherapist. Other part-time non-permanent

members (Phd students) are taking part of the research activities of these projects. The projects have

recruited 23 master students of which 19 were directly financed by SMART. Other recruitments are

planned next year.

SMART agenda

2 Future SMART actions

SMART will support research projects (over a period of 1 to 4 years) in the scientific areas of our

Labex. After the first call for projects in 2013 (6 projects have been selected), a new call for research

projects is planned in 2015. In addition to the recommendations of the External Scientific Board,

SMART research workshops will be organized in spring of 2015 for identifying possible scientific

priorities of the call.

In addition to the funding scheme of 2013 call (consortium composed by only SMART partners), we

will consider co-funding schemes with industrial partners, academics and hospitals at national and

international levels. Being funded in the context of Sorbonne Universités, SMART will have specific

local actions both in research and education.

2014

January

2nd progress report2nd call for internships

1st Call for Invited Visiting Prof.

April

3rd Call for internships

1st Call for Post-docs1 Post-doc selected

SeptemberJuly

2nd Call for PhD thesis3 Phd thesis selected

November

2013

26 March

1st progress reportSMART kick- off

April

1st Proposers Day

1st Call for interships

1st Call for projects 6 projects started

Consortium Agreementsigned

26 May September DecemberJanuary

2015

19 Jan.

St eering Com.

External Scient ific Com.

SMART Industry Seminars

SMART PhD program

21 Jan.SMART Research Workshops

2nd Call for projects

2nd Proposers Day

Feb. June September

N ew projects

2019

2011

February

1st Call for PhD thesis2 Phd thesis selected

Labex implementation

SMART select ed

Labex submitted

2012

September

88

The SMART action plan is described here:

R&D PROJECTS WITH INDUSTRIAL PARTNERS After this first phase of SMART, partners are working on actions that will foster technology transfer:

SMART Industrial Seminars: These seminars will allow (1) presentations of SMART

activities, (2) identification of potential collaborations with companies with a particular

attention to SMEs.

Co-funding schemes of actions: The next 2015 call for projects will consider co-funding

schemes that will allow companies to collaborate with SMART partners.

These actions will be performed in close collaborations with relevant structures such SATT-LUTECH

(Structure specialized in transfer and commercialization of innovative technologies), and French

Business Clusters for innovation such as CAP DIGITAL and MEDICEN.

SMART partners have already started actions in this line by submitting R&D projects in calls

organized by ANR (French National Agency for Research).

INTERNATIONAL PARTNERSHIPS AND COOPERATIONS During the first phase of SMART, our actions at the international level were organized through the

Invited Visiting Professor Program. SMART scheme allows combining this program with other

actions (supervision of PhD students, participation to projects). Several projects have invited

internationally recognized seniors to participate to SMART research activities. As a result, strong

collaborations have been established with our invited professors.

To enhance international collaborations, we will consider co-funding schemes for the next 2015 call

for projects. These international SMART projects could take benefits from grants to invite

international collaborators as well as to consider short-term missions of PhD students supported by

SMART.

SPECIFIC ACTIONS WITH SORBONNE UNIVERSITÉS PARTNERS SMART Labex is one of the actions of Sorbonne Universités and within this structure various relevant

collaborations could be established. Among the possibilities, there is the IUIS (Institute of Healthcare

Engineering) for clinical applications (a clinician is already funded by IUIS and working on SMART

activities).

We are currently working this the INSEAD-Sorbonne Behavioural Lab (“Centre Multidisciplinaire des

Sciences Comportementales Sorbonne Universités-INSEAD”), which offers the opportunity to collect

behavioral data during complex scenarios (e.g., group of people interacting with avatars) with large

populations (>100) in controlled settings (including ethical issues).

These actions will be performed in collaboration with Sorbonne Universités.

EDUCATIONAL PROGRAM A comprehensive education plan supports SMART objectives through:

Involvement of SMART researchers in teaching units at both Master (Computer Science,

Engineering) and “Licence” (undergraduate) levels.

Supervision of trainees and projects related to SMART research axes.

Funding Master internships through a specific call (21 students)

In 2015, a specific call for teaching activities is planned to support platforms and/or educational

projects. The teaching departments will employ these platforms during projects.

At the doctoral level, we plan to improve our partnership with the three doctoral schools: EDITE

(Computer Science, Telecommunication and Electronics), SMAER (Mechanics, Acoustics,

Electronics and Robotics) and 3C (Cognition, Brain and Behaviors) for evaluation of PhD candidates.

In addition, a comprehensive SMART Doctoral Program will be proposed with these schools on

SMART research areas, while allowing inter-disciplinary PhD thesis.

89

SMART Publications

90

91

2014

Journal articles

1. Buisine S., Courgeon M., Charles A., Clavel C., Martin J.C., Tan N., Grynszpan O. (2014) The

Role of Body Postures in the Recognition of Emotions in Contextually Rich Scenarios,

International Journal of Human-Computer Interaction, 30 (1).

2. Courgeon M., Rautureau G., Martin J.C., Grynszpan O. (2014) Joint Attention Stimulation using

Eye-Tracking and Virtual Humans, IEEE Transactions on Affective Computing.

3. Delaherche E., Dumas G., Nadel J., Chetouani M. (2014) Automatic measure of imitation during

social interaction: a behavioral and hyperscanning-EEG benchmark, Pattern Recognition Letters,

in press.

4. Garnier A., Vidal A., Huneau C., Benali H. (2014) A neural mass model with direct and indirect

excitatory feedback loops: identification of bifurcations and temporal dynamics. To be published

in Neural Computation.

5. Gonzalez F., Gosselin F., Bachta W. (2014). Analysis of Hand Contact Areas and Interaction

Capabilities During Manipulation and Exploration. IEEE Transactions on Haptics. In press.

6. Ivaldi, S.; Anzalone, S.M.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2014) Robot initiative in a

team learning task increases the rhythm of interaction but not the perceived engagement. Frontiers

in Neurorobotics. Vol 8, No 5, DOI 10.3389/fnbot.2014.00005

7. Jarrassé N, Ribeiro AT, Sahbani A, Bachta W, Roby-Brami A. (2014) Analysis of hand synergies

in healthy subjects during bimanual manipulation of various objects. J Neuroeng Rehabil. 2014 Jul

30;11:113. doi: 10.1186/1743-0003-11-113.

8. Petrillo L., Mavridis T., Sarrazin J., Benlarbi-Delai A., De Doncker P. (2014) “Statistical On-

Body Measurement Results at 60 GHz”, IEEE Transactions on Antennas and Propagation, DOI:

10.1109/TAP.2013.2287524.

9. Vérité F., Bachta W., Morel G., (2014) Closed loop kinesthetic feedback for postural control

rehabilitation. IEEE Transactions on Haptics, Special Issue: Haptics in Rehabilitation and Neural

Engineering. IEEE Trans Haptics. 2014 Apr-Jun;7(2):150-60. doi: 10.1109/TOH.2013.64.

International Conferences

1. Ady R., Bachta W., Bidaud, P. (2014). Development and control of a one-wheel telescopic active

cane. IEEE RAS/EMBS BioRob Pages 461 - 466

2. Aklil N., Marchand A., Fresno V., Coutureau E., Denoyer L., Girard B., Khamassi M. (2014)

Modelling rat learning behavior under uncertainty in a non-stationary multi-armed bandit task.

Fourth Symposium on Biology of Decision Making (SBDM 2014). Paris.

3. Campano S., Durand J., Clavel C. (2014) Comparative analysis of verbal alignment in human-

human and human-agent interactions, In Proceedings of LREC 2014, Reykjavik.

4. Chetouani M. (2014) Role of Inter-Personal Synchrony in Extracting Social Signatures: Some

Case Studies, International Workshop on Roadmapping the Future of Multimodal Research, in

conjunction with the ACM International Conference on Multimodal Interaction (ICMI'14),

Istanbul, Turkey.

5. Contardo G., Denoyer L., Artières T., Gallinari P. (2014) Learning States Representations in

POMDP. CoRR abs/1312.6042 (2013) and ICLR 2014 (Short paper).

6. Denoyer L., Gallinari P. (2014) Deep Sequential Neural Network (2014) - Workshop Deep

Learning NIPS 2014.

7. Dulac-Arnold G., Denoyer L., Thome N., Cord M., Gallinari P. (2014) Sequentially Generated

Instance-Dependent Image Representations for Classification, Internation Conference on Learning

Representations – ICLR 2014

8. Françoise J., Schnell N., Bevilacqua F. (2014) MaD: Mapping by Demonstration for Continuous

Sonification ACM SIGGRAPH 2014 Emerging Technologies, Aug 2014, Vancouver, Canada,

France. ACM, pp.16:1-16:1.

9. Garnier A., Huneau C., Vidal A., Wendling F., Benali H. (2014) Identification of dynamical

behaviors in epileptic discharges using a neural mass model with double excitatory feedbacks.

Proceedings of ICCSA 2014: Normandie University, Le Havre, France, 205-210.

92

10. Ivaldi, S,; Anzalone, S.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2014). Robot initiative

increases the rhythm of interaction in a team learning task. Workshop Timing in Human-Robot

interaction, in HRI 2014, Bielefeld, Germany.

11. Petrillo L., Mavridis T., Sarrazin J., Benlarbi-Delai A., De Doncker P. (2014), Experimental On-

Body Shadowing on Torso at 60 GHz, International Conference on Body Area Networks

(BodyNets), London, September 29 – October 1.

12. Langlet C., Clavel (2014) Modelling user's attitudinal reactions to the agent utterances: focus on

the verbal content, LREC Workshop on Emotion, social signals, sentiment & linked open data.

13. Razafimahatratra S., Sarrazin J., De Doncker P., Benlarbi-Delai A. (2014) Horn antenna design for

BAN millimeter wave onbody communication", IEEE Antennas and Propagation Symposium

(APS), Memphis (USA).

14. Roby-Brami A., Van Zandt-Escobar A., Jarrassé N., Robertson J., Schnell N., Boyer E. O.,

Rasamimanana, Hanneton S., Bevilacqua F. (2014) Toward the use of augmented auditory

feedback for the rehabilitation of arm movements in stroke patients. 17th European congress of

physical rehabilitation medicine. Marseille May 2014.

National Conferences 1. Campano S., Glas N., Clavel C., Pelachaud C. (2014) Production d'Hetero-Répétition chez un

ACA, In Proc. Workshop Affect, Compagnon Artificiel, Interaction.

2. Contardo G., Denoyer L., Artières T., Gallinari P. (2014) Apprentissage Sous Contraintes

Budgetisées – Application à la Recommendation – Poster CAP 2014

3. Contardo G., Denoyer L., Artières T., Gallinari P. (2014): Apprentissage Sous Contraintes

Budgetisées – Application à la Recommendation – Poster CAP 2014.

4. Langlet C., Clavel C. (2014) Modélisation des questions de l’agent pour l’analyse des affects,

jugements et appréciations de l’utilisateur dans les interactions humain-agent, In Actes de TALN

2014, Marseille.

5. Michelet S., Achard C., Chetouani M. (2014) Evaluation automatique de l'imitation dans

l'interaction, Reconnaissance de Formes et Intelligence Artificielle (RFIA 2014).

6. Sanlaville K., Bevilacqua F., Pelachaud C., Assayag G. (2014) Adaptation in an Interactive Model

designed for Human Conversation and Music Improvisation: a preparatory outline, Workshop

Affect, Compagnon Artificiel Interaction (WACAI’1), 2014, Rouen.

2013

Journal articles

1. Boyer E.O., Babayan BM., Bevilacqua F., Noisternig M., Warusfel O., Roby-Brami A., Hanneton

S., Viaud-Delmon I. (2013) From ear to hand: the role of the auditory-motor loop in pointing to an

auditory source. Front Comput Neurosci. 2013 Apr 22;7:26. doi: 10.3389/fncom.2013.00026.

Conferences

2. Bevilacqua F., Van Zandt-Escobar A., Schnell N., Boyer E. O., Rasamimanana N., Françoise J.,

Hanneton S., Roby-Brami A. (2013) Sonification of the coordination of arm movements. «

Multisensory Motor Behavior: Impact of sound ». Org Pr A. Effenecberg & Gerd Schmitz,

Leibnitz University Hanover. September 2013.

3. Ivaldi, S.; Anzalone, S.; Rousseau, W.; Sigaud, O.; Chetouani, M. (2013). Cues for making a

humanoid child more human-like during social learning tasks. Workshop on Towards social

humanoid robots: what makes interaction human-like? - IROS 2013.

4. Rousseau, W.; Anzalone, S.; Chetouani, M.; Sigaud, O.; Ivaldi, S. (2013). Learning object names

through shared attention. Workshop on Developmental Social Robotics - IROS 2013.

5. Vérité F., Bachta W., Morel G., (2013) Closed-loop control of a human Center-Of-Pressure

position based on somatosensory feedback. IEEE Intelligent Robots and Systems (IROS).

93