Who's afraid of job interviews? Definitely a Question for User Modelling

Kaśka Porayska-Pomsta, Paola Rizzo, Ionut Damian, Tobias Baur, Elisabeth André, Nicolas Sabouret, Hazael Jones, Keith

Anderson, Evi Chryssafidou

1

Context

• Young people Not in Education, Employment or Training (NEET)

• 18-25 years old• High risk of social exclusion• In EU: ~20%.• Lack of social skills for job

interviews

Context Continues

• TARDIS European project: – serious game for practicing job interviews with a virtual

recruiter

User Modelling in TARDIS• user modelling in job interviews:– need to detect youngsters’ social cues – infer complex mental states from social cues in real time to

tailor the virtual recruiter’s behaviour– Display the information to the youngsters and helping

practitioners

openssi.net

Questions and studies• What social cues and mental states are relevant?• What is feasible to detect with non-intrusive technology?

– What aspects of the interaction lead to (detectable) nonverbal behaviours in users?

– How to evaluate anxiety?

4 types of studies:– field-based human-to-human– field-based computer-mediated human-to-human– lab-based WOZ– field-based human-to-agent

1. Human-to-human job interviews

• Goal: identify social cues and hidden mental states

• video recorded interactions among 10 youngsters and 5 practitioners

• Post-hoc video walkthroughs with practitioners

• Manual annotation of social cues and mental states

• identification of 19 cues and 8 mental states

1. Human-to-human job interviews

8 Mental States:•Stressed•Embarassed•Hesitant•Ill-at-ease•Bored•Focused•Relieved •Relaxed

2. Computer-mediated human-to-human

• Goal: verify automatic detection of social cues

• mock interviews (5 youngsters, 2 practitioners) mediated through video link and headsets

• cues recorded by social cue recognition component using Kinect & mics

• refinement of social cues according to sensitivity of devices, background noise, available software libraries

SSI-NoVA display shows no cues!

9

3. Lab-based Wizard of Oz

• Goals: – try sensors that could enhance

cue recognition – ascertain the impact of specific

questions on participants• sensors: Kinect, headset, eye

tracking glasses, motion tracking glove, SC/BVP sensors

• subjects: 3 university students

3. Lab-based Wizard of Oz

• even with the more challenging scenario, users still performed very few and small physical movements

• SC values showed the impact of the interview questions on users– e.g. “What are your

weaknesses?” or “I don't think you are right for this job” correlated with higher SC values

4. Field-based human-to-agent

• Goal: pilot a pop-up questionnaire to elicit self-reports about anxiety

• 7 subjects, 2 virtual recruiters: “Demanding” vs “Understanding”

• 3 question categories: (i) skills required, (ii) knowledge of the job, (iii) salary sought

4. Field-based human-to-agent

• small sample with no statistically significant effects of:– 2 recruiter conditions– three question categories:

(i)skills required, (ii) knowledge of the job,(iii) salary level sought

• nevertheless, 2 possible trends:– trait anxiety– some types of questions may

lead to greater anxiety

Conclusions

• one lesson learnt: non-intrusive sensors in field conditions, and the emotion suppression in this interaction domain, lead to a reduced set of detectable cues

• need for an initial training phase of the user model during which individual users' baseline of social cues can be established – allows for a tailored parameter adjustment based on the

frequency of a given users' cues– users' behaviours are compared to their typical baseline

and peak behaviours are identified

14

• Focus on key social cues, such as voice, that can be reliably detected through the sensing technologies, coupled with a reduced focus on state anxiety

• A complementary approach, currently piloted: open user modelling –the models generated online are displayed to the users who can accept or correct them according to their self-perception– this allows to both validate TARDIS' user models and to

foster self-awareness in the youngsters - a pre-requisite job interview skill

15