human-computer interaction for...

11/04/2018

1

Julian Fierrez – Seminar at CIMAT, Guanajuato, MEXICO – April 2018 – Slide 1 / 50

Human-Computer Interaction

for Biometrics

Prof. Julian FIERREZ

Universidad Autonoma de Madrid - SPAIN

http://atvs.ii.uam.es/fierrez


Funding Acknowledgements

Public Private

http://atvs.ii.uam.es/fierrez

11/04/2018

2


Outline

1. Biometrics in Access Control Scenarios

2. Authentication in On-Line Applications

3. Biometrics based on HCI

–Handwriting and Signature

–Graphical Passwords

–Swipe Biometrics

–Mouse Dynamics

–Keystroke Dynamics

5. Future Trends & Conclusions


Biometrics in Access Control Scenarios

• Biometric access control deployments have concentrated much

efforts and resources in the last 15 years

• Led the market of biometric authentication technologies

• In those scenarios, the user is physically present (in-situ)

• No specific actions from the user are needed (other than placement)

• Once checked, access granted / denied

• If denied, human supervision as an alternative

• Physiological (morphological) modalities are in general better

adapted to these type of tasks

11/04/2018

3


Human Activity Patterns

• Human activity patterns are clearly stablished from childhood

• As patterns, they are stable and reproducible, though subject to

variability

• Neuromotor coordination of gestures, movements and speech

• Continuous identity monitoring possible

• User is an active part of the play

• Multilevel strategy: from dynamic trajectories to expressions,

context, habits, stylometry, experiences

• Not fixed patterns but changing and adapting ones


Behavioral biometrics, but not neuromotor HDI

Not online HDI (just sensing) for authentication

Human-Computer Interaction for Biometrics

• User is monitored / connected at least for several minutes

• Not only security checking but user convenience scenarios

• Human-device interaction (HDI) is rich in terms of human activity

and behavioral patterns

– Keystroking

– Mouse dynamics

– On-Line Handwritting & Signature

– Graphical Passwords

– Swipe and Gesture Biometrics

– Speech

– Gait

11/04/2018

4


Biometric Market by Modality

• Decreasing: Fingerprint, from 48% to 15% (18% excluding AFIS)

• Growing: Iris from 9% to 16% (19%) and Face from 12% to 15% (18%)

• Huge growing: Speech from 6% to 13% (15,5%) and Signature, from 2% to 10% (12%)


Active Authentication by DARPA

11/04/2018

5


On-line Signature Verification

Feature-based (Global Features)

Distance-based classifiers• Mahalanobis• Euclidean [Nelson et al., 1994]

Statistical/other classifiers• Gaussian Mixture Models

(GMM)• Parzen Windows

Function-based (Local Features)

Time-Sequence matching techniques

• Hidden Markov Models (HMM) [Dolfing et al., 1998]

• Gaussian Mixture Models (GMM) [Richiardi et al., 2005]

• Dynamic Time Warping (DTW) [Sato and Kogure, 1982]

sample index

0 50 100 150 200 250 300 350 4000

2000

4000

x

0 50 100 150 200 250 300 350 4000

1000

2000

y

0 50 100 150 200 250 300 350 4000

500

1000

z

0 50 100 150 200 250 300 350 4001000

1200

1400

azim

uth

0 50 100 150 200 250 300 350 400400

500

600

altitu

de

Dynamic signature matching

J. Fierrez and J. Ortega-Garcia, "On-line signature verification", A.K. Jain et al. (Eds), Handbook of Biometrics, 2008.

M. Martinez-Diaz and J. Fierrez, "Signature Databases and Evaluation", Stan Z. Li and Anil K. Jain (Eds.), Encyclopedia of Biometrics, Springer, pp. 1367-1375, 2015.


Signature Acquisition and Pre-processing

-80 -60 -40 -20 0 20 40 60 80-80

-60

-40

-20

0

20

40

60

80

Key step to greatly reduce sensor interoperability issues due to heterogeneous devices and

writing tools (stylus/finger)

Pre-processing

- Size normalization and centering

- Pressure normalization

- Resampling

M. Martinez-Diaz, J. Fierrez and S. Hangai, "Signature Features", Stan Z. Li and Anil K.

Jain (Eds.), Encyclopedia of Biometrics, Springer, pp. 1375-1382, 2015.

11/04/2018

6


Dynamic Time Warping (DTW)Hidden Markov Models (HMM)

Point-to-point correspondenceStatistical modeling of signature regions

M. Martinez-Diaz, J. Fierrez and S. Hangai, "Signature Matching", Stan Z. Li and Anil K. Jain

(Eds.), Encyclopedia of Biometrics, Springer, pp. 1382-1387, 2015.

Similarity Computation


• Deep Neural Network- Based System accomplishes:

• Linear Transformations

• Non-Linear Transformations (tangh, sigmoid, ReLU, etc) -> θ

J. Schmidhuber, “Deep Learning in Neural Networks: An Overview”, Neural Networks, vol. 61, pp. 85-117, 2015.

OUTPUT

θ θ

θ θ

θ θ

θ θ

OUTPUTINPUT

Use of Recurrent Neural Networks

11/04/2018

7


• Recurrent Neural Networks, well adapted to time sequences (speech, handwritting…)

• Different topologies:

• Long Short-Term Memory (LSTM)

• Gated Recurrent Unit (GRU)

• Also bidirectional versions: BLSTM y BGRU

• Siamese arquitecture

Time Sequence

Memory!

A. Graves, A.R. Mohamed, and G. Hilton, “Towards End-to-End Speech Recognition with Recurrent Neural

Networks”, in Proc. International Conference on Machine Learning, vol. 14, pp. 1764-1772, 2014.

R. Tolosana, R. Vera-Rodriguez, J. Fierrez and J. Ortega-Garcia, "Exploring Recurrent Neural Networks for

On-Line Handwritten Signature Biometrics", IEEE Access, Vol. 6, pp. 5128-5138, February 2018.

Use of Recurrent Neural Networks


Traditional Acquisition Scenario (2002-2013)

M. Martinez-Diaz and J. Fierrez, "Signature Databases and Evaluation", Stan Z. Li and Anil K. Jain

(Eds.), Encyclopedia of Biometrics, Springer, pp. 1367-1375, 2015.

sample index

0 50 100 150 200 250 300 350 4000

2000

4000

x

0 50 100 150 200 250 300 350 4000

1000

2000

y

0 50 100 150 200 250 300 350 4000

500

1000

z

0 50 100 150 200 250 300 350 4001000

1200

1400

azim

uth

0 50 100 150 200 250 300 350 400400

500

600

altitu

de

Altitude (0°-90°)

90°

270°

0°

Azimuth (0°-359°)

180°

Altitude (0°-90°)

90°

270°

0°

Azimuth (0°-359°)

180°

11/04/2018

8


Our HMM implementation

SVC-04 skilled forgeries

SVC-04 random (zero-effort, casual) impostors

http://www.cs.ust.hk/svc2004/

Benchmarks: SVC 2004


BioSec Signature Evaluation Campaign, BSEC’09

• DTW, HMM and Global Systems

• Score normalization

• Fusion of systems

False Acceptance Rate (%)

Fals

e R

ejec

tio

nR

ate

(%)

N. Houmani, et al., "BioSecure signature evaluation campaign(BSEC2009): Evaluating online signature algorithms dependingon the quality of signatures", Pattern Recognition, March 2012.

http://www.cs.ust.hk/svc2004/

11/04/2018

9


4 training signatures

16 signatures

31 signatures

41 signatures

Random Forg.

97.2 % 99.3 % 99.9 % 99.9 %

Skilled Forg.

88.3 % 93.1 % 95.9 % 99.3 %

• Accuracy (SLT Database):

• State of the art performance

• Template and system configuration update strategies in order to minimize the aging effect

R. Tolosana, R. Vera-Rodriguez, J. Ortega-Garcia and J. Fierrez, "Preprocessing and Feature Selection for Improved

Sensor Interoperability in Online Biometric Signature Verification", IEEE Access, Vol. 3, pp. 478 - 489, May 2015.

BIOTRACE100 Performance (2015)


• 5 devices (3 Wacom, 2 Samsung)

• 8 genuine signatures and 6 skilled forgeries per user and device

• Stylus and finger as writing tools (Samsung)

R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, J. Ortega-Garcia, “Benchmarking Desktop and Mobile

Handwriting across COTS Devices: the e-BioSign Biometric Database” PLOS ONE, 2017.

• 70 users, 2 capturing sessions

e-BioSign Database (2016-2017)

11/04/2018

10


Challenges on Handheld Devices

Lack of space [Simsons et al., 2011]

Higher client-entropy[Garcia-Salicetti et al., 2008]

Stylus (or finger)

Ergonomics [Blanco-Gonzalo

et al., 2013b]

Standing

position

Lack of in-air trajectories [Sesa-Nogueras et al., 2012]

Sampling

quality

Lack of pressure

and orientation

signals[Muramatsu and

Matsumoto, 2007]

F. Alonso-Fernandez, J. Fierrez and J. Ortega-Garcia, "Quality Measures in Biometric Systems", IEEE Sec. & Privacy, Dec 2012.


From Signature to Touch Gestures

• Graphical Password-based User Authentication with Free-form Doodles

M. Martinez-Diaz, J. Fierrez and J. Galbally, "Graphical Password-based User Authentication with Free-Form

Doodles", IEEE Trans. on Human-Machine Systems, August 2016.

M. Martinez-Diaz, J. Fierrez, and J. Galbally. “The DooDB graphical password database: Data analysis and

benchmark results”. IEEE Access, September 2013.

11/04/2018

11


Graphical Passwords

• Gesture-based authentication on touch-screens

• Slow typing in touchscreens

• Biometric-rich gestures

• Revocability

Behavioral Biometrics

Physiological Biometrics

Graphical Passwords

M. Martinez-Diaz, J. Fierrez and J. Galbally, "The DooDB Graphical Password Database: Data Analysis and Benchmark Results", IEEE Access, Sept. 2013.


Graphical Passwords: Related Works

Pattern Lock [Google]US Patent 20130047252 A1

Multi-touch gestures [Sae-Bae et al., 2012]US Patent 20130219490 A1

Draw a Secret [Jermyn et al., 1999]US Patent 8024775 B2

Pass-Go [Tao et al., 2008]

Picture Gesture Authentication [Microsoft]US Patent 20130047252 A1

1 2

3 4 5

(2,2)

(3,2)

(3,3)

(2,3)

(2,2)

(2,1)

11/04/2018

12


Graphical Examples

Doodles Pseudo-signatures

Genuine samples Forgeries Genuine samples Forgeries

M. Martinez-Diaz, J. Fierrez and J. Galbally, "The DooDB Graphical Password Database: Data Analysis and Benchmark Results", IEEE Access, Sept. 2013.


Current Work: Swipe Biometrics

Continuous user authentication through touch biometrics:- Security beyond the entry-point

Situation:- Freely interacting with the touchscreen while reading or viewing images

11/04/2018

13


Example of strokes captured for two different users

USER A USER B

Session 2, test set

Session 1, training set

High inter-user

variability:

distinctiveness

High intra-user

variability: Difficult to

model the user

Abdul Serwadda, Vir V. Phoha, and Zibo Wang. “Which verifiers work?: A benchmark evaluation of touch-based authentication algorithms”. In 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pages 1–8, 2013.


A. Pozo, J. Fierrez, M. Martinez-Diaz, J. Galbally and A. Morales, "Exploring a Statistical Method for TouchscreenSwipe Biometrics", in Proc. Intl. Carnahan Conference on Security Technology, ICCST 2017, October 2017.

Current Work: Swipe Biometrics

11/04/2018

14


Current Work: Mouse Dynamics


Mouse Dynamics

• Mouse dynamics analyze

behavior pattern of users for

classification, verification and

identifications.

• Mouse dynamics is a behavioral

biometric characteristic.

Challenges

• High error rates

• High intra-classvariability

Advantages

• No intrusive

• Easy acquisition

• Continuousmonitoring

Intra-class variability (example)

0 100 200 300 400 500 600 7000

100

200

300

400

500

600

700

Distance (pixels)

Dis

tan

ce

(pix

els

)

Sample 1

Sample 2

11/04/2018

15


S. Chao, C. Zhongmin, G. Xiaohong and R. Maxion, “Performance evaluation of anomaly-detection algorithms for mouse dynamics,” Computers and Security, vol. 45, pp. 156–171, 2014.

Mouse Dynamics: Features


S. Chao, C. Zhongmin, G. Xiaohong and R. Maxion, “Performance evaluation of anomaly-detection algorithms for mouse dynamics,” Computers and Security, vol. 45, pp. 156–171, 2014.

Mouse Dynamics: Performance

11/04/2018

16


Keystroke Dynamics


Keystroke Dynamics - History

‘Telegraphist style’ during World War II

Keystroke dynamics authentication

emerges with the massive deployment

of computers

Transparent, low-cost solution

Easy to implement in any keypad

device

R. S. Gaines, W. Lisowski, S. J. Press, and N. Shapiro, “Authentication by keystroke timing: Some preliminary results,” tech.

rep., DTIC Document, 1980.

11/04/2018

17


Keystroke Dynamics - History

‘Telegraphist style’ during World War II

Keystroke dynamics authentication

emerges with the massive deployment

of computers

Transparent, low-cost solution

Easy to implement in any keypad

device


Mobile

Buschek D, De Luca A, Alt F. 2015. Improving accuracy, applicability and usability of keystroke biometrics on mobile touchscreen devices. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15.

Traditional Keyboard

Text-dependent / passwords Text-independent

“In a village of La Mancha, the name of which I have no desire to call to mind, there lived not long since one of those gentlemen that keep a lance in the lance-rack, an old buckler, a lean hack, and a greyhound for coursing…”

Tablet

Keystroke Dynamics: Scenarios

11/04/2018

18


From Password-Based to Free-Text Authentication

Substitute password-based authentication + keystroke dynamics

Biometric system as a secondary security level to detect intruders

Free text usually divided into small strings (digraphs and trigraphs)

Match?

DDBB

Keystroke Biometric

DDBB

Text level Biometric level

Accept/Denied

Biometric TemplatesList of passwords

STEP 1 STEP 2

Password


From Password-Based to Free-Text Authentication

Substitute password-based authentication + keystroke dynamics

Biometric system as a secondary security level to detect intruders

Free text usually divided into small strings (digraphs and trigraphs)

Keystroke Biometric

DDBB

Matching at biometric feature level

Accept/Denied

Biometric Templates

Free Text

11/04/2018

19


M. L. Ali, J. V. Monaco, C. C. Tappert, and M. Qiu. "Keystroke biometric systems for user

authentication." Journal of Signal Processing Systems 86, no. 2-3 (2017): 175-190.

Input sample

Pre-processing

Feature Extraction


Score normalization

Decision Threshold

Accepted or rejected

Keystroke Verification Enrolled models

Identity claim

• Sequence alignment through DTW

Keystroke Verification: Sequence Alignment


Input sample

Pre-processing

Feature Extraction


Score normalization

Decision Threshold



Identity claim

Keystroke Verification: Features

• A. Morales, J. Fierrez and J. Ortega-Garcia , "Towards Predicting Good Users for Biometric Recognition based onKeystroke Dynamics", Proc. of European Conference on Computer Vision Workshops, Springer LNCS-8926, Sept. 2014.

• A. Morales, J. Fierrez, R. Tolosana, J. Ortega-Garcia, J. Galbally, M. Gomez-Barrero, A. Anjos and S. Marcel, "KeystrokeBiometrics Ongoing Competition", IEEE Access, Vol. 4, Nov. 2016.

11/04/2018

20


Input sample

Pre-processing

Feature Extraction


Score normalization

Decision Threshold



Identity claim

Classifiers based of algorithmic

fusion (GMM, SVM, normalized

Manhattan distance…)

Keystroke Verification: Similarity Computation


Keystroke Dynamics: Performance and Benchmark

Performance, strongly user-dependent and scenario-dependent.

Password Authentication (CMU benchmark, 2012) : 8% EER (only

genuine samples during training)

Free-Text, ICB 2015 Competition: 83% Identification Rate (100

characters as query samples, 500 characters as development

samples)

New Benchmark: Keystroke Biometrics Ongoing Competition*

J. V. Monaco, G. Perez, C. C. Tappert, P. Bours, S. Mondal, S. Rajkumar, A. Morales, J. Fierrez and J. Ortega-Garcia,"One-handed Keystroke Biometric Identification Competition", in Proc. IEEE/IAPR Int. Conf. on Biometrics, ICB, May 2015.

*A. Morales, J. Fierrez, R. Tolosana, J. Ortega-Garcia, J. Galbally, M. Gomez-Barrero, A. Anjos and S. Marcel, "KeystrokeBiometrics Ongoing Competition", IEEE Access, Vol. 4, pp. 7746-7746, November 2016.

11/04/2018

21


Biometrics Research:

A Look into the Future


* J. Fierrez-Aguilar, D. Garcia-Romero, J. Ortega-Garcia and J. Gonzalez-Rodriguez, "Bayesian adaptation for user-dependentmultimodal biometric authentication", Pattern Recognition, August 2005.**J. Fierrez-Aguilar, D. Garcia-Romero, J. Ortega-Garcia and J. Gonzalez-Rodriguez, "Adapted user-dependent multimodal biometricauthentication exploiting general information", Pattern Recognition Letters, December 2005.

Knowledge Base

+ Experiments

+ Experiments’

- Domain adaptation

- Transfer learning

- Inductive transfer

- ...

Bayesian adaptation*

Discriminative adaptation**

The Future of Biometrics based on HCI

Challenge 1: Adapting to New Application Scenarios

11/04/2018

22


• P. Aleksic, M. Ghodsi, et al. “Bringing Contextual Information to Google Speech Recognition”, Interspeech, 2015.• F. Alonso-Fernandez, J. Fierrez, et al, “Quality Measures in Biometric Systems”, IEEE Sec. & Privacy, Dec. 2012.• F. Alonso-Fernandez, J. Fierrez, et al., "Quality-Based Conditional Processing in Multi-Biometrics: application to

Sensor Interoperability", IEEE Trans. on Systems, Man and Cybernetics A, Vol. 40, n. 6, pp. 1168-1179, 2010.• J. Fierrez, et al., "Multiple Classifiers in Biometrics. Part 2: Trends and Challenges", Information Fusion, Nov. 2018.

Signal Quality, EnvironmentalData, etc.

Knowledge

Base


Challenge 2: Incorporating Contextual Information


44

• Failure to acquireevent

[Simon-Zorita et al. 03]

[Chen et al. 05]

• Q-based fusion

[Bigun et al. 97, 03]

[Fierrez et al. 05, 06]

[Nandakumar et al. 06, 08]

• Q-based featureweighting

[Chen et al. 05]

• Q-basedenhancement

[Hong et al. 98]

• J. Fierrez-Aguilar, J. Ortega-Garcia, J. Gonzalez-Rodriguez and J. Bigun, "Discriminative multimodal biometric

authentication based on quality measures", Pattern Recognition, May 2005.

• J. Fierrez, A. Morales, R. Vera-Rodriguez and D. Camacho, "Multiple Classifiers in Biometrics. Part 2: Trends

and Challenges", Information Fusion, November 2018.

11/04/2018

23


• J. Fierrez, J. Ortega-Garcia and J. Gonzalez-Rodriguez, "Target Dependent Score Normalization Techniques and their Application to Signature Verification", IEEE Trans. on Systems, Man and Cybernetics-C, August 2005.

• J. Galbally, M. Martinez-Diaz and J. Fierrez, "Aging in Biometrics: An Experimental Analysis on On-Line Signature", PLOS ONE, July 2013.

• J. Fierrez, A. Morales, R. Vera-Rodriguez and D. Camacho, "Multiple Classifiersin Biometrics. Part 2: Trends and Challenges", Information Fusion, Nov. 2018.

User-specificbehaviour

Knowledge

base


Challenge 3: Adapting to the User (e.g., Aging)


Auxiliary Information Obtained during Enrollment: Exploitation of the enrollment data (multiple samples) not only to create the templates/models but also to adjust in a user-dependent way some parameters of the system during verification

e.g., good/bad

users

[Hicklin et al., “The myth of goats: how many people have fingerprints that are hard to match”, NISTIR 7271, 2005].

11/04/2018

24


• UD fusion

[Jain et al., ICIP 02]

[Toh et al., TSP 04]

[Snelick et al., PAMI 05]

[Fierrez et al., PR 05]

• UD score normalization

[Fierrez et al., TSMC 05]

[Poh et al., MMUA 06, TASLP 08]

• UD modeling

[Martinez et al., ICFHR 08]

• UD decision (e.g., UD tresholds [Jain et

al., PR 02], failure to enroll events and exception handling)

• UD features

[Fairhurst et al., IPRAI 94]

J. Fierrez, A. Morales, R. Vera-Rodriguez and D. Camacho, "Multiple Classifiers in Biometrics. Part 2: Trends and Challenges", Information Fusion, Nov. 2018.

Exploiting the Zoo (User-Dependent Processing)


GLOBAL: Set of training scores from a pool of users (genuine and impostor)LOCAL: Set of training scores from the user at hand (genuine and impostor)

Bayesian and SVM user-dependent fusion algorithms

J. Fierrez, A. Morales, R. Vera-Rodriguez and D. Camacho, "MultipleClassifiers in Biometrics. Part 2: Trendsand Challenges", Information Fusion, Nov. 2018.

11/04/2018

25


Signer 1

Signer NKnowledge

Base

Big Data

Deep Learning

Anonymized

info

Ruben Tolosana, Ruben Vera, Julian Fierrez, Javier Ortega, “Exploring Recurrent Neural Network for Handwriting Signature Biometrics” IEEE Access, 2018.


Challenge 4: Exploiting Big Data


Conclusions

• Matureness of technologies (signature, keystroke)

• Major role in on-line and mobile remote applications

• User convenience to drive application development

• Room for substantial industry-applicable research

RevocabilityEasy of use, user acceptanceLess sensor-interoperability issuesEasy to integrate at low-costContinuous ID

User intra-variabilityMulti-sample trainingModel updatingMultilevel strategiesData scarcity

human-computer interaction for...

Documents