mind control to major tom: is it time to put your eeg headset on?

@spoole167 @Luc_At_IBM

Mind Control to Major Tom:

Is It Time to Put Your EEG Headset On?

CON2338


Who we areSteve Poole aka Obiwan

Java guru, Developer extraordinaireDelivery lead

Luc Desrosiers aka Lobot (Think Star Wars Cloud city)

Canadian expat living in the UKGadget & IoT enthousiast… and Cloud architect

@spoole167 @Luc_At_IBMhttps://www.flickr.com/photos/stawarz/


Who we really are

Terminators from the future…


Here to promote our book series

“Skynet for Dummies”

Today we’ll be talking about the basics for creating your own terminator


Skynet will be built with the greatest and best tools

We picked Java because, even in your day it’s on many, many devices

We will be using Java 8 - because even in the future Java 9 hasn’t shipped yet


Outline

Ways to interact with your human

understanding your human

controlling your human


https

://w

ww

.flic

kr.c

om/p

hoto

s/gy

duxa

/

Ways to interact with your human


Vision

AuditoryTouch

GustatoryOlfactory

Proprioception

Key input/output components of ahuman being


Human being interaction map

Virtual Smell

Electronic noses

Wearables

Haptic sensorsMotion sensors

Molecular analysis

Virtual Reality

Computer VisionText-to-Speech

Speech-to-Text

Touch and proprioception

Gustatory Olfactory

Auditory Vision

Author: Allan-Hermann Pool

https://commons.wikimedia.org/wiki/File:Five_senses.jpg


Convincing your human its in a new world

Teaching your terminatorto understand the real world

Virtual RealityAugmented Reality

World Domination


Goal is full immersive sensory replacement.

Environment Awareness

Object and Facial detection

IBM WatsonVisual Recognition


Virtual Reality, or the many challenges of fooling the human beings

Images by Pdenbrook - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=31920588

From user interface

You want to reduce motion sickness?Add a nose!

Pointman

https://www.wired.com/2015/04/reduce-vr-sickness-just-add-virtual-nose/

https://en.wikipedia.org/wiki/Pointman_(user_interface)


Wait, where did my hands go?frame.hands().forEach(hand->{

Arm arm = hand.arm(); if (arm.isValid()) { renderArm(arm);}

hand.fingers().forEach(finger->{if (finger.isValid()) {

for(int i = 3; i >= 0; i--) {Bone bone = finger.bone(Type.swigToEnum(i));renderCylinder(bone.width(), bone.length(),

bone.prevJoint(), bone.direction()); }

} });});


Making the humans feel the virtual world

Sense of touch has been attempted using:• Vibration• Small pockets of air balloons• Electrical impulse• Inertial sensors

Many startup and currently many failures…


Recognizing objects in a scene

Completely possible… However it requires a lot of training!

Complexity does not lie in the coding but in getting a good set of positive and negative images:- Positive images have the object you want to identify- Negative images do not

Group positive images together and we call them class.Group related class together and we call them classifier.


Demo: Visual Recognition with Watson

VisualRecognition service = new VisualRecognition(VisualRecognition.VERSION_DATE_2016_05_19);

service.setApiKey("{api-key}");

System.out.println("Classify an image"); ClassifyImagesOptions options = new ClassifyImagesOptions.Builder().images(

new File("src/test/resources/visual_recognition/car.png")).build();

VisualClassification result = service.classify(options).execute();

System.out.println(result);


Hasta la vista Baby:

it’s not just about vision


Understanding what your human is saying

Getting your human to understand who’s the boss

Speech-to-Text Text-to-Speech(aka)

World Domination


Demo: Text to Speech Furbynator speaks

TextToSpeech service = new TextToSpeech();

String text = "Hello world.";

InputStream stream = service.synthesize (text, Voice.EN_ALLISON, "audio/wav");

OutputStream out = new FileOutputStream("hello_world.wav");


Demo: Speech to Text

SpeechToText service = new SpeechToText();

RecognizeOptions options = new RecognizeOptions().contentType("audio/flac”).timestamps(true).wordAlternativesThreshold(0.9).continuous(true);

File file = new File("audio-file1.flac”);

SpeechResults results = service.recognize(file, options);

System.out.println(results);


Today VirtualReality

ComputerVision

HandTracking

Speech-to- Text

Text-to-Speech

Current Applicability 6/10 7/10 8/10 7/10 8/10Java Coverage 4/10 8/10 9/10 9/10 9/10Ease of use 4/10 6/10 9/10 8/10 8/10Reliability 6/10 7/10 8/10 8/10 9/10

FuturePotential for improvement High High Medium Medium Medium

Scorecard


Understanding the emotional state of your Human

https

://w

ww

.flic

kr.c

om/p

hoto

s/pi

xiet

art/


Understanding emotion requires combining detection and analysis of facial features, prosody and lexical content in speech

This is hard!


Recognizing facial features

Two approaches:

Take many pictures of humans in different moods and teach a neural net to do pattern matching

Detect interesting face points (nose tips, mouth corners, eyes etc and determine relationship between them.


OpenCV has good Face Detection

VideoCapture camera = new VideoCapture();

CascadeClassifier cc = new CascadeClassifier( "haarcascade_frontalface_alt.xml");

MatOfRect faces = new MatOfRect();

Mat frame = new Mat();

camera.read(frame);cc.detectMultiScale(frame, faces);

Rect[] hits=faces.toArray();

for(int i=0;i<hits.length;i++) {Mat face = new Mat(frame,hits[i]);Imgcodecs.imwrite("face"+i+".jpg", face);

}


Simple Face Recognition / Emotion detection is much harder

OpenCV has image training capability but it’s not available as a Java API

It’s computationally very expensive

You need a database of images that contain the items you want to detectYou need a database of images that do not contain the items you want to detect

OpenCV training requires creating images from the second set with items from the first set added in at various scales and rotational angles

http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html


Detection alternatives:

Get humans to score a series of pictures of other humans suffering emotions

http://www.ipsp.ucl.ac.be/recherche/projets/FaceTales/en/Home.htm

Apply various point analysis techniques to extract key parts of a face

http://www.codeproject.com/Articles/110805/Human-Emotion-Detection-from-Image

https://github.com/mpillar/java-emotion-recognizer


Creating test images of the various emotions

Challenge:

By Crosa (Flickr: Scream) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

https://www.flickr.com/photos/huphtur/

https://www.flickr.com/photos/auroredelsoir/

For some reason humans only ever scream?

https://www.flickr.com/photos/tenaciousme/https://www.flickr.com/photos/morton/


Analysing emotion from text and speech

This is still hard. Terminators are not good with speech

I’ll be back

asta la vista....baby

Come with me if you want to liveYou’ve been terminated

"Get out."

I need your clothes, your boots and your motorcycle.

Stay here. I'll be back


ToneAnalyzer service = new ToneAnalyzer(ToneAnalyzer.VERSION_DATE_2016_05_19);

ToneAnalysis tone = service.getTone(text, null).execute();

System.out.println(tone);

Watson Tone Analysis sample


Analysing tone with IBM Watson:

I need your clothes, your boots and your motorcycle.I’ll be back.Get out.asta la vista....baby.Stay here.I'll be back.You’ve been terminated.Come with me if you want to live

Conscientiousness

Extraversion

Analytical


Analysing tone: the more words the better:

Using your mind to interact with computers is a long-standing desire. Advances in technology have made it more practical, but is it ready for prime time? This session presents practical examples and a walkthough of how to build a Java-based end-to-end system to drive a remote-controlled droid with nothing but the power of thought. Combining off-the-shelf EEG headsets with cloud technology and IoT, the presenters showcase what capabilities exist today. Beyond mind control (if there is such a concept), the session shows other ways to communicate with your computer besides the keyboard. It will help you understand the art of the possible and decide if it's time to leave the capsule to communicate with your computer.


Analysing tone: the more words the better:


But, but, Terminators

can have a personality too!


Maybe, but Personality of Terminators tends to be too predictable… Exterminate??

So lets look at the personality of the father of Robotics

https://en.wikipedia.org/wiki/I,_Robot_(film)

Isaac Asimov

https://en.wikipedia.org/wiki/I,_Robot_(film

https://en.wikipedia.org/wiki/I,_Robot_(film


This analysis was based from an extract of the text of iRobot

We could not contact Isaacnext of kin to validate Watson’s findings…

but Wikipedia seems to agree:https://en.wikipedia.org/wiki/Isaac_Asimov

https://en.wikipedia.org/wiki/Isaac_Asimov


Watson Personality Insight sample

PersonalityInsights service = new PersonalityInsights();service.setUsernameAndPassword("{username}", "{password}");try {

JsonReader jReader = new JsonReader(new FileReader("profile.json"));Content content = GsonSingleton.getGson().fromJson(jReader, Content.class);

ProfileOptions options = new ProfileOptions().contentItems(content.getContentItems());

Profile profile = service.getProfile(options);System.out.println(profile);

} catch (FileNotFoundException e) {e.printStackTrace();

}


Today ToneAnalysis

PersonalityAnalysis

Current Applicability 8/10 8/10Java Coverage 9/10 9/10Ease of use 8/10 7/10Reliability 8/10 7/10

FuturePotential Improvement Medium Medium

Scorecard


Ways to control with your human

https://www.flickr.com/photos/adforce1/


We tried inserting machines into humans.

Too messy and somewhat obvious.

https

://w

ww

.flic

kr.c

om/p

hoto

s/au

spic

es/


We tried adding machines outside humans.

Mixed success

By C

cmsh

arm

a2 -

Ow

n w

ork,

CC

BY-S

A 4.

0, h

ttps

://c

omm

ons.w

ikim

edia

.org

/w/in

dex.

php?

curid

=449

2070

7

https://www.youtube.com/watch?v=aJwpG6LVaZQ

43IBM _

In fact it is much simpler!!! Just hack the brain!


Capturing brain activity (when applicable) of a human

IOT Command• Push• Pull• Stop

BluetoothBB8.roll

IOT Event•Push•Pull•Neutral


Today MindControl

Current Applicability 3/10Java Coverage 7/10Ease of use 4/10Reliability 2/10

FuturePotential Improvement High

Scorecard


So its clear that we’re not quite ready to take over the world…

https://www.youtube.com/watch?v=f8T8eWBmls0&list=PLMtQR4NXkj1BUU1JcQSEXBMJqyYYYVQfw


“we’ll be back”

mind control to major tom: is it time to put your eeg headset on?

Software