Tools for Ubiquitous HCIEduardo Velloso

Traditional HCI

Ubicomp HCI

Embedded

Ubicomp HCI

Tangible

Ubicomp HCI

Multitouch

Ubicomp HCI

Whole body

Overview

Embedded Interaction Tutorial 1: Digital Camera with .NET Gadgeteer

Tangible and Multitouch Interaction Tutorial 2: Homemade Reactable

Whole-Body Interaction Tutorial 3: Motion Capture with Kinect

Putting it all together

Embedded Interaction

“Embedded means enclosed; these chips and software are not considered computers. They are unseen parts of everyday things.”

Malcolm McCullough, Digital Ground, 2004

Components

Input Buttons, knobs, pressure, temperature, light,

accelerometers… Output

LEDs, displays, DC motors, servos, sound, vibration… Microcontrollers

PIC, Arduino, Gadgeteer, Phidgets…

Toolkits Arduino (starter kit £37)

arduino.cc

“Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It's intended for artists, designers, hobbyists, and anyone interested in creating interactive objects or environments.”

Toolkits Lilypad (starter kit £85) arduino.cc/en/Main/ArduinoBoardLilyPad

“The LilyPad Arduino is a microcontroller board designed for wearables and e-textiles. It can be sewn to fabric and similarly mounted power supplies, sensors and actuators with conductive thread. ”

Toolkits Phidgets (starter kit $230) www.phidgets.com

“Phidgets are a set of "plug and play" building blocks for low cost USB sensing and control from your PC.”

Toolkits .NET Gadgeteer FEZ Spider(starter kit £177)www.netmf.com/gadgeteer/

.NET Gadgeteer

Platform conceived to make it easier to design and build custom electronic devices and systems for a range of ubiquitous and mobile computing scenarios.

Consists of 3 components: Solder-less modular electronic hardware Object-oriented managed software libraries 3D design and construction tools

+ +

Philosophy

Flexibility Software + Hardware + Form factor

Accessibility Extensibility Low threshold for new users

Versatility High ceiling for advanced users Scales up to sophisticated device and complex systems

Extensibility Open source

Connections with sockets

Software Designer

Software

Automatic generation of code

Programming

Object-oriented C# The modules are the objects!

Each modules has Properties Methods Events

Arduino x Gadgeteer

Arduino

Setup() Initialisation of pins

and variables Loop()

Continuous read and write from pins

Gadgeteer

Constructor Initialise variables Set event listeners

Listeners Set the behaviour of

the system when the event happens

Tutorial: Digital Camera

Resources

Papers Hodges, Steve, et al. "Exploring Physical Prototyping

Techniques for Functional Devices using .NET Gadgeteer." (2013).

Villar, Nicolas, et al. ". NET gadgeteer: a platform for custom devices."Pervasive Computing (2012): 216-233.

Official website www.netmf.com/gadgeteer/

Tangible Interaction

Prioritises as principles of design: Tangibility and materiality Physical embodiment of data Bodily interaction Embeddedness in real spaces and data

Blurs the boundaries between input and output

Reactable

Tangible modular synthesizer Controlled by manipulating tangible acrylic pucks Multi-touch interaction Collaborative performances

reacTIVision

Underlying sensor component for the Reactable Open-source, cross-platform computer vision

framework for the fast and robust tracking of fiducial markers attached onto physical objects, as well as multi-touch finger tracking.

reacTIVision

reacTIVision Standalone

application Sends TUIO messages

via UDP port 3333 TUIO protocol

Designed for encoding the state of tangible objects and multi-touch events

TUIO client Your application!

reacTIVision vision engine

TUIO Client

Events

TUIO Objects Represent the fiducial markers Add, remove and update X, Y, orientation, motion and rotation speed and

acceleration TUIO Cursors

Represent fingers Add, remove and update X, Y, speed and acceleration

Tutorial: Reactable

Material: Box Glass Tracing paper Webcam Computer

Tutorial: Reactable

Download the reacTIVision server reactivision.sourceforge.net/

Print the symbols Build your controllers Program your client with the functionality you want by

listening to TUIO messages

Tutorial: Reactable Example TUIO client

Whole Body Interaction

The integrated capture and processing of human signals from physical, physiological, cognitive and emotional sources to generate feedback to those sources for interaction in a digital environment.

Motion Tracking

Optical: Infrastructure-based Inertial: Wearable

OpticalIR projection

Reflective markers

IR cameras

Triangulation

Optical

Commercial solutions: Vicon (£70,000 - £90,000)

www.vicon.com Optitrack (£6,500 - £20,000)

www.naturalpoint.com/optitrack/ Kinect (£90 - £200)

www.microsoft.com/en-us/kinectforwindows/ 

InertialAccelerometer

Gyroscope

Magnetometer

Inertial

Commercial solutions Xsens (£30,000-£60,000)

www.xsens.com Animazoo (£30,000)

www.animazoo.com Wiimote (£20)

www.nintendo.com/wii

Motion Tracking

Optical

Easier setup on the user More complex setup on

the infrastructure Not subject to magnetic

interference Cameras need a clear

field of view to the markers

Inertial

Harder setup on the user

Not much infrastructure needed

Subject to magnetic interference

Occlusion is not a problem

Microsoft Kinect

RGB camera Depth camera Microphone Motor

Microsoft Kinect

Developing for Kinect

Kinect for Windows Runtime Kinect for Windows SDK Kinect for Windows Developer Toolkit

Kinect Studio

Allows you to record and playback the data from the Kinect to test applications.

Putting it all together

What if we’ve got a whole bunch of sensors and we would like to combine what they are sensing to generate more sophisticated output?

Problems Different timestamps Different sample rates Modularity

Context Toolkit

“Facilitate the building of context-aware applications using the Java programming language”.

Georgia Tech Dey, A. K., Abowd, G. D., and Salber, D. 2001. A

conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum.-Comput. Interact. 16, 2 (Dec. 2001), 97-166.

Java Context Awareness Framework

“Java-based set of API’s for creating context-aware applications”

Aarhus University Jakob E. Bardram. The Java Context Awareness

Framework (JCAF) - A Service Infrastructure and Programming Framework for Context-Aware Applications. In Hans Gellersen, Roy Want, and Albrecht Schmidt, editors, Proceedings of the 3rd International Conference on Pervasive Computing (Pervasive 2005), Lecture Notes in Computer Science, Munich, Germany, May 2005. Springer Verlag.

Context Recognition Network Toolbox

“The Context Recognition Network (CRN) Toolbox allows to quickly build distributed, multi-modal context recognition systems by simply plugging together reusable, parameterizable components.”

ETH Zurich Bannach, D., Amft, O., Lukowicz, P.: Rapid Prototyping

of Activity Recognition Applications. In: IEEE Pervasive Computing. Vol 7:2, 2008, 22-31.  ISSN: 1536-1268.

Cosm “Cosm is a secure, scalable platform that connects

devices and products with applications to provide real-time control and data storage”

Originally called Pachube

What’s else?

Eye interfaces Gestural interfaces Wearable computing Sensory interfaces Affective interfaces

Contact e.velloso@lancaster.ac.uk www.eduardovelloso.com