assistive context-aware toolkit (english)
Post on 12-Apr-2017
137 Views
Preview:
TRANSCRIPT
Assistive Context-Aware ToolkitFelipe Pedroso
#IntelNoTDC
Agenda• Stephen Hawking and Intel• What is ACAT?• Technical Details• Architecture
• Creating Extensions• How to?• RealSense + ACAT• SharpSenses + ACAT
• Summary and Call 4 Action
#IntelNoTDC
• Professor Hawking first met Gordon Moore during a conference in 1997, where he was offered a computer with an Intel processor
• Intel upgrades his computer every couple of years and also supports him all his hardware requirements
• A cross-disciplinary team from Intel Labs developed ACAT to enable him to communicate and perform all his computer tasks more efficiently
Stephen Hawking and Intel
Images Sources: http://www.astrosurf.com/luxorion/hawking-hommage9.htm http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/
#IntelNoTDC
• Acronym for “Assistive Context-Aware Toolkit”
• Enable people with disabilities to communicate through keyboard simulation, word prediction and speech synthesis
• Developed by Intel Labs together with Professor Hawking
• Customizable and Extensible
What is ACAT?
#IntelNoTDC
Open Source
http://01.org/ACAThttps://github.com/01org/acat
#IntelNoTDC
Tech Details
• Runs on Windows 7 or higher• Code entirely written in C# using Visual 2012 (works fine on 2015)
and Microsoft .NET Framework 4.5• Microsoft Office Interop Assemblies are used to interact with Office
apps• Some tools used to develop the project:• CodeMaid• StyleCop• ReSharper
#IntelNoTDC
Architecture
Applications
ACAT Core Library
ACAT Extension Library
Extensions
Extension Extension Extension ...
#IntelNoTDC
ApplicationsACAT Qwerty ACAT Abc ACAT Tryout
Talk Window
#IntelNoTDC
ACAT Core Library - ComponentsWord
Prediction* Management
UI ManagementAnimationsPanels Widgets Themes
Text-to-Speech
ManagementTalk Window Management
*ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)
#IntelNoTDC
ACAT Core Library - Agents
#IntelNoTDC
ACAT Core Library - Actuators
Actuator Manageme
nt
Image Sources: http://www.psfk.com/2013/02/intel-stephen-hawking.html & https://commons.wikimedia.org/wiki/File:Assistive_device.JPG
Actuators
Proximity Sensor Assistive Switch Camera
#IntelNoTDC
ACAT Extensions Library
• Allows developers to extend the functionalities from ACAT• Dynamically loaded during startup using reflection• Needs to implement specific interfaces/base classes
Extension Types
UI
Word Predictors
Agents
Actuators
Text-to-Speech Spell Checkers
#IntelNoTDC
Creating an Actuator Extension
• Create a new “Class Library” project and add references to “ACATCore.dll” and “ACATExtension.dll”
• Extend the classes “ActuatorBase” and “ActuatorSwitchBase” to interact with ACAT
• Implement the abstract methods and the use the callbacks (Init, Resume, Pause, Dispose) to run the code of your extension (threads, config, etc)
• Call the methods OnSwitchActivated (Hold/Press), OnSwitchDeactivated (Release) or OnSwitchTriggered(Click) to sinalize the interaction from the actuator
• Deploy your DLL to the extension folder and update the configuration files
#IntelNoTDC
Intel® RealSense™
Intel® RealSense™ SDK
Scene Understandin
gEnhanced
Photography
Augmented Reality
Person & Face Tracking
Gesture Recognition
Background Removal
Object Recognition and Tracking
Speech Recognition
3D Object Scanning
Hand & Blob Tracking
User Facing World Facing
#IntelNoTDC
Tracking (Position)Landmarks (78 landmarks)Pose / OrientationExpressions RecognitionPulse estimation
Intel® RealSense™ - Face Module
#IntelNoTDC
• Objective: create an actuator driven by facial expressions
• Components:• RealSenseExpressionTracker:
RealSense SDK implementation to perceive the expressions and notify the Actuator using events
• RealSenseActuator: translates the intensity of the expressions to a button behavior (pressed and released)
Intel® RealSense™ - ACAT PluginRealSense Actuator
ExtensionExpressionTracker
Actuator Switch
Actuator Manageme
nt
#IntelNoTDC
• The Camera actuator extension is a socket client that keeps looking for the “ACAT Vision” app, that has a socket server
• Simple protocol and it’s possible to customize using config files
• ACAT Vision is not yet open-source, but its code will be released in upcoming releases
Another way to connect an actuator
ACAT Vision (Server)
Camera Actuator (Client)
Actuator Manageme
nt
#IntelNoTDC
• SharpSenses* is an open source C# library that simplifies the development of Intel® RealSense™ SDK apps.
• Created and maintained by André Carlucci (Intel Black Belt & Software Innovator)
• Available on Github* and NuGet*
SharpSenses*
#IntelNoTDC
• The idea is to create a ‘fake’ server, that replicates the communication between “ACAT Vision” and ACAT
• To notify that the input was triggered, just send the string “gesture=trigger;action=CT;conf=75;time=<TIMESTAMP>”
• ACAT also sends some events back to the server (HideUI, ShowUI, ExitApp)
SharpSenses* + ACAT
ACAT Vision
Camera Actuator (Client)
Actuator Manageme
nt
‘Fake’ Server
SharpSenses
#IntelNoTDC
Summary & Call 4 Action
• ACAT is a really nice option to allow people with disabilities to communicate with others
• Open Source: You can fork and extend ACAT for your needs!• Extensions: let’s help other people with new actuators, applications
agents and new features • RealSense and SharpSenses can also drive acessibility apps.
Q/A
#IntelNoTDC
References
• Stephen Hawking and Intel• http
://iq.intel.com/behind-scenes-intel-keeps-stephen-hawking-talking/• http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/
• ACAT• https://github.com/01org/acat• http://01.org/ACAT
• RealSense: https://software.intel.com/en-us/realsense/home• SharpSenses: https://github.com/SharpSenses/SharpSenses
Thanks!
BAckup
#IntelNoTDC
ACAT Server
SharpSenses + ACAT
SharpSenses
bool canRun = true;AcatServer acatServer = new AcatServer();acatServer.Start();acatServer.AcatNotificationReceived += (sender, a) => { Console.WriteLine("Notification received from ACAT: {0}", a.NotificationType); if (a.NotificationType == AcatNotificationType.ExitApp) { canRun = false; }}; ICamera camera = Camera.Create(Capability.FaceTracking, Capability.FacialExpressionTracking);camera.Start(); camera.Face.LeftEye.Blink += (s,a) => { Console.WriteLine("SharpSenses: detected a blink from left eye."); acatServer.SendGestureMessage("CT", "trigger");}; while (canRun) { }acatServer.Stop();
top related