assistive context-aware toolkit - g3ictg3ict.org/design/js/tinymce/filemanager/userfiles/file/2015...

Assistive Context-Aware Toolkit Sai Prasad Anticipatory Computing Lab Intel Labs

Opening the Door for Help

Asking What Matters

Striving for Independence

A Holistic Solution

6

Assistive Context Aware Toolkit (ACAT)

Mute

Contextual Menus Based on app

Speak

Search Files

Assistive Context Aware Toolkit (ACAT)

Launch Applications

Main Control

ACAT Available on Open Source

Assistive Context-Aware Toolkit (ACAT)

8

http://01.org/ACAT

ACAT Architecture

Assistive Context-Aware Toolkit (ACAT)

• Applications only link with the Core and Extension libraries

• All extensions are dynamically discovered and loaded.

9

Overview

Assistive Context-Aware Toolkit (ACAT)

10

ACAT Libraries

ACAT Core Library

Assistive Context-Aware Toolkit (ACAT)

Contains all the core management components of ACAT

11

ACAT Libraries

ACAT Core Library Components

Assistive Context-Aware Toolkit (ACAT)

12

Animation Management Scans widgets on panel

Widget Management Manages widget layout and appearance

Panel Management Creates/Displays Panels

Theme Management Manages colors of Panels

ACAT Libraries

Word Prediction Management Manages word predictors* for next-word prediction

*ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)

ACAT Core Library Components (contd.)

Assistive Context-Aware Toolkit (ACAT)

13

Hello world

Text-to-Speech Management Converts text to speech

Talk Window Management Creates, manages the Talk window

ACAT Libraries

ACAT Core Library Components (contd.)

Assistive Context-Aware Toolkit (ACAT)

14

Agent Management Manages Application Agents which provides contextual information from applications

One Application Agent per application

• E.g. Notepad Application Agent, MS Word Application Agent

ACAT Libraries

ACAT Core Library Components (contd.)

Assistive Context-Aware Toolkit (ACAT) 15

Actuator Management Manages Actuators and handles trigger events

Proximity Sensor Assistive Switch Camera

Actuators

ACAT Libraries

ACAT Extension Library

Assistive Context-Aware Toolkit (ACAT)

Contains base classes for Application Agents and Utility classes

16

ACAT Libraries

Assistive Context-Aware Toolkit (ACAT)

17

Extensions

ACAT Extensions

Assistive Context-Aware Toolkit (ACAT)

Extensions are plug-in’s that extend the capability of ACAT

18

Extensions

Extensions Discovery and Instantiation

Assistive Context-Aware Toolkit (ACAT)

ACAT Core loads Extensions on startup.

The manager for each Extension category traverses the depth of the top-level folder for that category.

1. Discovers all DLL’s under the folder

2. Enumerates all the .NET classes that implement the interface for the category.

- E.g. IWordPredictor for all Word Predictor extensions.

3. Caches the .NET Types of the extension classes

4. Uses .NET Reflection to create instances of the classes from the Types

19

Extensions

Assistive Context-Aware Toolkit (ACAT)

20

Extension

Development

Requirements

Assistive Context-Aware Toolkit (ACAT)

• Windows 7 or higher

• Visual Studio 2012 or higher • Code written entirely in C#

• Microsoft .NET Framework 4.5

• Microsoft Office Interop Assemblies • Bundled with Visual Studio Professional/Premium/Ultimate

• Optional Visual Studio Add-ons • StyleCop (free) to enforce .NET coding standards

• CodeMaid (free) to cleanup/reorganize the code

• ReSharper (paid) to optimize the code

21

Extension Dev.

Creating an Extension

Assistive Context-Aware Toolkit (ACAT)

1. Create a Visual Studio DLL project

2. Add references to ACATCore.dll and ACATExtension.dll.

3. Implement IExtension and the interface for the Extension category, e.g., IWordPredictor

4. Compile the DLL.

5. Deploy the DLL to the folder for the Extension category. E.g. Deploy a Word Predictor extension to <RunDir>\Extensions\Default\WordPredictors\MyWordPredictor\MyExtension.dll

6. Modify relevant ACAT XML files to configure the Extension.

• Depends on the Extension category

22

Extension Dev.

Extension Invoker

Assistive Context-Aware Toolkit (ACAT)

• Extension DLL’s are not directly referenced in any Project.

• Access methods/properties in the Extension objects through the Extension category Interface. E.g. IWordPredictor.Predict()

• To access custom methods/ properties

- Use .NET through the ExtensionInvoker helper class.

- Implement the IExtension interface.

- The GetInvoker() method returns an ExtensionInvoker object.

- Use this object to access custom methods/properties

- E.g. To get the name of the file selected from the FileBrowser extension through the SelectedFile custom property:

var file = fileBrowser.GetInvoker().GetStringValue("SelectedFile");

23

Extension Dev.

Assistive Context-Aware Toolkit (ACAT)

24

UI

Extensions

Overview

Assistive Context-Aware Toolkit (ACAT)

Three types of Panels

25

Scanners Menus Dialogs

UI

Examples:

Alphabet Scanner Cursor Scanner Punctuations Scanner . . .

Examples:

Contextual Menus ACAT Main Menu ACAT Settings Menu . . .

Examples:

General Settings Dialog Voice Settings Dialog Scanner Settings Dialog . . .

Panels

Assistive Context-Aware Toolkit (ACAT)

• All Panels are .NET Forms.

• Visual look-and-feel of the Panels, the animation sequences, the transition are all configured through a Panel Configuration file.

• Enables easy customization of the Panels without having to change the code

• Forms can be reused for different Panel configurations.

26

UI

Panel Configuration File

Assistive Context-Aware Toolkit (ACAT)

27

WIDGET ATTRIBUTES SECTION - Configure attributes such as the text, fonts,

colors

<WidgetAttributes>

<WidgetAttribute name="B1"

label="("

value="@CmdGoBack"

fontname="ACAT Icon"

fontsize="16"/>

<WidgetAttribute name="B2"

label="-"

value="-"

...

WIDGET LAYOUT SECTION

- Define widget hierarchy - Logically group widgets into boxes, rows

and columns

<Layout>

<Widget class="RowWidget" name="Row1">

<Widget class="ScannerButton" name="B1"/>

<Widget class="ScannerButton" name="B2"/>

...

...

<Widget class="ScannerButton" name="B7"/>

</Widget>

<Widget class="RowWidget" name="Row2">

...

...

</Widget>

</Layout>

ANIMATIONS SECTION

- Define scanning sequences - Transition between scanning sequences - Actions to initiate when the user activates a

widget

<Animations>

<Animation name="TopLevelRotation"

start="true"

iterations="@HalfScanIterations">

<Widget name="Row1“ onSelect="transition(Row1)"/>

<Widget name="Row2“ onSelect="transition(Row2)"/>

</Animation>

<Animation name="Row1"

iterations="@ColumnScanIterations"

onEnd="transition(TopLevelRotation)">

<Widget name="B1"/>

...

...

<Widget name="B14"/>

</Animation>

</Animations>

UI

Assistive Context-Aware Toolkit (ACAT)

28

Word Predictor

Extensions

Overview

Assistive Context-Aware Toolkit (ACAT)

• Auto-complete and next-word prediction speed up typing. - ACAT sends the text from the current sentence to the Word Predictor

- Word predictor uses this to make smart next-word prediction.

- ACAT displays a list of suggested words returned by the Word Predictor

- User selects words from the list.

• On average, < 20% of characters need to be typed.

29

Word Predictors

Creating a Word Predictor Extension

Assistive Context-Aware Toolkit (ACAT)

30

1. Create a Visual Studio DLL project, say, MyWordPredictor

2. Add references to ACATCore.dll and ACATExtension.dll.

3. Add a class MyWordPredictor derived from IWordPredictor

4. Implement these functions in MyWordPredictor:

- Init() to initialize the Word Predictor

- Predict() to return a list of suggested words. The text from the current sentence is passed to this function

- Learn()to enhance the Word Prediction model by learning the user’s writing style

- Dispose() to dispose the Word Predictor objects

5. Build and deploy MyWordPredictor.dll to:

<RunDir>\Extensions\Default\WordPredictors\MyWordPredictor

Word Predictors

Assistive Context-Aware Toolkit (ACAT)

31

Actuator

Extensions

Actuators Overview

Assistive Context-Aware Toolkit (ACAT)

• Actuators are used to drive the UI.

• Triggers from Actuators are translated into actions.

- E.g. Typing a letter, choosing a menu item.

• Each Actuators is an ACAT Extension.

• Actuators interface with the underlying hardware. For e.g.

- USB for HID Devices or Webcams

• ACAT has helper classes to interface with

- USB HID Devices

- TCP/IP Servers/Clients

• Support included for facial gesture recognition, proximity sensor, accelerometer ring

32

Actuators

Actuator Switches

Assistive Context-Aware Toolkit (ACAT)

• Each Actuator is composed of Switches.

- E.g. Left and right mouse buttons are two switches on a mouse

- For gesture input, each gesture can be treated as a Switch

• Events are associated with Switches

- Switch Down, analogous to Mouse LButtonDown

- Switch Up, analogous to Mouse LButtonUp

- Switch Trigger, analogous to Mouse LButtonClick

• Commands can be mapped with Switch trigger events.

- Commands executed when the trigger event occurs

- Command Examples: Select highlighted UI element, Delete previous word, select first word from prediction list

- Enables multi-modal interactions by mapping gestures to commands.

• Commands mapped through XML configuration files.

33

Actuators

Creating an Actuator Extension

Assistive Context-Aware Toolkit (ACAT)

1. Create a Visual Studio DLL project, say, MyActuator

2. Add references to ACATCore.dll and ACATExtension.dll.

3. Add a class MyActuator derived from ActuatorBase

4. Add a class MyActuatorSwitch derived from ActuatorSwitchBase

5. Implement these functions in MyActuator: - Init() to initialize the Actuator

- CreateSwitch() to return a MyActuatorSwitch object - OnSwitchActivated(), OnSwitchDeactivated(),

OnSwitchTriggered()to trigger Actuator switch events. - Dispose() to dispose the Actuator objects

6. Build and deploy MyActuator.dll to: <RunDir>\Extensions\Default\Actuators\MyActuator

34

Actuators

Proximity Sensor Actuator

Assistive Context-Aware Toolkit (ACAT)

35

RealSense Actuator

Assistive Context-Aware Toolkit (ACAT)

36