assistive context-aware toolkit - g3ictg3ict.org/design/js/tinymce/filemanager/userfiles/file/2015...

36
Assistive Context-Aware Toolkit Sai Prasad Anticipatory Computing Lab Intel Labs

Upload: dangkhanh

Post on 11-Jun-2018

252 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Assistive Context-Aware Toolkit Sai Prasad Anticipatory Computing Lab Intel Labs

Page 2: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Opening the Door for Help

Page 3: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Asking What Matters

Page 4: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Striving for Independence

Page 5: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

A Holistic Solution

Page 6: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

6

Assistive Context Aware Toolkit (ACAT)

Page 7: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Mute

Contextual Menus Based on app

Speak

Search Files

Assistive Context Aware Toolkit (ACAT)

Launch Applications

Main Control

Page 8: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Available on Open Source

Assistive Context-Aware Toolkit (ACAT)

8

http://01.org/ACAT

Page 9: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Architecture

Assistive Context-Aware Toolkit (ACAT)

• Applications only link with the Core and Extension libraries

• All extensions are dynamically discovered and loaded.

9

Overview

Page 10: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Assistive Context-Aware Toolkit (ACAT)

10

ACAT Libraries

Page 11: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Core Library

Assistive Context-Aware Toolkit (ACAT)

Contains all the core management components of ACAT

11

ACAT Libraries

Page 12: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Core Library Components

Assistive Context-Aware Toolkit (ACAT)

12

Animation Management Scans widgets on panel

Widget Management Manages widget layout and appearance

Panel Management Creates/Displays Panels

Theme Management Manages colors of Panels

ACAT Libraries

Word Prediction Management Manages word predictors* for next-word prediction

*ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)

Page 13: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Core Library Components (contd.)

Assistive Context-Aware Toolkit (ACAT)

13

Hello world

Text-to-Speech Management Converts text to speech

Talk Window Management Creates, manages the Talk window

ACAT Libraries

Page 14: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Core Library Components (contd.)

Assistive Context-Aware Toolkit (ACAT)

14

Agent Management Manages Application Agents which provides contextual information from applications

One Application Agent per application

• E.g. Notepad Application Agent, MS Word Application Agent

ACAT Libraries

Page 15: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Core Library Components (contd.)

Assistive Context-Aware Toolkit (ACAT) 15

Actuator Management Manages Actuators and handles trigger events

Proximity Sensor Assistive Switch Camera

Actuators

ACAT Libraries

Page 16: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Extension Library

Assistive Context-Aware Toolkit (ACAT)

Contains base classes for Application Agents and Utility classes

16

ACAT Libraries

Page 17: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Assistive Context-Aware Toolkit (ACAT)

17

Extensions

Page 18: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

ACAT Extensions

Assistive Context-Aware Toolkit (ACAT)

Extensions are plug-in’s that extend the capability of ACAT

18

Extensions

Page 19: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Extensions Discovery and Instantiation

Assistive Context-Aware Toolkit (ACAT)

ACAT Core loads Extensions on startup.

The manager for each Extension category traverses the depth of the top-level folder for that category.

1. Discovers all DLL’s under the folder

2. Enumerates all the .NET classes that implement the interface for the category.

- E.g. IWordPredictor for all Word Predictor extensions.

3. Caches the .NET Types of the extension classes

4. Uses .NET Reflection to create instances of the classes from the Types

19

Extensions

Page 20: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Assistive Context-Aware Toolkit (ACAT)

20

Extension

Development

Page 21: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Requirements

Assistive Context-Aware Toolkit (ACAT)

• Windows 7 or higher

• Visual Studio 2012 or higher • Code written entirely in C#

• Microsoft .NET Framework 4.5

• Microsoft Office Interop Assemblies • Bundled with Visual Studio Professional/Premium/Ultimate

• Optional Visual Studio Add-ons • StyleCop (free) to enforce .NET coding standards

• CodeMaid (free) to cleanup/reorganize the code

• ReSharper (paid) to optimize the code

21

Extension Dev.

Page 22: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Creating an Extension

Assistive Context-Aware Toolkit (ACAT)

1. Create a Visual Studio DLL project

2. Add references to ACATCore.dll and ACATExtension.dll.

3. Implement IExtension and the interface for the Extension category, e.g., IWordPredictor

4. Compile the DLL.

5. Deploy the DLL to the folder for the Extension category. E.g. Deploy a Word Predictor extension to <RunDir>\Extensions\Default\WordPredictors\MyWordPredictor\MyExtension.dll

6. Modify relevant ACAT XML files to configure the Extension.

• Depends on the Extension category

22

Extension Dev.

Page 23: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Extension Invoker

Assistive Context-Aware Toolkit (ACAT)

• Extension DLL’s are not directly referenced in any Project.

• Access methods/properties in the Extension objects through the Extension category Interface. E.g. IWordPredictor.Predict()

• To access custom methods/ properties

- Use .NET through the ExtensionInvoker helper class.

- Implement the IExtension interface.

- The GetInvoker() method returns an ExtensionInvoker object.

- Use this object to access custom methods/properties

- E.g. To get the name of the file selected from the FileBrowser extension through the SelectedFile custom property:

var file = fileBrowser.GetInvoker().GetStringValue("SelectedFile");

23

Extension Dev.

Page 24: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Assistive Context-Aware Toolkit (ACAT)

24

UI

Extensions

Page 25: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Overview

Assistive Context-Aware Toolkit (ACAT)

Three types of Panels

25

Scanners Menus Dialogs

UI

Examples:

Alphabet Scanner Cursor Scanner Punctuations Scanner . . .

Examples:

Contextual Menus ACAT Main Menu ACAT Settings Menu . . .

Examples:

General Settings Dialog Voice Settings Dialog Scanner Settings Dialog . . .

Page 26: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Panels

Assistive Context-Aware Toolkit (ACAT)

• All Panels are .NET Forms.

• Visual look-and-feel of the Panels, the animation sequences, the transition are all configured through a Panel Configuration file.

• Enables easy customization of the Panels without having to change the code

• Forms can be reused for different Panel configurations.

26

UI

Page 27: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Panel Configuration File

Assistive Context-Aware Toolkit (ACAT)

27

WIDGET ATTRIBUTES SECTION - Configure attributes such as the text, fonts,

colors

<WidgetAttributes>

<WidgetAttribute name="B1"

label="("

value="@CmdGoBack"

fontname="ACAT Icon"

fontsize="16"/>

<WidgetAttribute name="B2"

label="-"

value="-"

...

WIDGET LAYOUT SECTION

- Define widget hierarchy - Logically group widgets into boxes, rows

and columns

<Layout>

<Widget class="RowWidget" name="Row1">

<Widget class="ScannerButton" name="B1"/>

<Widget class="ScannerButton" name="B2"/>

...

...

<Widget class="ScannerButton" name="B7"/>

</Widget>

<Widget class="RowWidget" name="Row2">

...

...

</Widget>

</Layout>

ANIMATIONS SECTION

- Define scanning sequences - Transition between scanning sequences - Actions to initiate when the user activates a

widget

<Animations>

<Animation name="TopLevelRotation"

start="true"

iterations="@HalfScanIterations">

<Widget name="Row1“ onSelect="transition(Row1)"/>

<Widget name="Row2“ onSelect="transition(Row2)"/>

</Animation>

<Animation name="Row1"

iterations="@ColumnScanIterations"

onEnd="transition(TopLevelRotation)">

<Widget name="B1"/>

...

...

<Widget name="B14"/>

</Animation>

</Animations>

UI

Page 28: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Assistive Context-Aware Toolkit (ACAT)

28

Word Predictor

Extensions

Page 29: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Overview

Assistive Context-Aware Toolkit (ACAT)

• Auto-complete and next-word prediction speed up typing. - ACAT sends the text from the current sentence to the Word Predictor

- Word predictor uses this to make smart next-word prediction.

- ACAT displays a list of suggested words returned by the Word Predictor

- User selects words from the list.

• On average, < 20% of characters need to be typed.

29

Word Predictors

Page 30: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Creating a Word Predictor Extension

Assistive Context-Aware Toolkit (ACAT)

30

1. Create a Visual Studio DLL project, say, MyWordPredictor

2. Add references to ACATCore.dll and ACATExtension.dll.

3. Add a class MyWordPredictor derived from IWordPredictor

4. Implement these functions in MyWordPredictor:

- Init() to initialize the Word Predictor

- Predict() to return a list of suggested words. The text from the current sentence is passed to this function

- Learn()to enhance the Word Prediction model by learning the user’s writing style

- Dispose() to dispose the Word Predictor objects

5. Build and deploy MyWordPredictor.dll to:

<RunDir>\Extensions\Default\WordPredictors\MyWordPredictor

Word Predictors

Page 31: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Assistive Context-Aware Toolkit (ACAT)

31

Actuator

Extensions

Page 32: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Actuators Overview

Assistive Context-Aware Toolkit (ACAT)

• Actuators are used to drive the UI.

• Triggers from Actuators are translated into actions.

- E.g. Typing a letter, choosing a menu item.

• Each Actuators is an ACAT Extension.

• Actuators interface with the underlying hardware. For e.g.

- USB for HID Devices or Webcams

• ACAT has helper classes to interface with

- USB HID Devices

- TCP/IP Servers/Clients

• Support included for facial gesture recognition, proximity sensor, accelerometer ring

32

Actuators

Page 33: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Actuator Switches

Assistive Context-Aware Toolkit (ACAT)

• Each Actuator is composed of Switches.

- E.g. Left and right mouse buttons are two switches on a mouse

- For gesture input, each gesture can be treated as a Switch

• Events are associated with Switches

- Switch Down, analogous to Mouse LButtonDown

- Switch Up, analogous to Mouse LButtonUp

- Switch Trigger, analogous to Mouse LButtonClick

• Commands can be mapped with Switch trigger events.

- Commands executed when the trigger event occurs

- Command Examples: Select highlighted UI element, Delete previous word, select first word from prediction list

- Enables multi-modal interactions by mapping gestures to commands.

• Commands mapped through XML configuration files.

33

Actuators

Page 34: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Creating an Actuator Extension

Assistive Context-Aware Toolkit (ACAT)

1. Create a Visual Studio DLL project, say, MyActuator

2. Add references to ACATCore.dll and ACATExtension.dll.

3. Add a class MyActuator derived from ActuatorBase

4. Add a class MyActuatorSwitch derived from ActuatorSwitchBase

5. Implement these functions in MyActuator: - Init() to initialize the Actuator

- CreateSwitch() to return a MyActuatorSwitch object - OnSwitchActivated(), OnSwitchDeactivated(),

OnSwitchTriggered()to trigger Actuator switch events. - Dispose() to dispose the Actuator objects

6. Build and deploy MyActuator.dll to: <RunDir>\Extensions\Default\Actuators\MyActuator

34

Actuators

Page 35: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

Proximity Sensor Actuator

Assistive Context-Aware Toolkit (ACAT)

35

Page 36: Assistive Context-Aware Toolkit - G3ictg3ict.org/design/js/tinymce/filemanager/userfiles/File/2015 Summit... · Requirements Assistive Context-Aware Toolkit (ACAT) •Windows 7 or

RealSense Actuator

Assistive Context-Aware Toolkit (ACAT)

36