icra: intelligent platform for collaboration and interaction

ICRA: Intelligent Platform for Collaboration and Interaction

Lukas Tencer and Marta Reznakova

Synchromedia Laboratory

École de Technologie Supérieure

Outline

Introduction

Collaboration

Context and definitions

Characteristics of collaborative work

Application domain and functionality

Architecture of collaborative work

ICRA

Retrieval

New retrieval paradigms

Sketch-based retrieval

Photo-based retrieval

Interaction

Gesture recognition

Online learning

Fuzzy Models, ART Network

Synchromedia Laboratory

Director: Professeur Mohamed CHERIET

Mission: Develop an advanced network infrastructure that integrates

various aspects of multimodal perceptual information to support

collaborative work

Main Projets

Image Processing and historic documents

Handwriting recognition

Indian Ocean World (IOW): Image Enhancement, retrieval, interaction

Virtualization and Green IT, GreenStar and Green Telco Cloud

Open positions and Web Address

http://www.synchromedia.ca

COLLABORATION

Context and definitions

CSCW: Computer-supported collaborative work

Allows multiple people to work together through a technological

infrastructure

Groupware: software which aims to foster collaborative work mode

Why CSCW?

Globalization: Linking people and build virtual teams

Teleworking and Mobility

Saves time: Booking meeting rooms, travel

Increased productivity through the use of appropriate tools

Reduces hierarchical barriers and strengthens team spirit

Better management and sharing of knowledge

Source: Lewis R. (1995). « Editorial : Professional learning », Journal of Computer Assisted Learning, 11 (4), 193-195.

Knowledge: core and proximal development

• In community core knowledge of individuals overlap and border areas

coincide with core areas of others.

Source: Kuutti K. (1996). « Activity Theory as a Potential Framework for Human-Computer Interaction Research », in Context and consciousness : Activity theory and human computer interaction (ed. B.A. Nardi) p. 17-44. Cambridge, MA : MIT Press.

The basic structure of an activity

Features / Functions of groupware

Communication

Exchange of information between users

Coordination

Arrangement of tasks / Managing User Roles

Production

Editing Documents

Whiteboards Communication

ProductionCoordination

Application areas and features

Areas of application

Games

Workflow

Teaching and research

Shared working spaces

ConnectNow (Adobe)

WebEx (Cisco)

BigBlueButton (Open source)

Some features

Chat

Audio and video

Viewing and editing documents

Whiteboard

Sharing of programs

Concepts and constraints associated with groupware

Concepts

Group

Session

Communication and sharing

Consistency

Group consciousness

Security and privat life

Identity theft

Upload of personal documents

Constraints

WYSIWIS: What You See Is What I See, Real-time synchronization

Response time and almost zero delay

Architecture: Distribution of the different components of groupware

Source: Jörg Roth, A taxonomy for synchronous groupware architectures

Advantages / Disadvantages of distributed vs.

centralized architecture??

Distributed vs. Centralized Architecture

Centralized architecture

Ease of implementation

Higher response time

Distributed architecture

Implementation complexity

Fast synchronization of the data model

Centralized architecture is almost always more efficient than

distributed architecture, regarding audio and video streaming! Why?

Architecture: Models according toPatterson 1/4

Different levels of consistency states

Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

Architecture: Models according toPatterson 2/4

Consistency of applications after the states sharing

Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

Architecture: Models according toPatterson 3/4

Consistency of applications after the synchronization state

Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

Architecture: Models according toPatterson 4/4

Hybrid architecture

Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications

Case: Groupware and Software Engineering

Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering Tools Using CAISE, an Architecture for Supporting Tool Development

Case: Groupware and Software Engineering

Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering Tools Using CAISE, an Architecture for Supporting Tool Development

Case: Groupware and Software Engineering

Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering Tools Using CAISE, an Architecture for Supporting Tool Development

Problems linked to this architecture??

Collaborative editing and resolution of conflicts

Source: Abdessamad Imine, Flexible Concurrency Control for Real-time Collaborative Editors

Solutions??

Collaborative editing and resolution of conflicts

Source: Abdessamad Imine, Flexible Concurrency Control for Real-time Collaborative Editors

• Maintain a history of changes and identify dependencies to treat

History: Phase 1

Tele-education project

Technology deployed on the network of the Université du Québec

(VLAN linking ETS, Téluq-Mtl, Qc-Téluq, UQAM)

Collection of collaborative tools: chat, whiteboard, audio / video,

virtual laboratory

History: Phase 1

History: Phase 2

The technological infrastructure is extended to connect theConcordia and Waterloo, via networks RISQ, CA * net 4 andInternet2.

History: Phase 2

New Vision

Document Sharing

Sharing desktop applications

New Infrastracture

Multimedial room at Synchromedia

Multimedial room at Synchromedia

ICRA

ICRA: Snapshots

ICRA: Snapshots

ICRA: Snapshots

ICRA: Use case

ICRA: Processes

ICRA: Architecture of N Layers

ICRA: video encoding

Using the MPEG-4 H.264

10/15 frames per second

Consistent image quality (80%) with variable flow

Opportunity to improve performance by adjusting the image quality.

ICRA: audio encoding

Used codec: Speex

Open-source and free

Sampling rate

8 kHz, 16 kHz, 32 kHz

Bit-rate

2.15 kbps to 44 kbps

Quality

Variable between 0 and 10

Lossy data compression

Variable Bit-Rate VBR

Allows the codec to adapt its speed according to the difficulty of encoding

(eg vowels require greater throughput for a given quality)

Detection of voice and discontinuing of transmission

ICRA: Future versions and Research

Whiteboard

Sharing desktop applications

Image processing

Handwriting recognition and handwritten annotations

Document Retrieval

Gesture Interaction

Research

Demo

Retrieval

Document retrieval for collaborative work

Motivation: To collaborate based on large document collections, we

should be able to retrieve these documents first

Main components of document retrieval system:

Feature extraction

Distance metric

Semantic Representation

Indexing

Retrieval

User Feedback

Query refinement

Our vision: Explore new paradigms for document retrieval

New retrieval paradigms for documents

Sketch-based document retrieval

Mission: Based on user’s sketch, retrieve document’s which include

graphic similar to user’s sketch

Photo-based document retrieval

Mission: Based on visual example (mobile phone picture) retrieve

relevant documents

Sketch-based document retrieval

Mission: Based on user’s sketch, retrieve document’s which include

graphic similar to user’s sketch

SBIR challenges

Preprocessing

Sketch representation

Invariance

Semantic vs. Visual Similarity

User Feedback / Query refinement

Result Visualization

f(Q,I)

SBIR representation and invariance

Gradient-based techniques:

HoG, EHD, SIFT

Angular Techniques

Angular partitioning

SBIR similarity

L1, L2, Entropy, what next?

Neighbourhood-based distance (Graph Transduction)

(Yang et al. 2008) (Bai, Yang, Latecki, Liu, & Tu, 2010)

52

SBIR Graph Transduction

53

SBIR Indexing and Retrieval

Representation for indexing

Bag-of-words

Topic Model - LSI

Spatial pyramid matching

Effective indexing, structure of index?

Vector space

Inverted index

Hash table

Combine Visual and Semantic similarity

RankBoost algorithm

User-specific model

Index Representation:

Vector Space

Bag-of-words + inverted index

Constellation Models

Retrieved Results:

Best n-results

Clustering of Results

Feedback:

Implicit: Select relevant images

Explicit: Temporal information

Query refinement

54

SBIR final system

Photo-based retrieval

Input: Photo from your mobile camera, database of document

Output: Matching Document

Photo-based retrieval approach

Distribution of connected components in local neighbourhood

Interaction

Interaction by handwritten gestures Where to use?

Tablets, whiteboards, smartphones, ...

Simple shapes, letters, ...

Perform some action using a connected gesture

Interaction by handwritten gestures Why to use?

Simplify the mouse/keyboard access

Natural interaction

Fast interaction

Interaction by handwritten gestures Example

[x, y] coordinates, pressure, time, penUps, penDowns, ...

Ordered sequences

Handwritten gesture recognition To recognize gestures (or anything else), the ‘how to’

needs to be learned

Handwritten gesture recognition To recognize gestures (or anything else), the ‘how to’

needs to be learned

Problem?

For user-friendly and adaptive system no predefinition should be needed. ->Where to get the database if it is not known? How to learn our system? -> online learning

Online learning For each new gesture:

Recognize the gesture

Perform learning of the system (or a model)

Go back to next gesture

Data (gestures) are treated sequentially (incremental learning)

Usually no revisit of old data is allowed (== forget the processed data)

Online learning Problems?

Low accuracy at the beginning <- learning from few examples

Difficult generalization, since the data are not seen as a whole package; future data are not known

Online learning User-friendly applications:

No predefinition of classes, their number, ...

Starting from a scratch

=> System must be capable of learning all the parameters it is using, if any.

Our solution Fuzzy models -> applying fuzzy logic into simple

neural networks

Rules: IF a is ‘P’ THEN ‘B’

Taking the opinion of all rules

How to generate the rules?

Our solution ART (Adaptive Resonance Theory)

Incremental clustering (unsupervised) method for detecting similarities

Starting from a scratch

=> can generate rules for fuzzy models

Questions?

Director of the Research Laboratory Synchromedia

Mohamed CHERIET

mohamed.cheriet@etsmtl.ca