icra: intelligent platform for collaboration and interaction
TRANSCRIPT
ICRA: Intelligent Platform for Collaboration and Interaction
Lukas Tencer and Marta Reznakova
Synchromedia Laboratory
École de Technologie Supérieure
Outline
Introduction
Collaboration
Context and definitions
Characteristics of collaborative work
Application domain and functionality
Architecture of collaborative work
ICRA
Retrieval
New retrieval paradigms
Sketch-based retrieval
Photo-based retrieval
Interaction
Gesture recognition
Online learning
Fuzzy Models, ART Network
Synchromedia Laboratory
Director: Professeur Mohamed CHERIET
Mission: Develop an advanced network infrastructure that integrates
various aspects of multimodal perceptual information to support
collaborative work
Main Projets
Image Processing and historic documents
Handwriting recognition
Indian Ocean World (IOW): Image Enhancement, retrieval, interaction
Virtualization and Green IT, GreenStar and Green Telco Cloud
Open positions and Web Address
http://www.synchromedia.ca
COLLABORATION
Context and definitions
CSCW: Computer-supported collaborative work
Allows multiple people to work together through a technological
infrastructure
Groupware: software which aims to foster collaborative work mode
Why CSCW?
Globalization: Linking people and build virtual teams
Teleworking and Mobility
Saves time: Booking meeting rooms, travel
Increased productivity through the use of appropriate tools
Reduces hierarchical barriers and strengthens team spirit
Better management and sharing of knowledge
Source: Lewis R. (1995). « Editorial : Professional learning », Journal of Computer Assisted Learning, 11 (4), 193-195.
Knowledge: core and proximal development
• In community core knowledge of individuals overlap and border areas
coincide with core areas of others.
Source: Kuutti K. (1996). « Activity Theory as a Potential Framework for Human-Computer Interaction Research », in Context and consciousness : Activity theory and human computer interaction (ed. B.A. Nardi) p. 17-44. Cambridge, MA : MIT Press.
The basic structure of an activity
Features / Functions of groupware
Communication
Exchange of information between users
Coordination
Arrangement of tasks / Managing User Roles
Production
Editing Documents
Whiteboards Communication
ProductionCoordination
Application areas and features
Areas of application
Games
Workflow
Teaching and research
Shared working spaces
ConnectNow (Adobe)
WebEx (Cisco)
BigBlueButton (Open source)
Some features
Chat
Audio and video
Viewing and editing documents
Whiteboard
Sharing of programs
Concepts and constraints associated with groupware
Concepts
Group
Session
Communication and sharing
Consistency
Group consciousness
Security and privat life
Identity theft
Upload of personal documents
Constraints
WYSIWIS: What You See Is What I See, Real-time synchronization
Response time and almost zero delay
Architecture: Distribution of the different components of groupware
Source: Jörg Roth, A taxonomy for synchronous groupware architectures
Advantages / Disadvantages of distributed vs.
centralized architecture??
Distributed vs. Centralized Architecture
Centralized architecture
Ease of implementation
Higher response time
Distributed architecture
Implementation complexity
Fast synchronization of the data model
Centralized architecture is almost always more efficient than
distributed architecture, regarding audio and video streaming! Why?
Architecture: Models according toPatterson 1/4
Different levels of consistency states
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications
Architecture: Models according toPatterson 2/4
Consistency of applications after the states sharing
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications
Architecture: Models according toPatterson 3/4
Consistency of applications after the synchronization state
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications
Architecture: Models according toPatterson 4/4
Hybrid architecture
Source: John F. Patterson, A Taxonomy of Architectures for Synchronous Groupware Applications
Case: Groupware and Software Engineering
Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering Tools Using CAISE, an Architecture for Supporting Tool Development
Case: Groupware and Software Engineering
Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering Tools Using CAISE, an Architecture for Supporting Tool Development
Case: Groupware and Software Engineering
Source: Carl Cook, Neville Churcher, Constructing Real-Time Collaborative Software Engineering Tools Using CAISE, an Architecture for Supporting Tool Development
Problems linked to this architecture??
Collaborative editing and resolution of conflicts
Source: Abdessamad Imine, Flexible Concurrency Control for Real-time Collaborative Editors
Solutions??
Collaborative editing and resolution of conflicts
Source: Abdessamad Imine, Flexible Concurrency Control for Real-time Collaborative Editors
• Maintain a history of changes and identify dependencies to treat
History: Phase 1
Tele-education project
Technology deployed on the network of the Université du Québec
(VLAN linking ETS, Téluq-Mtl, Qc-Téluq, UQAM)
Collection of collaborative tools: chat, whiteboard, audio / video,
virtual laboratory
History: Phase 1
History: Phase 2
The technological infrastructure is extended to connect theConcordia and Waterloo, via networks RISQ, CA * net 4 andInternet2.
History: Phase 2
New Vision
Document Sharing
Sharing desktop applications
New Infrastracture
Multimedial room at Synchromedia
Multimedial room at Synchromedia
ICRA
ICRA: Snapshots
ICRA: Snapshots
ICRA: Snapshots
ICRA: Use case
ICRA: Processes
ICRA: Architecture of N Layers
ICRA: video encoding
Using the MPEG-4 H.264
10/15 frames per second
Consistent image quality (80%) with variable flow
Opportunity to improve performance by adjusting the image quality.
ICRA: audio encoding
Used codec: Speex
Open-source and free
Sampling rate
8 kHz, 16 kHz, 32 kHz
Bit-rate
2.15 kbps to 44 kbps
Quality
Variable between 0 and 10
Lossy data compression
Variable Bit-Rate VBR
Allows the codec to adapt its speed according to the difficulty of encoding
(eg vowels require greater throughput for a given quality)
Detection of voice and discontinuing of transmission
ICRA: Future versions and Research
Whiteboard
Sharing desktop applications
Image processing
Handwriting recognition and handwritten annotations
Document Retrieval
Gesture Interaction
Research
Demo
Retrieval
Document retrieval for collaborative work
Motivation: To collaborate based on large document collections, we
should be able to retrieve these documents first
Main components of document retrieval system:
Feature extraction
Distance metric
Semantic Representation
Indexing
Retrieval
User Feedback
Query refinement
Our vision: Explore new paradigms for document retrieval
New retrieval paradigms for documents
Sketch-based document retrieval
Mission: Based on user’s sketch, retrieve document’s which include
graphic similar to user’s sketch
Photo-based document retrieval
Mission: Based on visual example (mobile phone picture) retrieve
relevant documents
Sketch-based document retrieval
Mission: Based on user’s sketch, retrieve document’s which include
graphic similar to user’s sketch
SBIR challenges
Preprocessing
Sketch representation
Invariance
Semantic vs. Visual Similarity
User Feedback / Query refinement
Result Visualization
f(Q,I)
SBIR representation and invariance
Gradient-based techniques:
HoG, EHD, SIFT
Angular Techniques
Angular partitioning
SBIR similarity
L1, L2, Entropy, what next?
Neighbourhood-based distance (Graph Transduction)
(Yang et al. 2008) (Bai, Yang, Latecki, Liu, & Tu, 2010)
52
SBIR Graph Transduction
53
SBIR Indexing and Retrieval
Representation for indexing
Bag-of-words
Topic Model - LSI
Spatial pyramid matching
Effective indexing, structure of index?
Vector space
Inverted index
Hash table
Combine Visual and Semantic similarity
RankBoost algorithm
User-specific model
Index Representation:
Vector Space
Bag-of-words + inverted index
Constellation Models
Retrieved Results:
Best n-results
Clustering of Results
Feedback:
Implicit: Select relevant images
Explicit: Temporal information
Query refinement
54
SBIR final system
Photo-based retrieval
Input: Photo from your mobile camera, database of document
Output: Matching Document
Photo-based retrieval approach
Distribution of connected components in local neighbourhood
Interaction
Interaction by handwritten gestures Where to use?
Tablets, whiteboards, smartphones, ...
Simple shapes, letters, ...
Perform some action using a connected gesture
Interaction by handwritten gestures Why to use?
Simplify the mouse/keyboard access
Natural interaction
Fast interaction
Interaction by handwritten gestures Example
[x, y] coordinates, pressure, time, penUps, penDowns, ...
Ordered sequences
Handwritten gesture recognition To recognize gestures (or anything else), the ‘how to’
needs to be learned
Handwritten gesture recognition To recognize gestures (or anything else), the ‘how to’
needs to be learned
Problem?
For user-friendly and adaptive system no predefinition should be needed. ->Where to get the database if it is not known? How to learn our system? -> online learning
Online learning For each new gesture:
Recognize the gesture
Perform learning of the system (or a model)
Go back to next gesture
Data (gestures) are treated sequentially (incremental learning)
Usually no revisit of old data is allowed (== forget the processed data)
Online learning Problems?
Low accuracy at the beginning <- learning from few examples
Difficult generalization, since the data are not seen as a whole package; future data are not known
Online learning User-friendly applications:
No predefinition of classes, their number, ...
Starting from a scratch
=> System must be capable of learning all the parameters it is using, if any.
Our solution Fuzzy models -> applying fuzzy logic into simple
neural networks
Rules: IF a is ‘P’ THEN ‘B’
Taking the opinion of all rules
How to generate the rules?
Our solution ART (Adaptive Resonance Theory)
Incremental clustering (unsupervised) method for detecting similarities
Starting from a scratch
=> can generate rules for fuzzy models
Questions?
Director of the Research Laboratory Synchromedia
Mohamed CHERIET