1/26

Sharing multimedia and context Sharing multimedia and context information between mobile terminalsinformation between mobile terminals

Soft computing Laboratory

Yonsei University

박문희Dec 5, 2006

2006 지식기반 시스템 수업 발표 자료 : 정보생성

International Symposium on Applications and the Internet (SAINT'06), pp.124-129, 23-27 January 2006.

T. Horozov, N. Narasimhan, and V. Vasudevan

2/26

OutlineOutline

• Introduction

• Techniques for creating and sharing multimedia– Crossing application boundaries– Maps

• Context information

• Prototype

• Summary and discussion

3/26

IntroductionIntroduction

• Sharing of experiences using mobile is becoming more common

• Due to physical limitations of mobile terminals– Requiring particular UI solution to present and process multimedia

• Context awareness of mobile terminals– Sharing and presenting contexts by showing context of their members– e.g context based call operation and messaging

• an approach for a user interface for mobile terminals – Sharing multimedia and context information– Map-based interface and domain object model-based user interface

4/26

Crossing application boundariesCrossing application boundaries

• Distinct applications– Map-based positioning , taking photos or video shots, playing the media– sharing the media files created and showing the current context of each user

• Should not be any separate applications, but objects and operations that can manipulate those objects

• The Naked Objects framework– maintaining correspondence of the domain or business object model and

the UI

• Our object model for enabling multimedia communication– People in the community– The multimedia files they create and share– A shared map acting as a container object for people and multimedia files

5/26

MapsMaps

• Characters of geographical maps– Direct representations of the real world already familiar to users– Useful for presenting and finding electronic services having an

unambiguous geographical location– The ability to visualize heterogeneous objects being either physical like

people or immaterial like video clips, if including location information

• Utilize this feature in this user interface– Sharing context information and multimedia objects on a map– Minimal user efforts to communicate their position and the context– The oldest media objects can be removed from the map

6/26

• Represent the current state of the object or its environment as pictures

• Classification of UI pictures for small interface– Iconic– Index– Symbolic picture

• In this system,– Iconic picture like Table 1.– First class

user activity

– Second class environment device activity

Table 1. Context information with classes used in user interface

Context informationContext information

7/26

PrototypePrototype

• Context-aware map-based interface for accessing situated services with mobile terminals

• On the Compaq iPAQ3660 PDA– Positioning: via WLAN– Context based control: via an

external sensor

• Exploring the Naked Objecs framework– for extending the Naked

Objects platform by implementing an OVM for PocketPC style devices

• Fig 1. Screenshots of a UI

8/26

Summary and discussionSummary and discussion

• UI solution for mobile terminals presenting and sharing multimedia with context information

• UI solution– Sharing interesting from the environment– Multimedia data are presented on the map as icons– Online sharing of context information (activity, device and environment) with

simple , not descriptive symbols

• Future works– Continuing the integration of the map interface to the Naked Objects

platform– Usability test in real usage situations

9/26

ComicDiary: Representing Individual ComicDiary: Representing Individual Experience in a Comics StyleExperience in a Comics Style

Soft computing Laboratory

Yonsei University

박문희Dec 5, 2006

2006 지식기반 시스템 수업 발표 자료 : 정보생성

UbiComp 2002: Ubiquitous Computing : 4th International Conference, Göteborg, Sweden, September 29 - October 1, 2002.

Yasuyuki Sumi, Ryuuki Sakamoto, Keiko Nakao, Kenji Mase

10/26

OutlineOutline

• Associative work

• Introduction

• Using comics to exchange individual experiences

• Related works

• Implementation of ComicDiary– System architecture of ComicDiary– Story generating – Rendering comic frames

• Comic examples and user evaluations– Examples of generated comic diaries– User evaluations and discussion

• Summary and conclusion

11 /26

Associative workAssociative work

• Life-like character-based systems developed in the context of our personal guidance system

• PalmGuide: a handheld tour guidance system

• AgentSalon: display showing conversations between personal agents according to their users’ profiles and interests

• ComicDiary: a system representing individual experiences in the style of a comic

12/26

IntroductionIntroduction

• C-MAP: Context-aware Mobile Assistant Project– To build a tour guidance system personalized according to its user’s

individual contexts– To facilitate knowledge communications among community members

• ComicDiary– A tool of to record their touring summaries– The process of generating a comic diary

1. determining the story stream of a comic diary from fragmentary data of the user’s touring records

2. rendering comic frames for assembling the necessary characters and scene backgrounds

13/26

ComicDiaryComicDiary

Fig. 1. Examples of hand-drawn diary in the style of a comic

14/26

Using comics to exchange individual experiencesUsing comics to exchange individual experiences

• Representing Individual Experiences in a Comic Style

• Characteristics of comic report– Structured according to the subjective viewpoint– Comic character has its own identity and personality as alter ego– Exaggerates her personal impressions projecting them onto the alter-ego

character– Highlighting impressive events– describing the surroundings and other visitors– Jokes and small talk – entertainment value

– not appropriate for precisely conveying detailed events– excellent for sharing personal impressions and edpisodes

15/26

Related WorksRelated Works

• Story generation in the context of AI and cognitive science– [Rummelhart’75] tried to build a grammatical schema of stories by analyzing

traditional fables– [Schank and Abelson’77] regarded AI issues e.g. problem solving and

planning, as story generation– Several systems of story generation [Okada’92, Ogata’96]

• Comic style as a user interface representation– Comic Chat for representing an online chat history as a comic [Kurlander’96]– Video Manga for visualizing a video summary in comic style [Uchihashi’99]– Chat Scape for using captured camera images to represent an online chat

history as a comic [Ayatsuka’01]

16/26

Implementation of ComicDiaryImplementation of ComicDiary

• The first prototype of ComicDiary– Participants to an academic conference

Fig2 . Scenario of system usage Fig3 . Accumulation of personal information

17/26

System architecture of ComicDiarySystem architecture of ComicDiary

• System Architecture of CmicDiary– Macromedia Flash running on web browsers– Diary generation was processed at a web server

• Processing for diary generation1. Clients certify users

user log in / automatically sent from the PalmGuide

2. Requests to generate comic diaries with the user IDs are sent to the server via CGI

3. Server extracts the user’s personal data from the community DB

4. Generate a comic diary

5. A Flash file (SWF) and parameter data are sent to the client

18/26

Story GenerationStory Generation

• Story Generation Engine– Use personal profile, community profile

• Potential streams of a comic story are prepared as a constraint network

representing mutual dependencies among scenes

19/26

Rendering Comic FramesRendering Comic Frames

• To increase the variety of comic frames from limited resources– Superimpose several

layers– Reuse background

layers

• Fig.6 Frame composition by multiple layers

20/26

ComicDiary IllustrationsComicDiary Illustrations

• 44 illustrations for each character

• 8 kinds of characters : 352 appearances of main character layers

• Templates of word layers– Presentation title– User name

• Fig. 7. Comic-part data (of main character layer)

21/26

Generated Comic Diary (1)Generated Comic Diary (1)• 3 types of story outlines

– For presenters, for active attendees, for non-active attendees

• Fig. 8. Example of generated comic diary (for a presenter)

Examples: For presenters

4 frames were used for the user's

presentation scene, followed by scenes

of meeting with other participants

22/26

Generated Comic Diary(2)Generated Comic Diary(2)

• Starts with a cheerful atmosphere

• Embedded with many scenes of attending presentations

• Fig. 9. Example of generated comic diary (for active attendance)

Examples: For active attendance

23/26

Generated Comic Diary(3)Generated Comic Diary(3)

• Presented with a negative (but amusing) mood

• Fig. 10. Example of generated comic diary (for non-active attendance)

Examples: For non-active attendance

24/26

A diary in hypertext styleA diary in hypertext style

• Ordinary method

25/26

User Evaluations and DiscussionUser Evaluations and Discussion

• 52 users participated in the digital assistant project in 2001

• After 5 weeks, 16 replied of survey.

How many people do you show your comic diary to

others?

Do you think the comic diary encourage conversations?

Does the comic exactly represent

your memory?

which is better to show to other people, comic style or hyper-text

style?

26/26

Summary and ConclusionsSummary and Conclusions

• Represent individual experience and interests in a comics style

• ComicDiary: a character agent – To summarize data ubiquitously obtained – To facilitate the exchange of personal experiences among users– As a digital assistant service for conference participants

• Future works– Knowledge processing techniques

To build a framework of flexible user-modeling from the user data To increase the variety of stories generated according to the user-model

– Computer graphics techniques To employ a morphable model of characters to decrease the cost of preparing

comic parts for a charater