POLITECNICO DI MILANO

Dipartimento di Elettronica e Informazione

HUMAN COMPUTER INTERACTION

NipponLugano Multitouch

Lucio Bordonaro, Silvia Zicca, Andres Fernando Arroyave Yepes

Academic Year 2011/2012

Abstract

Multitouch technologies are evolving quite fast, not only for single-user applications, but also for cooper-ative applications which allows users to interact with each others creating an engaging environment andachieving tasks together. It is quite easy to learn how to use this kind of new technology even for peoplethat are not used at all and it allow to explore a new way of interaction that is more natural with respectto the one of the personal computer, for example.

And this is why we developed a multitouch cooperative multi-user application which allows to navigatethe contents of a museum presented in an innovative way and that can be easily used also by childrenand old people who want to know information about the exhibitions presented in the museum.

2

Contents

1 Introduction 4

2 Requirements 5

3 Design 6

3.1 Gestures design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.2 Screens design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2.1 Main screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2.2 Exhibition screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2.3 Themes screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2.4 Gallery screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2.5 Image details screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3 States diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.4.1 A relevant non-cooperative scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.4.2 A relevant cooperative scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4 Prototype Implementation 23

4.1 Paper prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.2 Hardware and software architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2.1 Windows 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2.2 Vision system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2.3 Presentation layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.2.4 Core layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.2.5 Windows integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.2.6 Surface shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.3 Technical requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.4 Actual state of the implementation and open issues . . . . . . . . . . . . . . . . . . . . . . . . 29

4.5 Technical problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 Conclusions 30

3

1 Introduction

NipponLugano Multitouch is an application entirely dedicated to the Nippon Museum of Lugano, inSwitzerland. In this museum there are four exhibitions related to Japanese photographers. That mu-seum proposes some informative contents in its website in an hypertextual and multimedial way to allowguests to deepen the cultural context to which it is dedicated to, either before or after the visit.

The aim of the application is to repropose these contents in an innovative way towards a more naturalinteraction exploiting the new technology offered by Microsoft Surface 2.0, a multitouch surface of thelast generation which allows, among other things, to perform cooperative tasks between users and torecognize the direction of the touches.

The idea is to put the Surface with our application, inside the museum, near the entrance, so that po-tential guests of every age are tempted to buy a ticket after viewing a preview of the contents presentedin the museum. This is because it is very easy to understand how to interact with the Surface so even aperson who does not know how to use a computer, or a child can use it.

This application can be used either in a cooperative way (an user can pass some contents to another userto show it to him/her and discuss together) or in a non cooperative way (in case, for example, the Surfaceis used by people who do not know each other or do not need to exchange information).

Given the dimensions of the surface, the idea is to have users only along the long edges of the Surface.

4

2 Requirements

This application will have only one kind of users: the potential guests of the museum. We assume thatthey received the practical information (address, opening hours and so on) through the website and theyare just arrived to the museum. We can suppose that this big group is heterogeneous and composedby different kind of users (children in a school trip, old people, professionals photographers and alsocommon people with no particular characteristics), but they have the same requirements and needs.

In particular, considering the needs of every single independent user, he/she should receive the answerto some questions like:

• what are the exhibitions of this museum?

• what is every exhibition about?

• what are the themes faced by each exhibition?

• what are the photos related to a specified theme of the exhibition?

• what are the details of a selected photo?

For each one of the answers to the questions above, there will be an answer in one screen of our applica-tion which is structured to follow the questions that the user may have in order.

Besides these needs, there is also the possibility to use the Surface cooperatively. So every content canbe rotated and passed to another user that is placed on the other side of the Surface to discuss it togetherand showing it to other users.

NOTE: In this documents we use the words “screens” and “windows” as a synonymous.

5

3 Design

3.1 Gestures design

Microsoft Surface 2.0 supports a huge variety of possible gestures, so we had to select which ones to usein our application.

Here we report the gestures used to interact for the Surface:

• Move the windows: One or more fingers on a window to move or flick it.

Figure 1: Move the window

• Rotate the windows: there are three ways to rotate a window:

– Single finger rotate: One finger touches a window and drags it around in a circle so that itrotates.

Figure 2: Window rotation (1)

– Two-finger rotate: Two or more fingers on a window are dragged in opposite directions alongan arc.

6

Figure 3: Window rotation (2)

– Pin turn: One finger remains stationary acting as a pivot point while other fingers movearound it.

Figure 4: Window rotation (3)

• Screen simple touch: a simple tap on the screen with a finger.

Figure 5: Screen simple touch

• Zoom in and out: Two or more fingers on an item are dragged apart (used to resize the gallerywindow).

7

Figure 6: Zoom in and out

8

3.2 Screens design

3.2.1 Main screen

The main screen consists into a red circle on a white background which represents the Japanese flag(because the museum is related to Japanese photographers). The red circle in the middle is interactiveand it is split into four slices that represents the different exhibitions (identified ). The users can rotateit around its center and select the desired exhibition tapping on the slices. The use of a circle allows theusers to read it easily independently from his own position with respect to the surface.

Figure 7: Main screen

9

3.2.2 Exhibition screen

Once the user selected an exhibition, it will appear a screen with all the details related to that exhibi-tion (title, author, description and some random photos). The window can be rotated so that it can be“passed” to other users that want to read it, it can be moved wherever on the surface, it can be zoomedin and out, it can be closed or can allow to open the themes screen to allow to read more about the mainthemes of the selected exhibition.

Figure 8: Exhibition screen

10

3.2.3 Themes screen

The themes screen will show information about the themes of the selected exhibition (title, descriptionand one representative image). This window has a circular menu. Rotating it, it is possible to view in-formation about the different themes. Tapping on the representative image, the gallery of the theme willopen. This screen can be moved and it is possible to show or hide the description of each theme.

Figure 9: Themes screen

11

3.2.4 Gallery screen

Every theme has a gallery of images. This gallery window can be enlarged or reduced. Also the imagescontained into the window can be zoomed in and out and is possible to scroll horizontally and verticallyto view images not directly showed when the screen is too small.

Figure 10: Gallery screen

12

3.2.5 Image details screen

Every time you tap an image from a gallery or the exhibition library bar, it will open in an independentscreen with related details (title, author and description). It is possible for the user to pass the image toanother user without passing him/her the whole gallery of that theme. It is also possible to move, rotateand zoom the image. Text can be showed or hidden.

Figure 11: Image details screen

13

3.3 States diagram

The following diagram shows all the possible interactions that can be done with each window and howto pass from one to another. Consider also that every window can be closed by the user in every momentwithout influencing the behavior of the other. Also they can be opened more than ones simultaneouslyby the two side of the surface.

Figure 12: State diagram

14

3.4 Scenarios

3.4.1 A relevant non-cooperative scenario

Paolo wants to visit the Nippon Museum following the advice of a friend who has already gone there andwho has talked him vaguely what it is about. When he arrives to the museum, he has never visited thewebsite, so he knows only the practical information received by his friend (opening hours, address andticket price), so he decides to use the Surface to gain more detailed information about the contents of themuseum before buying the ticket.

From the main screen of the application, Paolo decides to retrieve information about the Araki’s exhibi-tion.

Figure 13: Araki exhibition

15

After he read about that exhibition and slided some of the images it contains, he decides see what are thethemes faced in that exhibition.

Figure 14: Araki themes

16

Chosen the theme Artistic Expression, Paolo decides to browse its gallery to have an idea about whatphotos he will find about that theme.

Figure 15: Araki theme: Artistic Expression

17

While zooming the gallery, Paolo is impressed by a photo, so he decides to zoom it in and to read someadditional information about it.

Figure 16: Araki photo details

Then, Paolo decides to close all the screens and to buy a ticket of the museum to visit it.

18

3.4.2 A relevant cooperative scenario

Marta and Sara are two sisters fond of photography who decide to visit together the Nippon Museum ofLugano. They walk towards the Surface and they place along the two long sides of the Surface and startbrowsing information about two different exhibitions: Araki (Sara) and Shunga (Marta).

Figure 17: Multiuser exhibition

19

While Marta is still reading, Sara opens the themes screen of Araki exhibition, she select the one sheprefers opening the gallery.

Figure 18: Multi-user

20

Marta closes the Shunga screen and decides to view information about Ineffable Perfection, meanwhileSara opens and zooms in a photo that she likes.

Figure 19: Opening a new exhibition

21

Sara passes the zoomed photo to her sister to listen to her opinion.

Figure 20: Passing content to another user

They are both interested into that particular photo, so they decide to go to buy the tickets to watch it.

22

4 Prototype Implementation

4.1 Paper prototype

After deciding which are the main elements of the application, we built up a paper prototype to simulatethe behavior of each component and analyzing if the user requirements were respected. As the term“prototype” says itself, the definitive version of the application presents some differences.

Here follows some photos about the paper prototype.

Figure 21: Exhibition selector

Figure 22: Exhibition screen

23

Figure 23: Themes screen

Figure 24: Gallery screen

24

Figure 25: Image details screen

25

4.2 Hardware and software architecture

The following figure represents the entire Surface platform, including the hardware layer and softwarecomponents that you must be aware of when you develop Surface applications.

Figure 26: Software architecture

4.2.1 Windows 7

Microsoft Surface 2.0 runs on the Windows 7 64-bit operating system. Windows 7 provides all the ad-ministrative, security, and directory functionality of the Surface. Developers and administrators workingwith a Microsoft Surface unit have full access to Windows functionality (in Windows Mode). However,when people interact with Microsoft Surface applications, the Windows user interface is suppressed (inSurface Mode).

4.2.2 Vision system

The Vision System uses PixelSense™ to process captured touch data into useful application data you canaccess through Surface SDK APIs. PixelSense™ enables each pixel in the Surface display to detect when aperson touches it or when someone moves a finger, tagged object, or untagged object over it. It does thiswithout the use of cameras, which is what makes newer Surface hardware much thinner than previousversions.

4.2.3 Presentation layer

The Presentation layer integrates with Windows Presentation Foundation (WPF) and includes a suiteof interaction controls designed for Microsoft Surface enabling you to quickly and easily build touchenabled applications.

26

4.2.4 Core layer

The Core layer exposes Microsoft Surface specific contact data and events so you can create MicrosoftSurface enabled applications with any user interface (UI) framework that is based on HWND (HandleWiNDows).

4.2.5 Windows integration

The tight integration between Microsoft Surface and the Windows operating system provides systemwide functionality on top of the Windows operating system. You must use this functionality to supportunique aspects of the Microsoft Surface experience, such as managing user sessions, switching betweenthe standard Windows user interface (Windows Mode) and the deployment experience (Surface mode),monitoring critical Microsoft Surface processes, and handling critical failures.

4.2.6 Surface shell

Surface Shell is the component that manages applications, windows, orientation, and user sessions andprovides other functionality. Every Microsoft Surface application must integrate with Surface Shell.

27

4.3 Technical requirements

To install and develop applications using the Microsoft Surface 2.0 SDK, you must you have the followingsoftware installed on your Windows 7 PC workstation, Windows 7 touch PC, or Surface unit that you areusing for development:

• Windows 7 operating system (64-bit recommended)

• .NET Framework 4.0

• Visual Studio 2010 or Visual C# 2010

• XNA 4.0 Redistributable or XNA Games Studio 4.0

• Microsoft Expression Blend 4.0 (recommended)

Additionally, you must create your applications to run on hardware designed for Surface 2.0, such as theSamsung SUR40 for Microsoft Surface, or Windows 7 touch PCs.

28

4.4 Actual state of the implementation and open issues

At the moment the application responds to all the user’s requirements specified in Chapter 2, but it isonly a prototype that needs to be tested with real users. Also there are still some open issues to fix.

First of all, the gallery of each exhibition is made to contain nine images related to the exhibition. Thenumber of nine was chosen because it is simple to dispose images in a 3x3 matrix and put in the middleof the square the selected image, so that the user can navigate all the figures starting from the chosenone. Also it is not a big number, so it does not overload memory and slow down the program execution.We decided to show different images from the same exhibition and the simplest way to do this was tochose one image for each theme of the exhibition. The problem is that some exhibitions do not havenine themes (but they have eight) so there can be some images repeated more than once.

The same problem is found when opening the gallery of a specified theme of the chosen exhibition: somethemes do not have nine images, so there are duplicates.

We analyzed the problem and one solution was to leave some empty spaces, but we decided that putduplicates was a better solution, because empty spaces could have confused the users. Another possibil-ity was to completely modify the structure of the gallery, but doing this we should have changed all thepossible interactions and GUI related to that screen.

Another issue consists in the fact that in the actual state of the implementation we have not set a max-imum number of screens that can be opened on the Surface simultaneously. Probably it is not a realproblem since the Surface automatically put the last touched window over the others.

Also we have not set the maximum dimensions that every screen can have.

4.5 Technical problems

The main problem we had to face during the implementation part of the project, is that Microsoft Surface2.0 was not on the market at the moment in which we developed the application, so we had to test it onnormal laptop with a screen resolution which is not like the one of the real surface (that is full HD). Alsowe had to use a touch simulator to simulate multitouch actions and other kind of interactions that arenot easily created using mouse movements.

Another minor problem was that some guidelines find on the Surface official website for developer, aretoo strict regards some aspects (i.e. the font size).

29

5 Conclusions

The project NipponLugano Multitouch is a useful application related to Nippon Museum of Lugano, butit can be easily modified and adapted to every kind of museum of exhibition losing only the metaphor ofthe Japanese flag in the main screen.

The main thing we learned is to work in contact with new technologies and in particular with multitouchworld which offers a huge number of interactions and possibilities that you cannot find in classical com-puters.

Also the Surface allows the simultaneous interaction of more than one users on the same device and thiscannot be provided by normal small devices with touch screen (i.e. smartphones, tablets...).

Working on this kind of technology, means to think in a totally different way about the human-computerinteraction and focus on the more natural interactions and gestures to allow the users to focus themselveson the contents and not on what they have to do to find and manipulate it.

30