tools for programming in the real world

User Interface for Programmers

Tools forProgramming in the Real WorldSelf Introduction for MSRA Fellowship 2012Jun Kato, The University of Tokyo1

Hello, thank you for watching this audio-PowerPoint.1Jun Kato2nd- year PhD candidateIgarashi Laboratory, The University of Tokyo

Concentration: Human-Computer InteractionEspecially,Tools for Programming in the Real World

2

This is Jun Kato, a 2nd-year PhD candidate from Igarashi Laboratory, The University of Tokyo. My research interest includes broad area of Human-Computer Interaction, but I have been especially focused on designing tools for programming in the real world, which involves physical interaction such as robots and Kinect programming.2AgendaMy motivation for the researchIntroduction of the projectsIdeas for new projects3

First, I would like to describe how I have evolved my research interests of designing tools for programming in the real world. Then, provided my background, I will introduce three concrete projects to show effectiveness of my approach. Next, I will propose ideas for new projects, which can be done in collaboration with Human-Computer Interaction group of Microsoft Research Asia.3My motivationfor the researchTools for Programming in the Real World4

CRISTAL[SIGGRAPH09 E-Tech]Tools for Programmingin the Real World

Multi-touch[CHI09 SRC]

Roboko[ CHI13 (to be submitted)]Sharedo[Ro-MAN12 (submitted)]

Phybots[DIS12]

Surfboard[UIST10 Demo]

IntelliViewer[UIST10 SIC]

Pressing[UIST09 SIC]

DejaVu[UIST12 (to appear)]5

Here is the list of my previous projects. I would like to introduce these projects from two perspectives: physical interaction in the real world and tools for programming.5Sharedo[Ro-MAN12 (submitted)]

Roboko[ CHI13 (to be submitted)]

DejaVu[UIST12 (to appear)]

Phybots[DIS12]

CRISTAL[SIGGRAPH09 E-Tech]


Phybots[DIS12]





Multi-touch[CHI09 SRC]In the Real WorldFrom Robot to Human6

My research career started with natural user interface and robots. In 2009, I followed the trend of multi-touch interface and used it for simultaneous control of multiple mobile robots. It was submitted to student research competition at CHI 2009 resulting in the honorable 1st place. It was just a combination of what I was interested in at that moment, but became a root of my research projects. / The interface was built-in to another tabletop research project named CRISTAL, which was presented at Emerging Technologies of SIGGRAPH 2009. / Programming locomotive things excited me very much, but I noticed some difficulties related to robotics. This insight led me to develop a software toolkit for making robotic things, named Phybots, which I will later introduce in detail. This toolkit research was for HCI researchers and interaction designers, but I also wanted to allow end-users to command robots. 6Sharedo[Ro-MAN12 (submitted)]


Phybots[DIS12]








DejaVu[UIST12 (to appear)]In the Real WorldFrom Robot to Human7

Therefore, I made a web-based to-do list interface on which people write down physical tasks such as cleaning the room and the robots respond to ask for the details or perform the tasks. There are also chances that the robots write down physical tasks to ask for help such as low battery or entangled power cable. The paper was submitted to IEEE Ro-Man 2012 conference. / All these robots I had dealt with were assumed to have simple hardware configurations with a few degree-of-freedom. Though, there are robots with more degree-of-freedom which can express richer emotional expressions and play more complex routine tasks. The current state-of-the-art interface for designing actions of robots use either text-based or visual representation-based editor, but both have their own difficulties. I addressed this issue by developing an integrated development environment, Roboko. I also applied the same method for sensing human motions using Kinect. This will be submitted to CHI 2013. / From the experience of Kinect programming, I saw general difficulties in the development process of camera-based programs, which were not sufficiently supported by the current development environments. I addressed this issue by creating an IDE named DejaVu, in collaboration with Xiang Cao and Sean McDirmid, researchers in Microsoft Research Asia, while I was a research intern at MSRA. This work was accepted to UIST 2012.7





Phybots[DIS12]




Phybots[DIS12]



Multi-touch[CHI09 SRC]Sharedo[Ro-MAN12 (submitted)]

Tools for ProgrammingFrom Keyboard to IDE8

My previous projects can be also understood from the perspective of tools for programming research. First, I was a programmer of an application. Next, I made a software toolkit. Then, I enhanced existing development environments. My research interests have evolved to support the whole development process including prototyping software, debugging the application during its run time, recording and replaying input to the program, and so on. / Besides the main stream of my research projects dealing with the high level usability matter, I have been also interested in how input devices can affect programming experience. Especially, I have made three attempts to leverage new use of a keyboard.8





Phybots[DIS12]






CRISTAL[SIGGRAPH09 E-Tech]Sharedo[Ro-MAN12 (submitted)]

Tools for ProgrammingFrom Keyboard to IDE9

First, I implemented a pressure-sensitive development environment where size of a new Ball instance reflects its typing strength. The idea comes after the development of pressure-sensitive multi-touch display. It was a project for Student Innovation Contest in UIST 2009 and was also demonstrated in a domestic conference. / Second, I noticed that many laptops have built-in microphones and they are capable of detecting hand gestures on top of their keyboards based on the noise made by hands. The prototype was implemented and demonstrated at UIST 2010. / Third, I implemented a user interface for helping navigation of source code in the integrated development environment. For each row of the keyboard, available namespaces, classes, member variables and methods are shown on the keytops. The user interface works on the prototype adaptive keyboard, developed by Microsoft Applied Science Group and was presented at Student Innovation Contest in UIST 2010.9Introduction ofthe (Selected) ProjectsTools for Programming in the Real World10

From now on, I would like to introduce three most recent projects, all of which are tools that support programming in the real world.10ProjectsPhybotsDejaVuRoboko11

The first one is a toolkit named Phybots that helps programming locomotive robotic things.11MotivationAdd mobility to our daily objects12

By saying robotic things, we mean things equipped with mobility. Our motivation for this work is to add mobility to small physical objects which are otherwise stationary.12Gulf of HCI and Robotics13Physical UIsToolkit for:RobotsHCI researchers &Interaction designersTarget users:Robotics peoplePrototypingFocus:ReliabilityLow-level&StaticSoftware API:High-level&ExtensibleSmall&CheapHardware:

Phidget Kit, $200Medium-Large&Expensive

K-Junior, $938

Existing toolkits for Physical UIs focus on support for the whole prototyping process and the software API tends to be very simple and low-level such as controlling on and off of the electric circuit. Typical hardware components are small and cheap. When thinking of building robotic things, the lasting issue is that they do not provide API for time-consuming high-level tasks such as localization and locomotion of objects in the environment. On the other hand, numbers of toolkits for robot programming have been developed in the field of robotics. Though, their target are people with prior knowledge of robotics and mainly focus on reliable software built on top of the expensive hardware including many sensors and actuators. They are, unfortunately, not as accessible to the HCI researchers and interaction designers as toolkits for physical UIs. Therefore, there is a gulf of HCI and robotics, an area not supported by both types of toolkits.13Goal of PhybotsAdd mobility to physical objectsIn a cheap and easy wayThrough high-level and extendible APIWith support for the whole prototyping process14

Led by these observation, we set the goal of our toolkit as shown here. It tries to add mobility to physical objects. It should provide full-spec software programming experience just as popular GUI programming. And it supports the whole prototyping process.14Prototyping with Phybots15

Hardware constructionProgramming with built-in APITest withruntime debug tool(Extend API when needed)

To achieve the goal, Phybots support programming and testing phase of the prototyping. The user must construct hardware beforehand, but is not difficult since its already supported by toolkits for Physical Computing such as Arduino-based robot boards or LEGO Mindstorms NXT.15

Localization and Locomotion APIHardware setup: easy and cheapCameraPC or MacRobotic things16

From our user study:Miniature drive recorder

For programming locomotive robotic things, our toolkit provides localization and locomotion APIs based on simple and cheap hardware setup, consisting of a camera, PC and robotic things.16Localization and Locomotion APINavigation by global coordinatesMovePushTracePath17Task move = new Move(mouseX, mouseY);move.assign(robot);move.start();

From our user study:Beach flags with obstaclesY-axisX-axis

APIs are based on global coordinates defined by the area camera is looking at, and the programmer can easily command the robotic thing to go to the specified location, to push an object to the destination, and so on. Since the programmer can write program based on the two-dimensional absolute coordinates, the programming experience is very much similar to that of GUI programming.17Localization and Locomotion APIVector field navigationEasy design of new behaviorex) Follow another robot

Combination of existing fieldsex) Move + collision avoidance18

Our toolkit uses a vector field to implement locomotion APIs, and allows the programmer to extend the toolkit by specifying a new vector field. He can also combine existing vector fields to achieve multiple purposes at the same, for example, moving to the destination while avoiding collision with other objects.18Runtime Debug Tool19

Entity Monitor

Service Monitor

Workflow Monitor

Our toolkit does not only provide APIs but also a GUI-based runtime debug tool with which the programmer can monitor and control usage of the toolkit APIs while the program is running. / Entity monitor shows low-level status information for robotic things. / Workflow monitor shows what kind of tasks are handled by the robotic things and allows their control. / Service monitor exposes any other features provided by the toolkit such as camera image acquisition and marker detection.19User StudiesAlpha version deploymentTo graduate studentsCurrent version deploymentTo HCI studentsTo robotics students20

Along with the development of the toolkit, we have conducted three user studies.20Lessons learned:Mere mobility was not enough.Most apps only used a single API call.Parameter configuration was painful.Alpha version deployment21

Lessons learned:Extendibility is important.Higher-level task management is desired.Support for testing phase is needed.

The first study was conducted at the early stage of the toolkit development. The toolkit only provided localization and locomotion APIs but any extendibility nor runtime debug tool. The participants successfully built many interesting applications, as some of them are shown here, but the study revealed some limitations. First, the participants wanted to extend the robotic things rather than just using the locomotion APIs. Second, they only used a single API call and did not make them handle complex tasks. Third, parameter configuration was painful for the participants. From these observations, we learned that21Current version deployment22

After the alpha version deployment, we revised the toolkit to provide the previously described full feature and conducted another user study. All the participants could successfully implemented various applications shown in this slide. They added mobility to a video recorder, a camera, an alarm clock, dishes with sweets, and so on.22PhybotsA toolkit for prototyping robotic thingsLocalization and Locomotion APIExtensible ArchitectureRuntime Debug Tool

Open-source software available at http://phybots.com

23

In summary, 23ProjectsPhybotsDejaVuRoboko24

The second project is named DejaVu, which is an integrated development environment for interactive camera-based programs.24MotivationInput by the users hands, body, or physical objectsCaptured by camerasProcessed in real-time

Support for computer-vision-based programming is needed

Wang, 2006Wilson, 2005

Cao, 200325

These days, there are many applications that use The programmer needs tailored support for such computer-vision-based programming.

PlayAnywhere (UIST), Face tracking (CHI)25Existing approachesHardware, Software APIBecomes more and more accessibleWebcam, Kinect, OpenCV, etc.

Development EnvironmentsDo not provide explicit supportEx) Breakpoint-based variable monitoring

OpenCVVisual StudioEclipse26

Hardware and software APIs are becoming more and more accessible. There are affordable webcams, Kinect with Kinect SDK, OpenCV libray, and so on. Though, current development environments such as Visual Studio and Eclipse do not provide explicit support for programming with these hardware and software libraries.26DejaVu

27

DejaVu is an enhancement to the existing development environment which specifically supports iterative development process of interactive camera-based applications. It adds two components named canvas and timeline.27

DejaVu CanvasFrame data visualizationFreehand sketch28

The programmer can drag and drop a variable to the canvas, and then, a placeholder appears to visualize its value in the canvas. The programmer can also draw freehand sketch on the canvas. While the program is running, the canvas interface continuously shows input from the camera, variable values, and window output at the newest frame.28DejaVu CanvasCapable of showing variable values at the different positions in the code

29

The canvas is capable of For example, it can visualize a segmentation mask before and after the erosion operation.29

DejaVu Timeline30

Whenever the program runs, DejaVu records all the input from the camera, variable values, and window output. Timeline interface shows the recorded data in a temporal fashion and allows the programmer to replay the recorded data in various speeds or to select a specific frame. In both cases, the current frame of interest is visualized in the canvas.30ConclusionDejaVu provides integrated support for the development of interactive camera-based programs that tightly matchestheir nature: Visual/Continuous/Non-reproduciblethe workflow: Record input/Change code/Replay

CanvasWorkflow31

31ProjectsPhybotsDejaVuRoboko32

The third project is also an IDE project, which integrates photos into a text-based editor.32

Programming robot actionsis difficult.33

DanceClearing dishesCookingMotivation:

Robots can dance with rich emotional expression, do complex routine work such as clearing dishes and cooking. My motivation for this project is the fact that programming such robot actions is difficult. Current typical development environment is text-based or visual-language-based and both of these have their own difficulties.

Retrieved from http://www.youtube.com/watch?v=Z1TK4D1Jr6k, http://robot.watch.impress.co.jp/cda/news/2008/12/18/1512.html and http://www.robospot.jp/cgi-bin/diarypro/diary.cgi?no=94.33Text-based Programmingjoint[0].setAngle(0.6);joint[1].setAngle(0.7);joint[2].setAngle(-0.24);arm.setPosition(4.3, 2.4);

robot1.setPose(pose32);34Problem:We cannot imaginethe actual pose of the robot

Typical approach:

In a text-based editor, the programmer can set pose of the robot by specifying raw values or by loading predefined poses. However, it is not easy to imagine the actual pose of the robot from these numbers and strings.34

Visual Editing35Problem:We cannot design flexible robot behavior such as repeat this for 3 times when Typical approach:

The other typical environment allows the designer to edit action of a robot along a timeline. This kind of editor is good for editing pose but is much less flexible than text-based programming since control flow of the actions cannot be specified. For example, the robot cannot be programmed to respond to user input.35

Overview of the IDE36

Here is the overview of the proposed development environment. On the left side, a text editor is integrated with visual representations. On the right side, the pose library is shown, which supports coding and could be hidden or minimized when its not used.36

Workflow of the Programmer37

1. Capture2. Drag and DropCameraRobotPC

The workflow of the programmer is as shown in this slide. First, the programmer record a pose by using a webcam. Second, the programmer can load the stored pose by dragging and dropping the image into the code editor, which will be shown as an inline image.

37Getting and setting pose38Supported APIs:

To support programming robot actions, our IDE provides a special set of API which takes pose parameters. The programmer can get the current pose to compare it against the stored pose. He can also apply a certain pose to the robot.38

Execute a sequential action39Supported APIs:

To ease the programming that applies a sequence of poses to a robot, there is an Action API to specify such sequential action.39Not only a robot but a human (Kinect)40Project outcome: new application

This work is an outcome of collaborationwith a MSR researcher, Xiang Cao

When we made the introduced IDE, we noticed that this integration of photo and text code can be also used for a certain type of Kinect programming. I did this in collaboration with Xiang Cao, a researcher at Microsoft Research Asia.40Integrating visual representations into text-based programming environment41

In summary, we have integrated visual representations into text-based programming environment. It aims to incorporate the benefits of general text-based programming and visual-language-based pose editing. It records pose data of a robot accompanied with a photograph captured by a camera in order to provide visual aids to the programmer.41Ideas for new projectsTools for Programming in the Real World42

Given the previous projects, I am thinking of 42Ideas for New Projects(CENSORED)(CENSORED)(CENSORED)43

Currently, I have several ideas that might help programming in the real world. The ideas vary from low-level Programming Language research to high-level application-centric end-user programming.43Thank you for listeningIve designed various tools for programming in the real world.

I am looking forward to further collaboration with MSRA researchers.44

tools for programming in the real world

Documents

research projects

toolkit research

research career

research interests

real worldfrom robot

phybotsdis12 dejavuuist12

kinect programming

real worldself introduction