make my desktop app great with touch, mouse, and pen

Make my desktop app great with touch, mouse, and pen

Nick WaggonerSenior Program Manager4-022

Demo: Photo annotation

Input and hit-testingWindows touch languagePanning and zoomingPen and ink

Agenda

Where we are todayDeep and mature

Diverse ecosystem

Primarily mouse and keyboard

PC Form Factorshave evolved.

Key platform APIs

Direct2D, DirectWrite

Direct3D

Dyna

mic

DPI

Desktop Applications

DirectComposition

Rem

aind

er o

f Win

32

Real

time

Styl

us

Ink

Reco

gniti

on

InteractionContext

WM_POINTER Family

DirectManipulation

Win32 apps

Windows kernel services

Input and Interaction Rendering and composition

Real

time

styl

us

Ink

reco

gniti

on

InteractionContext

WM_Pointer family

DirectManipulation

Input and interaction

Input and hit-testing

WM_POINTER

Unified messages for touch, mouse, and pen.EnableMouseInPointer(…)

GetPointerInfo(…) / GetPointerInfoHistory(…)

GetPointerTouchInfo(…) / GetPointerPenInfo(…)

Real

time

styl

usIn

k re

cogn

ition

InteractionContext

WM_Pointer family

DirectManipulation

Touch hit testing

RegisterTouchHitTestingWindow(…)WM_TOUCHHITTESTINGEvaluateProximityToRect(…) or EvaluateProximityToPolygon(…)PackTouchHitTestingProximityEvaluation(…)

Control C Button 1

Button 2Real

time

styl

usIn

k re

cogn

ition

InteractionContext

WM_Pointer family

DirectManipulation

Demo: Moving to WM_POINTER & WM_TOUCHHITTESTING

Windows touch language

Windows touch languageRe

altim

e st

ylus

Ink

reco

gniti

on

InteractionContext

WM_Pointer family

DirectManipulation

InteractionContextInitialization

CreateInteractionContext(…)SetInteractionConfigurationInteractionContext(…)RegisterOutputCallbackInteractionContext(…)

Feeding inputAddPointerInteractionContext(…) ProcessPointerFramesInteractionContext(…)

Handling output

Real

time

styl

usIn

k re

cogn

ition

InteractionContext

WM_Pointer family

DirectManipulation

void INTERACTION_CONTEXT_OUTPUT_CALLBACK( _In_opt_  void *clientData, _In_      const INTERACTION_CONTEXT_OUTPUT *output );

typedef struct INTERACTION_CONTEXT_OUTPUT { INTERACTION_ID     interactionId; INTERACTION_FLAGS  interactionFlags; POINTER_INPUT_TYPE inputType; float              x; float              y; union { INTERACTION_ARGUMENTS_MANIPULATION manipulation; INTERACTION_ARGUMENTS_TAP          tap; INTERACTION_ARGUMENTS_CROSS_SLIDE  crossSlide; } arguments; } INTERACTION_CONTEXT_OUTPUT;

InteractionContext – Output structures

typedef struct INTERACTION_ARGUMENTS_MANIPULATION { MANIPULATION_TRANSFORM   delta; MANIPULATION_TRANSFORM   cumulative; MANIPULATION_VELOCITY    velocity; MANIPULATION_RAILS_STATE railsState; } INTERACTION_ARGUMENTS_MANIPULATION;

InteractionContext – Output structures

Demo: Integrating the touch language

Panning and zooming

Drives panning and zooming in the systemSlide and pinch

Encapsulates the personalityInertiaBounce effectsSnap-points

DirectManipulationRe

altim

e st

ylus

Ink

reco

gniti

on

InteractionContext

WM_Pointer family

DirectManipulation

Pre-declare your behaviors/interactions

DirectManipulationRe

altim

e st

ylus

Ink

reco

gniti

on

InteractionContext

WM_Pointer family

DirectManipulation

Harness the power of Windows with DirectComposition

InitializationIDirectManipulationManagerIDirectManipulationViewport IDirectManipulationContent

Hit-testingSetContact()WM_POINTERDOWN / DM_POINTERHITTEST

Manipulation beginsWM_POINTERCAPTURECHANGED

DirectManipulation – Key interfaces/messages

Real

time

styl

usIn

k re

cogn

ition

InteractionContext

WM_Pointer family

DirectManipulation

Adding panning and zooming

Best performance by integrating DirectManipulation with a 3 thread model:

UI threadDelegate threadIndependent hit-test thread

IDirectManipulationManager::RegisterHitTestTarget(…)

DirectManipulation - threadingRe

altim

e st

ylus

Ink

reco

gniti

on

InteractionContext

WM_Pointer family

DirectManipulation

Pen and ink

Independent of the UI threadHandwriting recognition

RealTimeStylusRe

altim

e st

ylus

Ink

reco

gniti

on

InteractionContext

WM_Pointer family

DirectManipulation

Initialization IRealTimeStylus

AddStylusSyncPlugin(…)Processing Input

IStylusSyncPlugin StylusDown(…), Packets(…), StylusUp(…)

IStrokeBuilderBeginStroke(…), AppendPackets(…),

EndStroke(…)Handwriting Recognition

InkRecognizerContextRecognize(…), BackgroundRecognize(…), BackgroundRecognizeWithAlternates(…)

RealTimeStylus - Key interfaces/methods

Real

time

styl

usIn

k re

cogn

ition

InteractionContext

WM_Pointer family

DirectManipulation

RealTimeStylus - Key interfaces/methods

Real

time

styl

usIn

k re

cogn

ition

InteractionContext

WM_Pointer family

DirectManipulation

IRealTimeStylus

IStylusSyncPlugin

IStrokeBuilder

InkRecognizerContext

Adding ink and recognition

Performance is critical for both inking as well as panning and zooming experiences.

Recap

Use WM_POINTER for touch, mouse, and pen.

Use the InteractionContext to incorporate the Windows touch language into your app.

Use DirectManipulation for great panning and zooming.

Differentiate your app by making ink pop with the RealTimeStylus APIs.

TakeawaysRe

altim

e st

ylus

Ink

reco

gniti

on

InteractionContext

WM_Pointer family

DirectManipulation

ResourcesRelated talks

3-017 - Power up your desktop app with Windows 4-021 - DirectComposition: Smooth composition and animation for desktop apps4-9147 – Exploring customized touch interactions for your Windows store app

Reference documentationWM_POINTER - http://msdn.microsoft.com/en-us/library/windows/desktop/hh454916(v=vs.85).aspx InteractionContext - http://msdn.microsoft.com/en-us/library/windows/desktop/hh437192(v=vs.85).

aspx DirectManipulation - http://msdn.microsoft.com/en-us/library/windows/desktop/hh446972(v=vs.85).

aspx RealTimeStylus - http://msdn.microsoft.com/en-us/library/windows/desktop/ms701683(v=vs.85).aspx InkRecognizerContext -

http://msdn.microsoft.com/en-us/library/windows/desktop/ms696371(v=vs.85).aspxProcess touch input with DirectManipulation sample

http://go.microsoft.com/fwlink/?LinkID=309852&clcid=0x409Design guidelines

Touch target sizes - http://msdn.microsoft.com/en-us/library/windows/apps/hh465415.aspx#touch_targets

Device distribution starts after sessions conclude today (approximately 6:00pm) in the Big Room, Hall D. If you choose not to pick up your devices tonight, distribution will continue for the duration of the conference at Registration in the North Lobby.

Get your goodies

Acer Iconia W3, Surface Pro, and Surface Type Cover

Evaluate this session

Scan this QR code to evaluate this session and be automatically entered in a drawing to win a prize!

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.