adapting to input — smashing conference nyc

Adapting to InputJason Grigsby • @grigs • cloudfour.com

Web was created formless.

https://www.flickr.com/photos/chiaralily/3068047592

https://www.flickr.com/photos/chiaralily/3068047592

http://www.wired.co.uk/magazine/archive/2014/03/web-at-25/tim-berners-lee

http://www.wired.co.uk/magazine/archive/2014/03/web-at-25/tim-berners-lee

https://www.flickr.com/photos/inpivic/5206515374/

https://www.flickr.com/photos/tomokyo-blue/5812115264

https://www.flickr.com/photos/tomokyo-blue/5812115264

http://www.flickr.com/photos/60415054@N00/14301113/

So we told ourselves that the web was…



640 px

480 p

x


800 px

600 p

x

1024 px

768 p

x

http://www.flickr.com/photos/adactio/6153481666/

Then mobile came and made us realize…

http://www.flickr.com/photos/adactio/6153481666/

that it was a consensual hallucination all along.

http://www.flickr.com/photos/garibaldi/303085857/


The web never had a fixed canvas.

http://www.flickr.com/photos/paulocarrillo/124755065/

http://www.flickr.com/photos/paulocarrillo/124755065/

https://www.flickr.com/photos/giuliatanini/7125944117/


We’ve made tremendous progress.


But there is another consensual hallucination.



mobiledesktop

THE ART OF WEB DEVELOPMENTTHE ART OF WEB DEVELOPMENT

Web widgetsTHE ART OF WEB DEVELOPMENTTHE ART OF WEB DEVELOPMENT

Mobile widgets

It’s not that we’re technically incapable, but adapting a

phone UI to a tablet UI is not so dissimilar from trying to

automatically adapt desktop UI to a phone. They are

fundamentally different platforms with different

usability considerations…

—Todd Anglin, Kendo UI

http://www.kendoui.com/blogs/teamblog/posts/12-09-11/universal_mobile_apps_with_html5_and_kendo_ui.aspx

http://www.kendoui.com/blogs/teamblog/posts/12-09-11/universal_mobile_apps_with_html5_and_kendo_ui.aspx

It’s fairly certain that the highest-value use

will stay predominantly on desktop… Most

complex tasks have vastly better user

experience on the desktop and thus will be

performed there.

—Jacob Nielsen

https://www.nngroup.com/articles/transmedia-design-for-3-screens/

http://www.flickr.com/photos/royalsapien/2387707860

http://www.flickr.com/photos/royalsapien/2387707860

We envision desktop users as different from mobile.

http://demos.kendoui.com/web/grid/editing.html

http://demos.kendoui.com/web/grid/editing.html

http://www.flickr.com/photos/shantellmartin/4543010568

http://www.flickr.com/photos/shantellmartin/4543010568

https://www.flickr.com/photos/keh-s/6020885164/

It’s time to break free

of our assumptions

about input.

Four truths about input

#1 Input is exploding

An incomplete history of computer input

Focus on broad adoption Using Apple as a lens

Keyboard 1874

First typewriters in 1860s

Remington Typewriter 1874 Popularizes Qwerty keyboard

Qwerty patent granted 1878

Mouse 1984

Mouse prototypes 1960s by Engelbart and Mallebrein.

Xerox uses mouse in 70s, early 80s.

Apple Macintosh popularizes mouse and GUI in 1984.

Many alternatives to the mouse are created—trackballs, trackpads, and joysticks to name a few—but all replace the mouse and control the cursor.

No change in behavior.

Scroll Wheel 1996

First scrolling mouse in 1985. Scroll wheel on side.

Microsoft Intellimouse popularizes scroll wheel in 1996. Scroll wheel controls windows, not cursor.

Camera 2005 / 2006

Cameras are now included in phones and computers.

Cameraphones sold in 2005 alone greater than all digital cameras sold to date.

All Mac computers include cameras starting 2006.

iPhone 2007

Multi-touch screen Camera Accelerometer Proximity Sensor Ambient Light Sensor

Macbook Air January 2008

Multi-touch trackpad Gestures in Mac OS Controls more than cursor

iPhone 3G 2008

GPS (Location services)

iPhone 3GS 2009

Voice control Magnetometer (Compass)

iPhone 4 2010

Gyroscope Front facing camera

Magic Trackpad 2010

Multi-touch trackpad Gestures in Mac OS No longer limited to laptops

iPhone 4S 2011

Siri Bluetooth LE

iPhone 5 2012

Nothing?

iPhone 5S 2013

Fingerprint Sensor

iPhone 6 2014

NFC (Apple Pay) Barometer

iPhone 6S 2015

3D Touch

Same story could be told using Android or Microsoft.

19841874

142 years

1996

12 years

19841874 1996

1984 1996 2005

9 years 2 years

2007

1984 1996 2005 2007

2005 2007

2005 2007

Touch Camera

Accelerometer Proximity Sensor

Ambient Light Sensor

GPS Trackpad

2008

Voice control Magnetometer

2009 2010

Gyroscope Front-facing Camera

Magic Trackpad

2011

Siri Bluetooth LE

2012 2013

Fingerprint Sensor

2014

NFC (Apple Pay)

2015

3D Touch

2016

?

Touch 3D Touch Fingerprint sensor Camera (video and image) GPS (location) Compass Bluetooth LE Audio (Siri) Gyroscope Accelerometer Barometer Proximity sensor Ambient light sensor NFC (Apple Pay) iBeacon

Things aren’t slowing down.

https://www.flickr.com/photos/andreastsonis/8675283555

https://www.flickr.com/photos/andreastsonis/8675283555

Siri gets all of the hype…

http://www.98ps.com/viewnews-15222.html

http://www.98ps.com/viewnews-15222.html

but both Microsoft and Google have compelling voice input in their products.

How should web pages change to support voice control?

Google voice search

https://www.youtube.com/watch?v=kwn9Lh0E_vU

Remember Leap Motion?

https://www.youtube.com/watch?v=LJPxyWM9Ujg

https://www.youtube.com/watch?v=fiZkEYLXctE

#2 Input is a continuum

Supports hover and pointer events.

Keyboard and touch.

Even the iPhone can have a keyboard.

Are these laptops or tablets?

Desktop computer with 23” touch screen

Luke Wroblewski nailed it.http://static.lukew.com/unified_device_design.png

http://static.lukew.com/unified_device_design.png

We can no longer make

assumptions about input based on

screen size or form factor.

3. Input is undetectable

Higher precision with mouse means smaller targets possible

Hover state

Less precise than mouse and requires larger touch targets

Typing easier for many

No hover state

Typing often more difficult

Right clicking and “power” tools Single and multi-touch gestures

I got this. Detect touch.http://www.flickr.com/photos/28096801@N05/5012309802

http://www.flickr.com/photos/28096801@N05/5012309802

Whatever you may think, it currently isn't

possible to reliably detect whether or not

the current device has a touchscreen, from

within the browser.

—Stu Coxhttp://www.stucox.com/blog/you-cant-detect-a-touchscreen/

http://www.stucox.com/blog/you-cant-detect-a-touchscreen/

Chrome has entertained idea of enabling touch by default.

https://code.google.com/p/chromium/issues/detail?id=159527 https://docs.google.com/a/cloudfour.com/presentation/d/1-n1qyzewpagREbzW2zm0wOalq33UhbtbSkWf9mEdly8/edit#slide=id.gc2d80e5b_171

https://code.google.com/p/chromium/issues/detail?id=159527

https://docs.google.com/a/cloudfour.com/presentation/d/1-n1qyzewpagREbzW2zm0wOalq33UhbtbSkWf9mEdly8/edit#slide=id.gc2d80e5b_171

Detect a mouse? Not reliably.

Surely we can detect a keyboard?

NOPE

https://www.flickr.com/photos/mediaflex/4628277817/

https://www.flickr.com/photos/dentyone/2410917872/

Be thankful. This saves us from ourselves.

Interactive touch laptop experiments by Boris Smus

We can only detect input when it is used and that’s too late for our user interfaces.

4. Input is transient

Knowing what input is used one moment tells you nothing about what will be used next.

Input isexploding

continuum

undetectable

transient

Overwhelmed?https://www.flickr.com/photos/cheo70/2702682262/

https://www.flickr.com/photos/evilerin/3723714381/

We just need to rewire our brains again.

Adapting to input

#1 Design for the largest target by default

Fitts’s Law The time to acquire a target is a function of the distance to and size of the target.

After poking at this problem for a few weeks, my

conclusion is: every desktop UI should be

designed for touch now. When any desktop

machine could have a touch interface, we have to

proceed as if they all do.

—Josh Clarkhttp://globalmoxie.com/blog/desktop-touch-design.shtml

http://globalmoxie.com/blog/desktop-touch-design.shtml

#2 Design for modes of interaction instead of inputs

Display density settings

Couch Mode+ See allCentric TV’s videos / Recently viewed / Related videos

Th Dream Experience - …Uploaded 2 years ago

More Of The Dream Exp…Uploaded 2 years ago

The Dream Experience -…Uploaded 2 years ago

The Dream Experience …Uploaded 2 years ago

The Love King Breaks It…Uploaded 2 years ago

PROMOTED

War Paint for TreesFrom Lincoln Motor Company

TM + © 2013 Vimeo, LLC. All rights reserved. Terms • Privacy • Copyright • Cookies • Site Map Made with k in NYC. Language: English

Join Log In Create Watch Upload

3 CommentsDan Thil2 months ago

craigblaze7 months ago

Denis Rozenkov1 year ago

4

4

4

4

L Like this video S Share this videoF Full screen M More videos

26 LIKES

+ See all

RELATED COLLECTIONS

Aloe Blacc • Stones Throw

ALSO CHECK OUTMore stuff from Centric TV

399 Videos

1 Like

5 Channels

9 Following

NEED HELP?If you have questions about what’s on this page, lookhere first: Help / FAQKeyboard Shortcuts

SOULSESSIONS // Aloe Blaccfrom Centric TV PROPRO 2 years ago NOT YET RATEDNOT YET RATED

This conversation is missing your voice. Please join Vimeo or log in.

visualhybrid 2 years ago

Man, totally hypnotic song..

A.A. 1 year ago

The opening vocalise makes me shiver! Great voice and interpretation and the rest of theband really swings!

rossangeles 1 year ago

Directed, shot and edited by Ross Harris.

Vimeo

About Vimeo

Vimeo Blog

Vimeo Guidelines

Developers

Jobs

Help

Help Center

Basics

Video School

FAQ

Forums

Upgrade

Vimeo Plus

Vimeo PRO

Vimeo for Business

UploadUpload multiple videos atfaster speeds with ournew uploader.

Did you know?Hubnut lets you show multiple videos in arow, so people can keep watching. Andwatching. And watching.

1

3 Follow Stats 9 Switch to HTML5 Player

Search s

[ ]

– VIDEO

S

Vimeo Couch Mode







PROMOTED







4

4

4

4


26 LIKES

+ See all

RELATED COLLECTIONS



399 Videos

1 Like

5 Channels

9 Following






A.A. 1 year ago




Vimeo

About Vimeo

Vimeo Blog

Vimeo Guidelines

Developers

Jobs

Help

Help Center

Basics

Video School

FAQ

Forums

Upgrade

Vimeo Plus

Vimeo PRO

Vimeo for Business



1


Search s

[ ]–

VIDEOS







PROMOTED







4

4

4

4


26 LIKES

+ See all

RELATED COLLECTIONS



399 Videos

1 Like

5 Channels

9 Following






A.A. 1 year ago




Vimeo

About Vimeo

Vimeo Blog

Vimeo Guidelines

Developers

Jobs

Help

Help Center

Basics

Video School

FAQ

Forums

Upgrade

Vimeo Plus

Vimeo PRO

Vimeo for Business



1


Search s

[ ]–

VIDEOS

Vimeo Couch Mode

Vimeo Couch Mode

You’re designing for user need—not for a specific form factor or input.

#3 Make things accessible

D-pad remote controls

http://www.flickr.com/photos/stewc/6669743035/

TVs browsers that support d-pad, send arrow key events.

http://www.flickr.com/photos/stewc/6669743035/

If then

http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/


is undetectable.

If then



Support



Building accessibly increases the likelihood of support for future, unknown inputs.

#4 Design for multiple concurrent inputs

Interactive touch laptop experiments by Boris Smus

At minimum, don’t assume if one input is present that the person doesn’t have access to other types of input.

Look for opportunities to combine input to create more powerful experiences.

#5 Abstract baseline input

Mobile Tablet Desktop

Small Medium Large

TAP

https://www.flickr.com/photos/openexhibits/5884451755

https://www.flickr.com/photos/openexhibits/5884451755

POINT SELECT

Mouse events and touch events are fundamentally

different beasts in browsers today, and that makes it

hard to write cross-platform apps.

These incompatibilities lead to applications having

to listen to 2 sets of events, mouse on desktop and

touch for mobile.

https://github.com/jquery/PEP/

http://ie.microsoft.com/testdrive/ieblog/2011/Sep/20_TouchInputforIE10andMetrostyleApps_1.png http://www.w3.org/TR/pointerevents/

http://blog.webplatform.org/2013/02/pointing-toward-the-future/

New Pointer Events spec normalizes touch, mouse and stylus

Pointer Events builds on the DOM event model to offer a new way to handle input on the web, enabling developers to build touch-first experiences that work with mouse, pen, and other pointing devices as well…They are also designed from the ground up to allow modern browsers to accelerate the touch-surface performance, leading to a smoother user experience.

http://ie.microsoft.com/testdrive/ieblog/2011/Sep/20_TouchInputforIE10andMetrostyleApps_1.png

Shipped OpposedIn development

jQuery Pointer Events Polyfill

https://github.com/jquery/PEP/

#6 Progressively enhance input

Opportunity is knocking!

We’re pretty good at using Geolocation.

Compass.js

http://ai.github.io/compass.js/

http://ai.github.io/compass.js/

Warby Parker Gyroscope

https://www.warbyparker.com/eyeglasses/women/simone/tea-rose-fade

https://www.warbyparker.com/eyeglasses/women/simone/tea-rose-fade

Or hover

https://www.youtube.com/watch?v=7Yp0BpiDzXE

https://www.youtube.com/watch?v=Ow8gA0qe9SI

<input type="email" autocomplete="email" name="email"><input type="text" autocomplete="cc-name" name="card-name"><input type="text" autocomplete="cc-number" name="card-num"><input type="text" autocomplete="cc-exp-month" name="card-exp-month"><input type="text" autocomplete="cc-exp-year" name="card-exp-year"><input type="text" autocomplete="cc-csc" name="card-csc">

Remember to support browser auto-fill

and new Payment Request APIhttp://www.html5rocks.com/en/tutorials/forms/requestautocomplete/

You can use speech recognition too.

http://www.google.com/intl/en/chrome/demos/speech.html http://www.moreawesomeweb.com/demos/speech_translate.html

http://www.google.com/intl/en/chrome/demos/speech.html

http://www.moreawesomeweb.com/demos/speech_translate.html

Special thanks to Eric Bidelman http://moreawesomeweb.com

Speech Recognition API Support

http://moreawesomeweb.com

https://www.youtube.com/watch?v=_BUwOBdLjzQ

Web Bluetooth

https://www.youtube.com/watch?v=FYxQ5TeVFdE

Physical Web

https://www.youtube.com/watch?v=6z9ED4fmi1w

Physical Web &Web Bluetooth

#7 Make input part of test plans

Need to add input to our device labs

http://blog.adtile.me/2014/01/08/adtile-device-lab/

https://www.flickr.com/photos/jeepersmedia/14692542501

https://www.flickr.com/photos/jeepersmedia/14692542501

https://www.flickr.com/photos/yuckymuffintech/6988362001

https://www.flickr.com/photos/yuckymuffintech/6988362001

1. Design for the largest target by default

2. Design for modes of interaction instead of inputs

3. Make things accessible

4. Design for multiple concurrent inputs

5. Abstract baseline input

6. Progressively enhance input

7. Make input part of test plans

Input isexploding

continuum

undetectable

transient

www.flickr.com/photos/brunauto/5062644167/

Learn from our mobile context mistakes.

Who are we to judge which input is better?http://www.flickr.com/photos/fensterbme/4783366926

http://www.flickr.com/photos/fensterbme/4783366926

We need to learn to adapt.

http://www.flickr.com/photos/cdm/147947664/

http://www.flickr.com/photos/cdm/147947664/

Learn how to let go of illusions that comfort us.



Thank You!

adapting to input — smashing conference nyc

Internet