sculpting the augmented world: scale and narrative in augmented ux design
TRANSCRIPT
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
1/36
Thank&you&for&having&me&here&today.&This&morning&Im&going&to&present&some&ideas&Im&
working&through&about&designing&user&experiences&for&a&world&where&many,&if¬&all&
objects&in&your&environment&have&some&form&of&networked&digital&component.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
2/36
Let&me&begin&by&telling&you&a&bit&about&my&background.&Im&a&user&experience&designer.&I&
started&designing&Web&sites&back&in&the&Jurassic.&Heres&a&design&I&worked&on&for&HotBot,&an&
early&search&engine.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
3/36
Ive&also&worked&on&the&user&experience&design&of&a&lot&of&consumer&electronics&products&
from&companies&youve&probably&heard&of.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
4/36
Along&the&way&I&wrote&a&couple&of&books&based&on&my&experience&as&a&designer.&One&is&a&
cookbook&of&user&research&methodsthe&second&edion&came&out&last&falland&the&
second&describes&what&I&think&are&some&of&the&core&concerns&when&designing&networked&
computaonal&devices.&It&came&out&a&couple&of&years&ago.&The&ideas&for&this&talk&originated&
in&the&second&book.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
5/36
I&also&started&a&couple&of&companies.&The&first,&Adapve&Path,&is&primarily&focused&on&design&
for&the&Web.&With&the&second&one,&ThingM,&I&got&deep&into&developing&hardware.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
6/36
ThingM&primarily&designs&and&manufactures&a&range&of&RGB&LEDs&with&aached&controllers&
for&architects,&industrial&designers&and&hackers.&These&have&been&on&sale&for&about&five&
years&and&theyre&used&in&everything&from&flying&robot&prototypes&to&costumes&in&Lady&
Gagas&stage&show.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
7/36
We&also&spent&some&me&exploring&the&potenal&uses&of&augmenng&everyday&objects&with&
embedded&technology.&
&
On&the&leU&is&an&RFID&wine&rack&thats&faceted&classificaon&browser&to&a&cloudbased&
service.&For&example,&it&can&show&you&all&the&2003&Napa&Cabernet&Sauvignons&which&are&ready&to&drink&and&whose¤t&market&price&is&under&$50.&On&the&right&is&a&capacive&
sensing&kitchen&cabinet&knob&that&glows&when&you&touch&it.&Its&&an&exploraon&in&making&a&
digital&product&that&would&sll&be&useful&20&years&aUer&it&was&made.&
&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
8/36
In&January&of&this&year&I&joined&PARC&as&a&principal&in&its&Innovaon&Services&group.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
9/36
But&I&didnt&come&here&to&talk&about&myself.&&When&I&was&last&here&two&
years&ago,&I&advocated&renaming&augmented&reality&as&the&less&excing
sounding,&but&perhaps&more&accurate&somac&data&percepon.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
10/36
My&argument&was&that&visual&AR,&informaon&layered&onto&our&visual&field,&faces&a&big&
problem,&and&thats&the&limited&bandwidth&of&human&percepon.&We&can&start&with&how&
lowresoluon&our&eyes&are.&This&is&our&foveal&resoluon,&which&you&will&noce&wouldnt&
even&cut&it&as&the&cheapest&mobile&phone.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
11/36
To&compensate,&our&eyes&move&in&saccades&which&last&between&20ms&and&200ms,&or&5&to&50
mes&per&second.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
12/36
So&this&leads&to&a&back&of&the&envelope&calculaon&of&eye&bandwidth&between&100&bits&per&
second&and&10K&bits&per&second&
&
Thats&around&4&orders&of&magnitude&slower&than&a&modern&frontside&bus.&&
&The&brain&deals&with&this&through&a&series&of&ingenious&filters&and&adaptaons&to&create&the&
illusion&of&an&experience&of&all&reality,&but&at&the&core&theres&only&a&limited&amount&of&
bandwidth&available&and&our&visual&senses&are&easily&overwhelmed.&And&unfortunately&our&
eyes&and&brains&are¬&subject&to&Moores&Law.&Were¬&going&to&get&more&foveal&
resoluon&or&cognive&processing&capability&any&me&soon.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
13/36
My&argument&is&to&broaden&the&definion&of&AR&to&be&the&experience&of&any&kind&of&data&by&
any&sense,&to&change&our&language&away&from&augmented&reality&as&primarily&visual&
augmentaon&to&something&thats&more&representave&of&the&whole&experience&of&data&in&
the&environment,&the&bodily&experience&of&the&informaon&shadows&that&objects&we&
encounter&in&the&world&cast&into&the&internet,&the&personal&experience&of&the&Internet&of&
Things.&I&called&this&Somac&Data&Percepon.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
14/36
Once&we&broaden&our&definion&to&include&other&senses,&we&can&start&creang&hierarchies&
of&informaon&that&match&the&capabilies&of&our&senses&and&avoid&pung&secondary&
informaon&into&our&primary&sense.&We&can&begin&associang&secondary&data&with&
secondary&senses.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
15/36
There&are&plenty&of&ways&we&can&augment&the&world&that&doesnt&require&increasing&the&
load&on&our&visual&cortex.&Were&already&sufficiently&experienced&with&phone&buzzing&that&
we&can&feel&phantom&vibraons&because&our&brains&have&deeply&embedded&that&sensaon&
as&meaningful&and&when&theres&a&misfire,&we&jump.&Other&projects,&such&as&Jason&Wilsons&
Audio&Perfume&create&shiUing&soundscapes&that&are&ed&to&informaon.&You&can&think&of&
other&examples,&such&as&an&armband&that&gets&warmer&as&you&bike&past&an&apartment&thatsfor&rent&in&your&price&range.&These&arent&nearly&as&high&resoluon&signals&as&visual&ones,&
which&makes&them&feel&less&important,&but&I&think&theyre&actually&incredibly&important&
because&of&the&informaon&they&remove&from&the&visual&channel.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
16/36
Next,&I&want&to&say&that&Im&a&firm&believer&that&everything&is&going&to&be&augmented&with&
networked&informaon&processing&very&soon.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
17/36
For&example,&adidas&has&the&miCoach&and&miCoach&elite&systems,&in&which&a&soccer&ball,&the&
players&shirts,&their&shoes&and&somemes&the&space&theyre&in&are&all&networked&and&
instrumented.&The&screen&devices&they&carry&are&the&displays&of&the&informaon&shadows&
all&of&the&things&are&casng&onto&the&Net.&Very&soon&this&is&going&to&be&your&office,&your&
playground,&your&home&and&your&coffee&shop.&Everything&that&can&be&connected&to&the&net,&
including&things&that&probably&shouldnt&be,&will&be.&Full&stop.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
18/36
Now&we&get&to&the&interesng&user&experience&challenge.&How&do&we&design&for&the&
environment&where&everything&has&a&data&shadow&and&can&be&perceived&through&any&
sense?&Designers&work&well&with&constraints,&so&Im&focusing&on&two&ideas.&The&first&is&that&
the&scale&of&an&experience&is&a&crical&component&of&how&that&experience&is&perceived.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
19/36
Computers&have&tradionally&had&interfaces&that&are&roughly&sized&to&our&bodies.&If&you&
look&at&a&laptop,&the&inputs&are&the&size&of&your&hands&and&the&output&is&on&the&scale&of&your&
head.&We&dont&have&great&models&for&designing&experiences&for&different&scales,&and&when&
we&try&to&map&what&we&know&from&designing&PC&experiences&directly&onto&other&scales,&it&
fails,&as&everyone&who&has&ever&designed&a&phone&interface&for&the&first&me&knows.&
&
At&PARCs&ubicomp&program&in&the&late&80s&and&early&90s&they&defined&the&tab,&pad&and&
board&as&names&for&the&scales&of&the&devices&they&were&developing,&leveraging&the&form&
factors&of&office&supplies&that&had&proven&to&be&at&useful&sizes.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
20/36
This&is&the&scale&Ive&been&using.&Its&of&course&inspired&by&the&Eames&Powers&of&Ten&films,&
but&Ive&been&using&it&to&understand&the&constraints&and&potenal&of&devices&at&different&
scales.&Screens&dont&work&interacvely&when&you&approach&the&covert&scale,&which&is&why&
wristmounted&devices&can&at&best&be&output&devices&displaying&a&limited&set&of&visual&
informaon,&but&theyre&great&for&sending&a&small&message&no&one&else&knows&about.&Thats&
really&hard&to&do&with&a&big&display&such&as&a&TV.&Buons&dont&work&well&controlling&experiences&on&the&environmental&scale&and&above,&because&theyre&too&small&relave&to&
the&object.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
21/36
The&two&screen&experience,&that&mythological&consumer&electronics&grail,&is&really&a&
problem&of&mismatching&scales.&How&can&you&create&a&single&experience&that&
simultaneously&spans&the&mobile&and&personal&scale?&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
22/36
To&me&this&brings&me&to&my&second&idea,&which&is&that&for&an&experience&to&make&sense&
across&various&scales,&we&need&a&narrave&to&e&it&together.&I&mean&this&quite&literally.&We&
need&a&story&of&some&sort&that&will&help&us&make&sense&of&why&what&were&doing&at&any&
given&moment&maers.&The&desktop&metaphor&is&a&kind&of&a&narrave.&When&it&was&applied&
to&the&same&kinds&of&funcons&that&had&existed&with&the&command&line,&it&made&them&much
easier&for&people&to&understand&because&it&told&a&story&about&office&work.&It&wasnt&a&perfect&mapping,&because&it&was&a&metaphor,&but&it&was&beer&than&what&it&replaced.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
23/36
Look&at&the&way&we&explain&what&future&technology&is&going&to&do.&These&days&its&always&in&
the&form&of&stories&that&use&the&technology,&whether&its&being&a&caring&parent&while&
engaged&in&some&kind&of&serious&mulnaonal&project,&or&opmizing&your&crop&planng,&its&
always&a&story&about&what&the&end&result&is&going&to&be.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
24/36
Reeves&and&Nass,&two&Stanford&professors,&did&a&series&of&groundbreaking&experiments&that&
showed&that&no&maer&how&you&cut&it,&when&people&are&in&dialog&with&a&device,&they&treat&
it&like&a&person.&And&since&the&kinds&of&relaonships&we&have&with&people&are&essenally&
narraves,&this&in&effect&means&that&all&of&our&experiences&with&devices&have&a&narrave&
component,&whether&we&put&one&in&or¬.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
25/36
And&so&we&come&to&transmedia&storytelling.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
26/36
Many&of&you&are&probably&familiar&with&Henry&Jenkins&work&on&how&stories&are&now&told,&
with&bits&and&pieces&scaered&across&many&media.&Jenkins&canonical&examples&is&the&
Matrix,&where&characters&and&acon&in&the&second&and&third&film&didnt&really&make&sense&
unless&you&had&watched&the&Animatrix&cartoon,&read&the&comics&and&played&the&games.&&A&
more&recent&example&is&Jorgits&and&the&end&of&Winter,&which&is&a&set&of&childrens&stories&
that&are&told&in&an&app&thats&mostly&a&novel,&but&has&elements&of&video&games;&however,&in&addion&to&that&the&characters&have&their&own&Twier&feeds,&Facebook&pages,&Soundcloud&
streams&and&Pinterest&pages.&The&experience&is&carried&across&devices&and&media.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
27/36
In&the&same&way&that&you&can&keep&watching&a&nelix&movie&as&you&jump&from&device&to&
device.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
28/36
Or&read&a&Kindle&book&everywhere.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
29/36
So&heres&my&thought.&That&the&way&that&we&can&bridge&the&gap&in&our&ability&to&create&
compelling&user&experiences&that&span&scales&is&through&using&narrave&to&tell&stories&that&
link&together&the&individual&acons&across&devices&and&between&screens.&I&think&that&this&is&
already&happening.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
30/36
One&of&my&favorite&ubicomp&products&from&the&toy&world&is&five&years&old&and&was&a&
commercial&failure,&but&I&think&it&points&to&whats&possible.&It&was&called&Clickables&and&it&
was&developed&for&the&Disney&Fairies&lines&by&a&Hong&Kong&company&called&TechnoSource.&It
linked&the&virtual&world&Pixie&Hollow,&the&Fairies&virtual&social&network&and&chat&space,&with
the&physical&world&of&the&girls&who&were&the&target&audience&through&the&use&of&smart&
jewelry.&
&
Source:&Disney&Clickables&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
31/36
When&two&kids&put&their&Clickables&bracelets&together,&their&avatars&would&link&up&in&Pixie&
Hollow.&Moreover,&all&of&the&products&had&simultaneous&onlineoffline&existence,&so&for&
example,&when&a&girl&got&one&of&the&charm&bracelets,&she&could&touch&the&SBconnected&
jewelry&box&and&her&fairy&avatar&gets&a&version&of&the&same&charm,&plus&some&unique&magic&
buerflies&she&could&only&get&by&geng&one&of&these&charms,&or&being&friends&with&
someone&who&did.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
32/36
Now&the&execuon&of&the&Clickables&experience&leU&much&to&be&desired,&but&I&think&that&the&
basic&components&of&weaving&the&augmented&physical&device&with&the&social&network&was&
powerful.&The&mode&of&use&explained&how&the&charms&and&the&jewelry&worked&provided&a&
jusficaon&for&why&to&use&it.&The&narrave&is&a&kind&of&scaffolding&on&which&the&experience&
is&built.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
33/36
In&the&grownup&world,&we&have&this&collaboraon&between&Diageo&and&Evrythng,&a&Swiss&
Internet&of&Things&company.&Boles&of&a&giU&whiskey&have&a&unique&QR&code&that&leads&to&
an&app&that&allows&the&buyer&to&record&a&video&for&the&recipient.&The&technologywhy&its&
there&and&how&to&use&itis&embedded&in&a&narrave.&Its&a&giU.&Youre¬&going&to&drink&
this&special&bole,&its&for&giving,&and&heres&some&technology&to&help&with&that.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
34/36
This&is&Good&Night&Lamp,&a&set&of&paired&augmented&objects.&The&name&itself&creates&an&
implicit&narrave&that&tells&you&how&to&use&these.&The&narrave&is&what&drives&the&design,&
the&interacon&and&what&jusfies&the&technology.&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
35/36
So,&in&conclusion,&we&need&to&think&about&transmedia&narrave&as&a&means&to&unify&
augmented&world&experiences&across&devices,&across&uses,&and&across&scales,&to&bridge&the&
device&disconnects&that&vex&us&and&which&we&call&second&screen&experiences&and&
appcessory&experiences.&As&designers&its&our&jobs&to&tell&people&why&and&how&they&need&to&
use&our&technology.&In&effect,&we&already&tell&stories,&but&I&think&were&doing&a&terrible&job&
of&telling&stories&about&the&augmented&world,&and&why&its¬&just&important,&but&crical&that&people&understand&and&engage&with&it.&We&didnt&need&the&desktop&metaphor&forever,&
but&it&certainly&helped&during&a&crical&period&of&compung&to&give&people&a&structure&with&
which&to&understand&the&potenal&of&computers&and&how&they&could&take&advantage&of&
that&potenal.&Similarly&we&need&new&stories&to&tell&about&our&devices&today,&we&need&
stories&about&the&augmented&world.&
&
Star&Wars&meline&(CC)&by&williampcoleman:&hp://www.flickr.com/photos/20919058@N05/5850117502/&
-
7/28/2019 Sculpting the Augmented World: Scale and Narrative in Augmented UX design
36/36
Thank&you&