acting naturally with information - ia summit 2017

MarshaHaverty

Actingnaturallywithinformation

@mjane_h

2017InformationArchitectureSummit

TheTreeofLifeArtistic illustration of thetreeoflifebyEvogeneao

I started out in biology; the tree of life was my first taxonomy.(NOTE: this image is an artistic illustration of evolutionary complexity. For the latest structure with scholarly rigor, see Banfield and Hug: http://www.nature.com/articles/nmicrobiol201648 )

Then I learned that not just biology had a classification system…

...Any subject could.

Instance

Instance

Instance

Instance

Instance

Instance Instance

Instance

Instance Instance

Instance

InstanceInstanceInstance

Instance

Instance

Instance

Instance

Instance

InstanceInstance

Instance

Instance

Instance

Instance

Instance

Instance

Instance

Instance Instance

Instance

InstanceInstance

Instance

Instance

Instance

And you could structure the information.

Instance

Instance

Instance

InstanceInstance

Instance

Instance

Instance Instance

Instance

Instance Instance

Instance

Instance


Instance

Instance

Instance

Instance

Instance

Instance

Instance

InstanceInstance

InstanceInstance

Instance

Instance

Instance

Instance

Instance

Instance

Instance Instance

And the whole thing could move with an actor's meaningful action.

Instance

Instance

Instance

InstanceInstance

Instance

Instance

Instance Instance

Instance

Instance Instance

Instance

Instance


Instance

Instance

Instance

Instance

Instance

Instance

Instance

InstanceInstance

InstanceInstance

Instance

Instance

Instance

Instance

Instance

Instance

Instance Instance

Structure could even facilitate it.

Instance

Instance

Instance

InstanceInstance

Instance

Instance

Instance Instance

Instance

Instance Instance

Instance

Instance


Instance

Instance

Instance

Instance

Instance

Instance

Instance

InstanceInstance

InstanceInstance

Instance

Instance

Instance

Instance

Instance

Instance

Instance Instance

InformationArchitecture

I'll always love biology. But I'd found information architecture. That's home.

Earthimage:NASAApollo 17crewhttps://upload.wikimedia.org/wikipedia/commons/thumb/7/78/The_B lue_Marble.jpg /1023px-The_Blue_Marb le.jpg

Sagan(1994)

And, that’s home. That’s us. (Carl Sagan, Pale Blue Dot, 1994). A bunch of tribal hunter-gatherer poets. As designers, we’ve opened a lot of wormholes to new information around here, designed information. As humans, we’re really good at assembling our actions with designed information. Some designs are complicated, hard to understand, and we cope, and some are clear, aligned with out values, and we thrive. But whether we are engaging something difficult or clear, there’s something deeper about us as humans that makes the way we behave feel natural, or distinctly off.


GirlwithAmazonEcho:Android Central, Infraworks ARSandbox:Autodesk, TiltBrushVR:Google,Womanhailingdrone: Mercedez-Benz, self-assembling material:Adventurine,generativedesign forachair:DavidBenjamin, Autodesk

Varietyindesignedinformation

Manifestation

Agency

Actiontoengage

The information we’re experimenting with around here has more variety in how it manifests to actors, the kinds of actions actors must use to engage it, and in its agency (autonomy, shades of intelligence).


Naturalvs.Unnatural

Informationphysicallypresentvs.simulated

Actionstoengagenewvs.familiar

Doesnotdependon:

Some of these things feel pretty natural, and some of them feel like novelties that we don’t really want to be doing for very long. But the distinction doesn’t depend on whether the information is physically present or simulated; it doesn’t depend on whether the kinds of actions we use to engage is are new or familiar. Throwing a hologram across the room or talking at an appliance or hailing a drone can all come to feel perfectly natural. The distinction between natural and unnatural comes in something deeper still about human behavior.

TheObserver fromFringe:MichaelCerveres

Thenatureofhumanbehavior?

We don’t have a handy gadget like the one the Observer on the science fiction show Fringe had through which to peer at the nature of human behavior.

TheObserver fromFringe:MichaelCerveres

ECOLOGICALPSYCHOLOGY

Thenatureofhumanbehavior?

But we do have a very powerful lens called Ecological psychology. This field was founded in the 1970s and is just now coming into its own.

Golanka (2015)


Eventsofouractions Eventsofinformation

Humanbehaviorisaboutaligningevents

(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

Golanka (2015)




Golanka (2015)



Eventsofouractionsalignedwitheventsofinformation

assembleasystemwithdynamicsthatcompleteatask

Behavior is a system. Behavior includes us, but isn’t only us. It’s the events of information too.

We’realigningouractionswithinformation,butitfeels tousasifwe’rewieldingthedynamicsoftheworldtoactintask-fulfillingways

Barrett(2011)

We’realigningouractionswithinformation,butitfeels tousasifwe’rewieldingthedynamicsoftheworldtoactintask-fulfillingways

Barrett(2011)

We’renotactingtoimposeourwillontheworld,we’reactingtoreceivetheworldinausefulway

Image:BarcroftMediaviaDailyMail, 3Dimageofhumanbrainconnections

That changes the role of these beautiful brains we walk around with. Our brains don’t have to do all this processing, predicting exactly how we’re going to behave before we do it. Our brains act as coordinators: they coordinate the way we assemble our actions with information in the world around us.

Natural humanbehavior?

But, what makes behavior feel natural?

Naturalhumanbehavior

Selectingaction

Golanka (2015)

Controllingaction

Natural human behavior is about two things: selecting action and controlling action. (NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

SelectingactionSwitchingaction

Stylingaction

Selecting

Golanka (2015)

GifofAlondrade laNarra:http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de- la-parra-is

Selecting is about deciding for a given task which of the possible actions will we engage (if we’re thirsty and we see a glass of water we decide to drink from the glass), we decide when to switch actions (as we are reaching for the red glass, someone says, “take the blue glass” so we switch to reach for the blue glass), and we style the way the action unfolds (if we notice that the glass is fragile, we slow our actions and pick it up carefully). Animated gif: http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de-la-parra-is


Stylingaction

Selecting

Golanka (2015)

ACTORASCONDUCTOR

GifofAlondrade laNarra:http://indieclassical.tumblr.com/post/138392136778/exercisesinhumiliation-alondra-de- la-parra-is

When an actor does this with designed information, it feels to her like she’s conducting with information. Browsing a website, the actor anticipates the performance of the categories as she selects one; then aligns with the meaning of the results to anticipate and selecting more, sometimes switching to go in another direction. It’s this cascade of selecting and switching that feels like she’s conducting with information.


Stylingaction

ACTORASCONDUCTOR

Berrypicking model forsearch:MarciaBates(1989)Golanka (2015)

Cascadeofselecting,switching,stylingAnticipating-aligningwiththeperformanceofinformation

We’re also conducting when we perform digital search. In Marcia Bate’s berrypicking model for search, the actor anticipates the performance of the domain indexing in selecting and styling her initial wordings of her query. She then aligns with the meaning of the results, grabbing the good stuff, and then selecting and styling and switching her query to anticipate further performance, aligning with the meaning of the results, further refining her anticipation of the performance of the system. Actor-as-conductor is a cascade of selecting, switching, styling in which the actor is anticipating and aligning with the performance of the system of information.

Selectingaction

Controllingaction

Naturalhumanbehavior

Golanka (2015)

Selectingisbehavior:theactionofselectingaction

Selecting is an event: the action of selecting action. Now, we’ll look at controlling action. (NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

ImagebypermissionCherrico PotteryLLChttp://www.cherricopottery.com/

Controlling

Controllingactionunfoldingcontinuouslyovertime

Golanka (2015)

ACTORASSCULPTOR

We are always orienting ourselves physically in place and time with our surroundings. When we act in placetime, we control the way our physical actions unfold continuously over time with information in our surroundings. When an actor engages designed information that does this, it feels to the actor like she’s sculpting with information.

ACTORASSCULPTOR

Golanka (2015)


The industrial design of a car supports actor as sculptor. The steering wheel, the gas and brake pedals, and the transparent windshield are all streaming information that the actor relies on to control action unfolding over time; the actor feels as if she is wielding the dynamics of the surroundings so that she and the car are receiving the right road, and the lane stays centered around her and the car, and the other cars flow off to her sides instead of into her.

Golanka (2015)


ACTORASSCULPTORACTORASCONDUCTOR

The industrial design also supports actor-as-conductor of information. A glance at a map to select the right road. A glance at the speedometer to decide to switch her actions to slow down: she engages actor-as-sculptor behavior, relying on the shifting position of her foot on the brake pedal, and the way the visual information is streaming through the windshield to control how she slows down continuously over time. (NOTE: Experienced drivers are really bad at mimicking the act of driving. That’s because drivers don’t decide in their heads how to handle the steering wheel or adjust the gas and break pedal positions. They rely on the continuous flow of information to help them choreograph the behaviors of slowing down and choosing the right road and staying in the lane and moving around a car. Without this continuous flow of information to rely on, the meaning of the behavior is broken.)


Historically,ourfieldlivedhere

Historically, our field lived here– actor as conductor– and actor as sculptor was the stuff of industrial design. For actor-as-conductor designs, there is of course some physical action that has to happen to get the cursor on the pixels, or get the fingertip on the target, or say the words to trigger the response of the system, but there were lots of degrees of freedom for how the actor controlled physical actions over time. Whether the actor was sitting down, or walking, or reclining, and the arc and speed of the motions to work a mouse or tap a screen didn’t participate directly in the meaning of the activity.


Historically,ourfieldlivedhere

Thethreadofmeaningiswanderinghere…

But, now the thread of meaning is wandering over here, to actor as sculptor.


AutodeskAlias samplecarconcept

Same with physical product design at smaller scales. Designers of physical things like cars and products are always looking for ways to experience their designs before they are brought into the world in physical materials.

Feelhumanproportionality,howdesignedobjectswilloccupysurroundings


AutodeskAlias samplecarconcept;womanwithTiltBrushbyGoogle;TiltBrushcardoodle bywordpresshelp1234

They are experimenting with sketching in 3D with simulated light trails. They walk around in 3D drawing the geometry exactly as it will take up space. They feel the curves, they can walk around the designs, feeling their proportionality to humans, how the designed objects will physically occupy their surroundings. This is actor-as-sculptor of information: the way they control action continuously over time to draw the light trail is the meaning. There are no degrees of freedom in their physical actions because the physical actions participate in the meaning directly.


Designs inAutodeskRevitandVRbySamuelArsenault-Brassard

Building architects are experimenting with the handoffs across conducting and sculpting with information. This architect has detailed a building in architectural software (conducting with information). He brings the building into 3D space in order to sketch some new concepts. He scales the building down so he can walk around it and look down on it. He relies on information about the way the building occupies space to guide his hand motions as he sketches new towers into the existing geometry. He can feel how the curves will occupy space with respect to the rest of the building.


Designs inAutodeskRevitandVRbySamuelArsenault-Brassard

Then, he can bring his rough sketch back into the software and do all the conducting (selecting, switching, styling) to detail the sketch and rationalize it with the building.


Workwithhologramsasphysicalobjects

MIT is experimenting with focused sonar that creates mechanical force against an actor’s hand. Now actors can do things like touch holograms, and explore their surfaces and textures. An actor could work with holograms as physical objects, relying on haptic information to help control physical actions continuously over time. (Image: MIT)


High-precisionmotiontrackingcollapsesdegreesoffreedomofspacetime actionintomeaning

Image:GoogleSoliproject

Google is using radar to very precisely track the position and motion of human fingers. The actor can mimic things like turning a dial, moving a slider, pushing a button. These are still things for conducting (selecting, switching, styling), but there is a tiny bit of actor-as-sculptor: the actor’s fingers become tactile information for each other to rely on to control action continuously over time. It’s just fingers now, but soon it will be hand, arm, the actor’s entire body position at high precision that could participate in sculpting with information.

ActorasConductor

Selecting action, switchingaction, styling action

Dynamicsof

surroundings

Concepts

INFORMATION

Physicalalignment

Semanticalignment

MEANINGGolanka (2015)

Golanka (2015)

ActorasSculptorControlling action

unfolding overtime

Structureofnaturalhumanbehavior

Those are just a few examples of how we’re stating to need to worry about both human behaviors. Now I want to build out the structure of natural human behavior for both of these. For actor as conductor, the meaning is semantic alignment: why we’re selecting an action, why we’re switching action, why we’re styling the action. For Actor as sculptor, the meaning is physical alignment. For actor as conductor, the information is about concepts; for actor as sculptor, the information is about the dynamics of the surroundings. Let’s look at what that means.(NOTE: if you read this Golanka 2015 paper, I recommend as background, first reading her paper with Andrew Wilson, “Embodied cognition is not what you think it is,” available in full from Frontiers in Psychology.)

Thisisnotinformationforsurfing

Underlyingdynamics

This is not information for surfing. This is how wind transfers friction to water molecules, pushing and pulling them to form a wave. This is the underlying dynamics that create a wave, but this is not information for surfing.

Thisisinformationforsurfing

Invariantstructure–theshapeoftheopening–createsaffordancesforasurfertostayinsidethisplacemadeofliquidinformation

This is information for surfing. This expert surfer doesn’t see these dynamics, he sees the enclosure of a wave as a place to inhabit. And he aligns his actions with information about the wave continuously over time to keep the enclosure around him. The shape and slope of the opening of the wave is information that he can use to position the tip of his surfboard. He is calibrated to the way he must align his actions to this information; the information becomes affordances for surfing for him. And when the shape of the opening changes, when he feels water hitting his back, he knows that his place is collapsing and he needs to switch action, engaging other affordances, to quickly get out.

Skeletalmuscleanimation:EleanorLutz;

Thisisnotinformationforgoalkeeping(soccer)

Underlyingdynamics

This is not information for goal keeping for soccer. This is the underlying dynamics for how our muscles contract to move our skeletons.

Thisisinformationforgoalkeeping(soccer)

Point light image:https://quote.ucsd.edu/cogs91/

This is information for goal keeping for soccer. Twelve points of light are enough to give information to a goal keep about a human kicking a soccer ball. The goal keep can align her actions to this information to prepare to stop the ball. She doesn’t have to have seen this particular human kicking a ball, she doesn’t have to have seen ball kicking from this particular perspective in this particular light. She sees the relationships among how one foot plants while another moves with respect to the knee. This becomes invariant structure, information about ball kicking, that work as affordances to align with for a goal keeper. (NOTE: visit the URL above to see the animated gif to get the idea of this information.)

Informationaboutconceptsisdifferent

MacrophotoofKindle Readere-ink;vocaltractanatomyBrunoDubuc

Pointtomeaningbyconvention

FREEDOM

SPOON

(Notaboutunderlyingdynamics)

When we hear or read words like Spoon and Freedom, those things are not about the underlying dynamics of the chemicals or electrical process creating the text, or the function of the larynx creating the sounds. These words point to meaning by convention. We have decided what these mean in agreement spanning time and culture.

For actor as conductor, we assemble semantic alignment (why are we selecting, switching, styling an action?) with information about concepts by way of convention. For actor as sculptor, we assemble physical alignment with information about the dynamics of our surroundings by way of affordances. We see that conducting with information is not dependent on time. We can conduct in quick succession, or over weeks or lifetimes. Time does not factor into whether conducting with information feels natural or not. For actor-as-sculptor, natural behavior is directly dependent on being continuous over time. And that places some very important demands on the kind of information that can support actor as sculptor.

For actor as sculptor, information must be dense, persistent, and lawful. And it must be made of perceptual information. Language is not dense enough, persistent enough, (though it acts as if it’s lawful across an actor’s lifespan) to rely on to control action continuously over time. The two key things actors use perceptual information for to sculpt with information are: orienting & wayfinding, and prospecting with objects. We’ll look at both of these.

Orienting&Wayfinding

Dense,persistent,lawful

VisualInformation

Surfaces

Edges

Textures

Proprioception,Kinesthesia

We are always orienting ourselves physically in our surroundings. We humans are wired for proprioception and kinesthesia: we know where our bodies are in space, and we know when we’re moving. We also rely on visual information for information about the relationships among surfaces, edges, and textures in our surroundings that we can use as affordances for us to move: this path is walkable to me, these stairs are climbable to me, this gravelly hill is not passable to me. And this information is dense, persistent, and lawful. We can rely on it to control our actions continuously over time.

Wayfinding

HapticInformation

Dense(enough)persistent(enough)lawful

Simulates surfaces,edges

This woman has low vision and can’t rely on visual information for wayfinding. She’s wearing a device that simulates information about obstacles as haptic information. The haptic information is much less dense than visual information, and is not persistent (it’s not a constant flow, but kicks in with proximity to obstacles). But it’s dense enough, persistent enough, and lawfully specifies obstacles in her surroundings that she can use as information for orienting and wayfinding.

Wayfinding

SonicInformation


IncomingcarsTrain/notrainCrossinglighttimersConversations

Physicalobstacles

Incoherentwith“everydaysounds”forwayfindingGaver (1994)

UNNATURALBEHAVIOR

This system can use sound instead of haptic information about obstacles. Sound is dense enough, persistent enough, and lawful, but the problem with sounds is that it’s directional, but not spatial. It piles up. That’s great for music, but not for wayfinding. Haptic information can use spatial relationships to represent multiple obstacles and their relative positions. Sounds just pile up. The other problem with sound fir wayfinding is that humans rely on many other very important sounds: incoming cars, train/no train, crossing light timers, conversations. And the sounds about physical obstacles steps on those so they are not dense, persistent, and lawful. This is unnatural behavior. (NOTE: see Gaver 1994 in references for an excellent discussion of ecological taxonomy of “everyday sounds” humans rely on for information.)

So we see that we rely on many kinds of perceptual information for orienting and wayfinding, and we see that, in our designs to support this, we must ensure the information we use is not only dense enough, persistent enough, and lawful, but coherent with the other perceptual information actors are relying on for the activity. So, we’ll add coherence as a design consideration.

VRhasacoherenceproblem

"Istartedforgetting whichwallswerevirtualandwhichoneswerereal,andIendedupwalkinginto afewverysolidwalls"

“Aplayerattemptedtopushhishandthrough thefloor tosculpt something’”

Disruptscriticalinformationforplacetime orientingandwayfinding


UNNATURALBEHAVIOR

Virtual Reality also has a coherence problem. VR disrupts critical information for placetime orienting and wayfinding. Players of VR games report things like, “A player attempted to push his hand through the floor to sculpt something.” Another: “I started forgetting which walls were virtual and which ones were real, and I ended up walking into a few very solid walls.” These designs are mitigating this by adding in wall alerts and visual grids to show were physical walls are. These things are dense enough, persistent enough, and lawful to represent the walls. The problem is that they are like aroras from another reality slicing through the information from the present reality. They disrupt the dense, persistent, lawful information in the present reality for orienting and wayfinding and ask the actor to align with two sets of conflicting information. That’s unnatural behavior. Quotes from: http://www.businessinsider.com/virtual-reality-gamers-are-literally-colliding-with-the-physical-world-2016-4

Imagecredit:Designboom (AlisonBrooksArchitects‘TheSmile’)

“Oneseestheenvironmentnotjustwiththeeyesbutwiththeeyesintheheadontheshouldersofabodythatgetsabout”

J.J.Gibson, founderEcologicalPsychology

Gibson (1979)

The other problem with virtual reality is that we humans naturally need to explore our environment. JJ Gibson, the founder of Ecological Psychology says…

Imagecredit:Designboom (AlisonBrooksArchitects‘TheSmile’)

PrincipleofactionforVR:“Aparticipantmusthaveaffordancesformovingaboutinthescene…takeactionintheworldandperceivetheeffects.Thisispartofthelargersenseof personalagency.”

BrendaLaurel

Laurel(2016)

Brenda Laurel has a principle of action for VR...

So, we’ll add explorability as a design concern.

Prospectingwithobjects

Image:https://quote.ucsd.edu/cogs91/

Colorrelationshipsbecomeaffordancesforbananaedibility

Surfaces

green:yellow:brown

We don’t just start using objects, we prospect with them using information about them. If we have experience eating bananas, the relative amounts of green:yellow:brown color on the surface of the peel becomes information to us. We become calibrated to the relative colors to know that a banana like this will have a peel that droops, we anticipate seeing brown spots on the fruit, we’d expect the fruit to be a little mushy and sugary. This information becomes affordances that we can rely on about whether this banana is edible to us.

object affordances: Gibson (1979)

Prospectingwithobjects

Surfaces Edges

Textures

“smooth”

“glassy”

Relationshipsofsurfaces,edges,texturesbecomeaffordancesforfindingmyphonebydynamictouch

Curveofedgestoface

Endedges:sideedgesEdgethickness

We can also prospect with objects using dynamic touch. I know my phone visually, but I also know my phone by touch. If I’m reaching into my bag to retrieve my phone, I don’t just touch it once and know that it’s my phone, I must explore the relationships in the haptic information about the surfaces, edges, textures of the phone. I understand that the surface is glassy and smooth. I understand the relationships among the end edges to the sides; the edge thickness; the curve of the edges to the face. This haptic information becomes affordances to me to retrieve my phone.

dynamic touch: Gibson (1979)

Areoursimulatedobjectsexplorable?Dotheyknowwhatkinds ofobjectstheyare?

Prospectingsimulatedobjects

Whatkindsof informationshould theysimulatetorevealnew underlying dynamics?

Newkindsoftransformations

Permutational Computational

SmartMaterials Sameconcernassimulatedobjects

Tactile/HapticVisual Sonic Sameinformation asphysicalobjects

What about prospecting simulated objects? We can simulate the same kinds of infomraitno that physical objects have: visual, tactile, sonic. But we can also infuse simulated objects with new kinds of underlying dynamics that physical objects don’t have: new kinds of transformations. But, we also need to infuse them with information about these new kinds of dynamics that humans can rely on. We need to start thinking about what a permutational affordance is like; what a computational affordance is like. Smart materials have the same concern as simulated objects: smart materials are physical materials that are infused with new underlying dynamics. How do we coax smart materials to radiate information about their underlying dynamics for humans to calibrate with and use as affordances? (NOTE: We also need to ask, do we truly need a spacetime simulation? If the information is more abstract than the way the object orients and embeds in spacetime and relative to the actors proportions, are we just making a 3D hologram versio of a skeuomorphism?)

Simulationshavenomass!


“There'snophysicalfurnituretobreakyourfallwhenyoutrytositinavirtualchair.

Someoneasked,Wheredidthatbruisecomefrom?

I achievedpresence.”

UNNATURALBEHAVIOR “Breakingtheecologicallaw”Wilson(2012)

Yet…theyprovideaffordancesforphysicality COHERENCE

Tactile/HapticVisual Sonictouch-ability, throw-ability, transformability

sit-on-ability,climb-on-ability

Yet…onephysicalityaffordanceisnotlawful

Visual

Explorability leads to a coherence problem for simulated objects. Simulations have no mass! Yet, they provide affordances for physicality, we perceive them as objects that take up spacetime embedded in our surroundings. Get dense, persistent, lawful visual and haptic and tactile and sonic information, depending on how they’re wired up, about touch-ability (surfaces and textures), pickup-ability, throw-ability, transformability, and so on as far as our imaginations take us. Yet, one aspect of their affordances for physicality is not lawful. We get visual information for things like sit-on-ability, climb-on-ability. Surfaces and textures suggest the lawful presence of mass. These are affordances we’ve been attuned to our whole lives, and we will engage with. One person in VR recently said: “There’s no furniture to break your fall when you sit on a virtual chair. Someone asked, ‘Where did that bruise come from?’ I achieved presence.” Andrew Wilson, a present day Ecological Psychologist, would tell us we are breaking the ecological law by simulating affordances for properties that our objects do not lawfully provide. That’s unnatural behavior.

Truemixedreality:3Dscannedobjects+simulatedobjectscoexisting


Scannedobjectshavephysicalaffordancessimulatedobjectsdon’t

Howcanwehelpdistinguish themsotheycancoherentlyco-exist?

When we get away from this hard divide between virtual reality and holograms in “augmented reality,” and get to true mixed reality, we will have 3D scanned physical objects co-existing with simulated objects. Scanned objects have physical affordances that simulated objects don’t. How can we help distinguish them so they can coherently co-exist?(NOTE: see Brenda Laurel’s reference for a description of mixed reality as a spectrum.)

So, that’s the structure of actor as sculptor behavior. Now let’s build out actor as conductor. Information to support conducting can take the form of both language and perceptual information. Language for conducting is about coordinating and understanding. We’ll start with coordinating.

Ecological psychology tells us that natural human conversation is about coordinating. When someone speaks and someone listens, that evokes concepts. Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of cultural possibilities among the participants. Something as seemingly here-and-now as asking which movie should we see tonight locates speakers and listeners in place and time, and evokes all of their placetime histories as an array of cultural possibilities. Speaking is anticipating cultural possibilities by improvising wordings. Listening is aligning with the discovered nature of the cultural possibilities, and entraining with rhythms, timing, pronunciation, grammar of the participants in the conversation. A conversation is a creative tension: anticipating and aligning; improvising and entraining. The acts of conversation feed back into the structure of the cultural possibilities, changing the structure over time. The overall behavior of this system of conversation is about coordinating actions among the participants to realize values.

NaturalHumanConversation

Hodges(2014)

Acreativetensioninwhichparticipantsresonatetothelargerstoryinwhichtheyareparticipating,orientinganddirectingthemtowardagoodcontinuation

Coordinatingactiontorealizevalues

Giant,ongoing,distributedconversationwithhighlyrefinedentrainmentofgrammaracrossitsplacetime histories

Manyoverlappingconversationswithparticipantsresonatingtoverydifferentstructuresforgoodcontinuation

When someone that aligns with a particular cultural structure for good continuation encounters acts of conversation that resonate to a very different structure for good continuation, there is nothing to align with. The array of cultural possibilities is invisible to both parties. Is there any way for us to visualize these structures to help facilitate some kind of understanding (if not aligned values) across conversations?

Conversation

Ifyouwanttovisitvirtualreality,justspeaktosomeone,andlisten,withintenttowardagoodcontinuation

Avirtualsemanticarraymadeofyourcollectivehistoriesofcultureateveryplacetime levelwillemergeallaroundyou

So, we see that natural human conversation is about coordinating. Now let’s look at using language for understanding. (NOTE: I am using “understanding” as the label here to mean the notion of design for understanding in Morville, Rosenfeld, Arango (2015).)

When an actor engages a design that supports something like shopping, banking, creating, hailing/styling a service, etc., the actor is engaging with language based on classification: the underlying information architecture of the designed system. And the purpose of the behavior is understanding/performing for a good continuation. The actor engages the design and that evokes concepts. Concepts point to meaning. Meaning points to values. And that’s enough to evoke a virtual array of classificational possibilities. This takes the form of the underlying system of information architecture. The actor anticipates the performance of the classification by improvising selecting/switching/styling. Selecting/switching/styling can take the form of browsing, searching, commanding, hailing, parameterizing, etc., whatever the design facilitates. The actor then aligns with the meaning of the performance of the system (discovered nature of the classificational possibilities). And entrains with the labeling and grammar of the system. Entraining is vital here: if the actor does not entrain to the labeling and grammar of the system, the virtual array of meaning falls apart. We know from our field of IA that a system must be clear, and resilient across channels and contexts. The behavior of the system is about understanding/performing for a good continuation.

Now we are playing with the parts of this system: autonomous digital agents offload from the actor the tension of anticipating/aligning with the classificational possibilities. The agent does this work. But, when the agent needs to interrupt the human with a decision point or status, the agent must bring the right kind of information to let the human anticipate/align with the digital agent for a good continuation. We are also injecting machine learning into the system. That can greatly expand what is allowable for entraining with the system. Entraining is still vital, but the human has more flexibility in what is allowed for labeling and grammar. Machine learning in which the system is expanding the nature of the underlying classificational possibilities will expand the scope of what the human needs to anticipate and align with. I think that things like machine learning and conversational UI are really about expanding the tolerance for imprecision the human must have to engage the system.

Human-Systemimprecisiontolerances

NARROWTolerancePreciseentrainment

WIDE ToleranceForgiving entrainment

NaturalLanguageProcessing MachineLearning(entrainingactor’swordings)

ConversationalUI

Standardsearching,voicecommanding

GesturalUI(degrees offreedomofmotion,subtlevselaborate)

For any given system, the system has a tolerance for imprecision in what is needed to engage it. A narrow tolerance for imprecision would be a system that requires very precise entrainment with labeling and grammar. A wide tolerance for imprecision would be a system that has forgiving entrainment with labeling and grammar. Something like standard searching and voice commanding have narrow tolerance for imprecision. We can do things like autosuggest, and spelling forgiveness, but these are generally very precise labeling and grammar requirements. Conversational UI expands the tolerance for imprecision by suggesting the classificational possibilities ahead; this gives the actor a hint of the performance of the system as she is selecing/switching/styling. Natural language processing expands the tolerance for imprecision. Machine learning in which the system entrains on the actor’s particular wordings greatly expands the tolerance for imprecision with a very forgining entraining requirement on the human. Something like gestural UI right now has a narrow tolerance for imprecision. I believe the tolerance for imprecision should be much more forgiving to mime concepts, both in terms of the degrees of freedom of motion to give the system the gist of the concept, and in whether the human can use subtle motions or must do elaborate gestures.

So, we’ll add imprecision tolerance as a design consideration. For actor as conductor, we can also use perceptual information. This takes the form of mapping: infusing concepts with perceptual qualities in the form of visualizations and notifications: maybe a sound or haptic buzz or color change to signal a state change.

Mapping

Air flowandheattransfer simulations, Autodesk

Visualization Values-realizing, goodcontinuation, clear,resilient

Curvature mappedtoairflowColormappedtotemperature

This is a visualization of the air flow and heat transfer in a hospital operating room on the left, and an airport walkway on the right. Color is mapped to different temperature ranges. The arc of the lines shows the direction of the airflow. We could describe something like this in words, but this visualization captures the ambient comfort of these places as a quick, visceral understanding. That’s because perceptual information and language feel different in terms of the ease of the flow of engaging them.

Perceptionflowseasily

Tacit

Reflexive

photo credit: Jason Pratt via Creative Commons

Languageisviscous,moreworktoflow

Awareness

Associativity

Requiresattention

These kinds of information have different viscosities, or ease of flow.

Perceptualinformation

Language

Mercedes-BenzF015ConceptCar

VISCOSITY:easeofflowofconducting

visceral

Lowviscosity,visualinformationaboutdensity,directionality,color

Situationalvisualizationhumancanmanipulate?

intenseconcentration

concep

tual

Highviscosity,fullentrainmentwithauthor’swordings

Viscositywithprecisegrammar/labelentrainment,linearoversimultaneous

Self-drivingcarhandofftohuman…?

Howwillthecarinterruptus?Howwillthecarmapsculptinginformationtoconcepts?

intensecoordination

Something like the airflow visualization is all perceptual information, no language. It is a very visceral way to understand with information. Something like reading a novel is all language, high viscosity entraining with concepts and the author’s particular wordings. It is a conceptual way to engage meaning and requires active attention. Something like talking to Alexa also is all language, and requires precise entrainment with the grammar and wordings. It is also high viscosity because it requires linear order (you can only do one thing at a time). What about when we think about the interaction among a human and a self-driving car? The car is offloading all of the sculpting behavior of driving a car. What about when the car needs to interrupt the human for a critical decision? How will the car sum up all of the perceptual information it has been aligning with to control the actions of the car continuously over time into concepts to align with humans? Will it use wordings? Will it offer an ongoing visualization of the situation that the human can manipulate for a decision point? Like a flight tracker the human can tweak? We need to worry about the form that information will take in this kind of important handoff between sculpting (the car) and conducting (the human). NOTE: I fear my brief examples seem to imply language viscosity is negative. On the contrary! Please see my 2015 IAS talk for a more detailed treatment of viscosity: https://www.slideshare.net/mjane_h/what-we-mean-by-meaning-new-structural-properties-of-information-architecture-ias15-47485499?qid=95845315-8b4c-4c4b-8949-48b464ba8547&v=&b=&from_search=5

So, we’ll add viscosity as a design concern. We see that the structure of natural human behavior for actor as sculptor is physical alignment using the dynamics of the surroundings by way of affordances. It relies on perceptual information for orienting & wayfinding, and prospecting with objects. Perceptual information for sculpting can take many forms, but must always be coherent with the way other kinds of information are being used for the activity. Information for sculpting must be dense enough, persistent enough, and always lawful. Natural human behavior for actor as conductor is semantic alignment using concepts by way of convention. Both language and perceptual information can be used to support conducting. Language for coordinating takes the form of human conversation. Language for understanding takes the form of systems of classification. Perceptual information for conducting is a mapping to concepts using visualization or notification. Information to support conducting must be values-realizing, a good continuation, clear, and resilient.

ACTORASCONDUCTOR

Controlling action

ACTORASCONDUCTOR

ACTORASSCULPTOR

Whichbehaviorisourdesignsupporting?

Degreesoffreedomincontrollingaction Actionunfolding continuouslyovertimeparticipatesinthemeaning


Followthethreadofmeaningwhereveritmaywander

As designers and information architects, we follow the thread of meaning wherever it may wander.

Naturaldoesn’thavetobefamiliar

Image:HughHochman

Natural doesn’t have to be familiar, or even physically present. Nascent technologies will always race around, and we should knock ourselves out experimenting with the possibilities.

Image:DMITRIS_KBigThink

Disruptingtheecologyofinformation

If our designs feel deep-down unnatural, and we can’t pin it on usability or usefulness or lack of clarity, or any of our other ways to say what’s good, maybe it’s because we’re disrupting the ecology of the ways our human actors are relying on information to behave.

Human behavior is events aligning,

Tunedtoalignouractions

For 100s of 1000s of years, and more across the branches of the tree of life, our perceptual systems have been tuned to align our actions…

with informationaroundus

…with information all around us.


(Not the underlaying dynamics causing the information.)


But the information. We align our actions with information around us...

torealizeourvalues

…to realize our values.

Allhumanbehaviorisinformationbehavior

From an ecological psychology POV, we see that all human behavior is information behavior.

MarshaHaverty

Thankyou

@mjane_h

2017InformationArchitectureSummit

References

Bates,Marcia(1989).TheDesignofBrowsingandBerrypickingTechniques fortheOnlineSearchInterface

Sagan,Carl(1994).ThePaleBlueDot

Golanka,Sabrina(2015).Lawsandconventionsinlanguage-relatedbehaviors,EcologicalPsychology27:236-250.

Gibson, JamesJ.(1979).AnEcologicalApproachtoVisualPerception

MarshaHaverty @mjane_h

Hodges,Bert(2014).Rightinglanguage:theviewfromecologicalpsychology.Language Sciences,Jan

Cowley,StephenandHarvey,Matthew(2015).Theillusionofcommon ground,NewIdeasinPsychology p.1-8.

Wilson,Andrew(2012).Breakingthe(ecological)law:whyillusorysoundsdon’tmakeforsafercars.Onlinehttp://psychsciencenotes.blogspot.com/2012/10/breaking-ecological-law-why-illusory.html

Barrett,Louise(2011).BeyondtheBrain:HowBodyandEnvironmentShapeAnimalandHumanMinds,PrincetonUniversityPress

Arora,Rahulet.al(2017).Experimentalevaluationofsketchingonsurfaces inVR.CHI2017,May6-112017.

FoundedEcologicalPsychology

Science incontemporaryEcologicalPsychology

Laurel,Brenda(2016).WhatisVirtualReality?PublishedonlineonMedium,15June

Thibald PaulJ.(2011).First-OrderLanguaging DynamicsandSecond-Order Language:TheDistributedLanguageView.EcologicalPsychology23(3): 210-245.

Gaver,WilliamW.(1994).Whatintheworlddowehear?Anecologicalapproach toauditoryperception.EcologicalPsychology5(1):1-29.

Covert,Abby(2014).Howtomakesenseofanymess.CreateSpace.

Hinton,Andrew(2014).UnderstandingContext:Environment,Language, andInformationArchitecture.O’ReillyMedia.

Resmini, AndreaandRosatti,Luca(2011)PervasiveInformation Architecture.MorganKaufmann.

Rosenfeld, Louis,Morville,Peter,Arango,Jorge(2015).Information ArchitecturefortheWebandBeyond.O’ReillyMedia.