PRESENTED BY

Andrew Hinton | @inkblurt | InteractConf London 2014

LANGUAGE IS INFRASTRUCTUREThoughts on the material of IA practice (updated)

HI. I’M ANDREW.AND I’M OBSESSED WITH CONTEXT.

2

contextbook.com(Due December 2014)

#shamelessplug

Hi I’m Andrew and I’m obsessed with context. In fact, I’m finishing a new book on the topic, due out this December. Writing this book was an amazing learning experience for me, and one of the things I learned the most about was the role language plays in the work we do, and in the environments we design for people.

3

“Make it easier to use the shopping cart.”

Here’s a hypothetical example most of us can relate to. Let’s say you’re doing user-experience work for a retailer’s e-commerce site. The project you’ve been assigned has what sounds like a clearly defined goal: Make the shopping cart easier to use. Ok, great. We all know what the cart is -- it’s been around on websites for a long time now. But is it really so clear-cut?

OK, BUT ... WHAT IS “CART”?

4

For starters -- what is the “cart”? It’s the kind of question that would seem to invite eye-rolls and derision ... what do you mean what is “cart”? It’s right there on the website!

http://upload.wikimedia.org/wikipedia/commons/archive/5/50/20110713132415%21ShoppingCart.svg

WILL THE REAL CART PLEASE STAND UP?

5

Cart isn’t just one thing ... sure, there’s an icon in the top right of the website. Ok.>> But the cart actually shows up in many places on the site, in various forms, and enmeshed with all sorts of other functions, like authentication, store choice, product availability, and shipping information. And that’s before we even get to checkout.

6

=

Business rulesSoftware services

User behaviors

InterfacesSerial processes

What parts of “cart” do you mean?

All of these involve language.

The cart isn’t just the cart -- it’s a signifier we use as shorthand for a hugely complex cluster of business rules, software infrastructure, not to mention interactions and user behaviors.

>> And all of these involve, or are made of, language.

The only way to work this out is to have a conversation about it, unpack the parts, and gain some consensus or direction on what the specific mission is. This is a simplified example, but I know I’m not the only one in the room who has encountered similar scenarios. I’ve worked with retailers who’ve actually organized their teams based on parts of their website: so what would the “Cart and Checkout” team actually work on?

It would be great to do the work of digging into the meaning of what we’re working on... but more often than not there’s not time or willingness to stop.

7

Physical Object

Virtual Object

Design is obsessed with tangible objects...

Design is obsessed with the tangible. Everything else feels secondary, peripheral. Make the object, and we’ll name it later. Stop talking about it and start making stuff. But what’s missing here is the context that makes these physical and virtual objects valuable or meaningful, the stuff that connects them and defines them.

http://commons.wikimedia.org/wiki/File:Samsung_galaxy_Note_-_Snap_2567.JPG

8

“That’s just semantics...”

A phrase I hear a lot is “that’s just semantics.”

>> And that, right there, crystallizes the problem. That word “just”... think of all it implies: That semantics is small, inessential, a deviation of purpose.

9

Like this fish and its relationship with water, we are so immersed in language -- it is such a native medium for human beings -- that we see right through it, and take it for granted. But it is a medium, and it does have properties that we should understand.

Language is a real medium for making.

10

Today, my talk really has just one message I hope you take away: language is a real medium for making.

That semantics isn’t fluff, it isn’t inessential. Instead, it’s crucially essential, and central.

It's not a disembodied layer of abstraction that we optionally add to the objects, places, and systems we design. It is instead both the body and soul of those objects, systems, and places.

In particular, language is what we use when designing the structure of information environments. That is, when we do information architecture. Language makes structures that are as real and critical to our lives as the walls, ceilings, and floors we see around us in this room.

http://commons.wikimedia.org/wiki/File:Kneden_van_klei.jpg

11

ORGANIZATIONS

Let’s start by looking at how semantics function as part of the environment of organizations.

LANGUAGE MAKES MAPS THAT ARE TERRITORIESENVIRONMENTS WE INHABIT TOGETHER

12

Organizational diagram of the New York and Erie Railroad, 1855

Organizations depend on language as a sort of map that they live in as a territory.

This gorgeous, early org chart from the 1800s is a fascinating example of early attempts to show structure before the mechanical, engineering metaphor had fully infiltrated the way we think of organizations. But either way, org charts are representations of social relationships, processes, collaborative structures. This railroad company, spread out geographically over space and time zones, needed an architecture that virtually organized its relationships -- an augmented-reality architecture for existing as a corporate entity.

13

REQUIREMENTS & STORIES ARE MOLDS FOR MAKING

When we create requirements or user stories, we’re putting a lot of weight on language to define the shape of what we will produce. Even though the language isn’t the product, it casts the product in its image.

card: http://itsadeliverything.com/agile-requirements-snail-feature-to-user-story-to-scenariocasting: http://tombanwell.blogspot.com/2010/09/olifant-neoprene-trunk-hose.html

HOW WE TALK ABOUT THE WORLD FINDS ITS WAY INTO OUR SYSTEMS & ENVIRONMENTS

14

“These different terms have slightly different connotations. Whatever terminology we use in our architectures, it needs to reflect these connotations.”

Jesse James Garrett “Brand Driven Information Architecture” February 2004

Ten years ago, at the IA Summit, Jesse James Garrett talked about how an organization’s brand depends on the foundational meanings of the language it uses, and that the structures we make need to take this language seriously. How we talk about the world will find its way into our products and communications whether we’re paying attention to it and bringing the necessary rigor to it or not -- so, best to do it on purpose.

15

from The First 90 Days, Michael D. Watkins

CULTURE IS A CRITICAL BUSINESS CONCERNAND OUR INTERFACE WITH IT IS LANGUAGE

In the classic new-management advice book, the First 90 days, the author stresses that to work effectively as a manager or executive, you have to understand the cultural norms and fundamental values of the organization. Management has as much to do with the language of the company as with the budgets and reporting employees.

ORGANIZATIONS HAVE STORIESTHAT DRIVE THE CONVERSATION ABOUT WHAT IS MADE

16

Underlying Story

Interface

Organizations have collective memories, and shared narratives. The arbitrary pathways of the past are often remembered as a linear story, that can be very convincing, because it leaves out all the paths that weren’t taken. And that story can influence future decisions. >>Likewise, even on a project level, that underlying story ends up being articulated in terms of interfaces. “We need to add another tab to our global navigation” or “we need an app” or “we need to be more like pinterest” are often the best way stakeholders know how to explain their problem. The interface is a semantic construction that plays a poor proxy to defining the underlying problem to begin with.

INFORMATION THAT REQUIRES ARCHITECTUREVERTICALLY & HORIZONTALLY

17

Language plays a critical role both vertically and horizontally in organizational life.

Vertically, it starts as deep as the data architecture. Data structures are pure language, structured and defined for machines. Get something defined wrong or conflated down in a database entity structure, and it can cripple an organization’s effectiveness. There are the software services that connect the data with everything above. Those, too, are developed based on definitions and requirements or user stories that were discussed, composed, and organized into priorities and structures. Then there are business rules -- the “if this then that,” cause and effect definitions that do the work of making a company’s business model be an active, working mechanism in the world. Of course, there are also what I’m calling here “corporate semantics” -- a catch-all term for how the organization culturally talks about what it does, the narratives in the organization, and even the acronyms and taboo phrases. Then all of this bubbles up to customers and end-users, in various interfaces -- digital or analog.

Horizontally, everything in the vertical stack touches all the various channels that the organization needs to manage, from websites to product names, to customer support scripts and store signage.

Information architecture has a role to play in all of these areas, because as a practice it’s about architecting coherent structures across all of these contexts. Systems that make sense.

18

ENVIRONMENT

So language has a real, tangible effect on the work we do, and the organizations for whom we do it. How can that be? It’s because language functions as a real part of our environment.

19

Language is physical.

Language is physical. We make it and consume it with our bodies. When we speak, we’re using a great deal of our body to breathe and create the sound vibrations needed for articulation, not to mention gesturing and “body language.”

When we write, we’re adding physical information to the environment, that we depend on being picked up and interpreted correctly by a reader.

We learn to read by reading aloud, so even when you read to yourself silently, your body still subvocalizes, firing neurons as if your body were making the sounds. In fact, NASA has used this subvocalized activity in new technologies that allow users to give commands without having to literally say them4.

Silent reading: http://neuro.hut.fi/~jjkujala/pdfs/perrone_bertolotti_et_al_JNeurosci2012.pdf

NASA story: http://www.nasa.gov/home/hqnews/2004/mar/HQ_04093_subvocal_speech.html

http://commons.wikimedia.org/wiki/File:Larynx_(PSF).png

http://commons.wikimedia.org/wiki/File:Witten_Blackboard.jpg

20

There’s mounting evidence that language has been with our species long enough to be a factor in natural selection — part of the environment that shaped our evolution. For example, anthropologists have shown there’s a connection between language use and the ability to create sophisticated tools and weapons. This stone arrow tip dates to many thousands of years before the emergence of homo sapiens.

Image/article: http://www.theatlantic.com/technology/archive/2013/12/researchers-discover-the-hot-new-technology-throwing-javelins/281976/Spear: http://commons.wikimedia.org/wiki/File:Spear_(PSF).png

LANGUAGE IS ENVIRONMENT

21

Language is “a form of mind-transforming cognitive scaffolding...”

... “Simple labeling adds a new realm of perceptible objects.”

- Andy Clark, Supersizing the Mind

http://en.wikipedia.org/wiki/File:CantileverScaffold.jpg

Language is environment -- it’s stuff we put into the world that allows us to make more stuff, including whole civilizations. Philosopher and writer Andy Clark, in his book on embodied and extended cognition, talks about how language is a kind of scaffolding. We create it so that we can go on to create more environment together. Likewise, names and categories allow us to rearrange the world. And they also add new structures to the environment, new objects and places.

22

A name is like a TARDIS.

Small on the outside; huge on the

inside.

Labels are like Tardises. Innocuously small and mundane on the outside, but immensely powerful, mysterious, and huge on the inside. Labels carry with them much of what the world is to us. They allow us to rearrange the world and move whole universes of meaning with a breath or a scribble.

http://commons.wikimedia.org/wiki/File:TARDIS-trans.png

23

“The list is the origin of culture.” - Umberto Eco

And when we put labels into structures, such as lists, we create new parts of our environment that enable new kinds of human activity. The list, as Umberto Eco has said, is the origin of culture. In archeological digs, lists are the most common sorts of writing found. Most of them are about commerce and mundane recordkeeping. Why? Because writing was a necessary infrastructure for markets to expand over space and time, past the natural limits of human memory.

1 http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_gawande

http://commons.wikimedia.org/wiki/File:List_of_possessions_of_a_temple_AO8110_mp3h9037.jpg

Eco quote: http://www.spiegel.de/international/zeitgeist/spiegel-interview-with-umberto-eco-we-like-lists-because-we-don-t-want-to-die-a-659577.html

24

“Taxonomies provide the lenses by which we perceive and talk about

the world we live in.”- Patrick Lambe, Organising Knowledge

As Patrick Lambe says in this really amazingly wonderful book, Organising KNowledge, “Taxonomies provide the lenses by which we perceive and talk about the world we live in.” Keep in mind, taxonomies aren’t just hierarchies; taxonomies can be lists, spectrums, facets, polyhierarchies, and all sorts of arrangements of language that create environmental structures for action and understanding.

25

The perceived functional properties of objects, places and events in relation to an individual perceiver. AFFORDANCE

PhysicalAffordance

SemanticFunction

James J Gibson, who invented the concept of affordance, talked about how the structure of the surfaces and objects in our environment has a direct, mutually coupled relationship with our bodies, which pick up information about what our bodies can do with those structures. So, these stairs are picked up by our perceiving bodies as a structure that will take us upward in space if we climb them. But we don’t know where it’s going to take us. We don’t know the contextual significance of the stairs unless we go up them and find out for ourselves, or someone tells us. >> So the intrinsic affordance of the stairs is supplemented by a label -- something I was until recently calling “semantic affordance” but I’ve changed. I’m now calling it “semantic function.” I like this phrase “semantic function” because it helps remind us that even though it’s not strictly an affordance, it still isn’t just an ephemeral bit of vapor — language has physical properties, and physical significance. It’s as much part of our bodies as anything else; it functions as part of the machinery of our environment.

[longer bit i’m skipping]Why? Because I realized that calling language an affordance wasn’t really in keeping with how Gibson framed the concept. Affordances aren’t mediated in the way language is, and they don’t require convention for meaning. Stairs don’t need to be interpreted; they will work the same for a body whether that body speaks English or Chinese. Calling language an affordance can mislead us into thinking language can be intrinsically meaningful, when it’s actually the opposite. Language is made of symbolic information, so it’s intrinsically ambiguous. It requires even more context. Donald Norman calls these signifiers -- and they are signifiers, no doubt -- but signifiers are ultimately grounded in the same physical and cognitive mechanisms that we use to climb stairs, or walk through doors.

26

SemanticFunction

PhysicalAffordance

This distinction between semantic function and physical affordance is not meant to say they are binary opposites. I think they are intimately connected, part of something more like a continuum than separate categories. The truth is, people perceive all of it as one environment. But for purposes of designing these elements, it’s important for us to know the differences and have names for them. This is part of understanding our materials. JJ Gibson touched on language and cultural systems in his work. When developing the theory of affordances, explained that affordance isn’t only about simple perception between my body and a surface; he argued that affordances between people -- because of language in particular -- create extremely high levels of behavioral complexity and meaning. These affordances coalesce to become compounded structures in our environment that are cultural constructs rather than physical ones.

27

SemanticFunction

PhysicalAffordance

Gibson talks about how a mailbox affords mailing a letter to another place. The intrinsic structure of the mailbox is just a hollowed out object that affords putting small objects into. But for people encultured in a complex system of postal mail, we perceive it as something beyond its immediate physicality. In the terms I’m framing here — The immediate physical affordance of the mailbox is an object to put something into. But semantic function allows us to know this is a special box that is connected to other systemic physical affordances beyond our perception — mail trucks, postal offices, mail handlers. >>What software does is create simulated versions of these affording systems — made entirely out of semantic information, which is what we use to make programming languages and network protocols. It abstracts the compound affordances of “mail” into something digital machines do for us. So we have to use visual and textual metaphors to represent what was physical in our world. An email inbox is not a box. It strips away the physical affordance and leaves us just with the semantic function — which requires even more care and attention to definition and context.

http://commons.wikimedia.org/wiki/File:Post_office_drivethrough_lane.jpg

28

PLACEMAKING &

SENSEMAKING

A couple of important dynamics that we depend upon language for every day are placemaking and

sensemaking.

29

When you see a city scene like this, you see placemaking happening everywhere, through the signs and labels on buildings, but also through the conversations, publications, arguments, and cultural narratives. Language certainly does work as a sort of scaffolding that enables us to build and live in places like this. But what I actually see here is also the reverse relationship: a lot of infrastructure that is built to supplement the conversations of culture and commerce. Perhaps the built environment is here to better enable the semantic environment, more than the other way around. After all, we’ve been users of language longer than we’ve been builders of cities.

http://commons.wikimedia.org/wiki/File:Taipei_City_Nanyang_Street_20130127.jpg

WHAT ONCE WAS SPACEIS NOW A PLACE

30

wikimedia.org

All it takes for humans to make something into a place is to say something about it. To name it, or mark it in some way. There were names for craters and mountains on the moon long before anyone walked on it. But when the first humans actually set foot on the surface, they marked it with this flag. No text is on it, but it’s still a rhetorical act. A semantic utterance. The official line was “we come in peace, for all mankind.” But the subtext was, “Hey Russia, look what we can do.”

31

“A one-star increase in Yelp rating leads to a 5-9

percent increase in revenue.”

- Michael Luca, Harvard Business School Working Paper

Digital networks make semantic environments a super-charged, geometrically more immersive and influential infrastructure that ends up changing the physical realities of behavior and place. Yelp adds a meaningful layer to the environment that changes the sorts of places these are -- it has a real, physical effect that has increased business to independent restaurants, taking it away from the chains, because now there’s information in the environment that customers can use to feel comfortable trying a new place. http://www.hbs.edu/faculty/Publication%20Files/12-016.pdf

32

Poetry room photo http://www.flickr.com/photos/weanders/3675931616/

LABELS AND CONNECTIONS CREATE PLACES

In bit-based digital environments, we have very little physical structure to help us clarify what our language means -- we depend on the language for all of our structural affordances. The stairs to the poetry room and the poetry room itself conflate into the information inferred by the label. This was the generative problem that spawned web-oriented information architecture. How to organize space where everything is made of language, without the intrinsic structures of physical place to guide us.

WHAT PLACES ARE INSTANTIATED HERE? THE LABELS, CONNECTIONS, AND RULES ARE MUDDLED

33

So when the labels don’t make sense in relation to one another -- when the labels, connections, and rules are muddled, we’re disoriented. Sensemaking is stymied, and placemaking is dissonant. In this example from the Facebook mobile app, you can choose what mode you prefer for your News Feed. Each mode instantiates a different sort of place. But what are these places. There’s a News Feed inside of News Feed, and a Most Recent option, which one might normally assume is also “News”. Then there’s All Friends, which is really confusing -- are the other feeds not including all my friends? How are these structures nested in relation to one another? There are hidden rules under these confusing labels that the environment doesn’t make structurally clear.

THE “CLOUD”

34

I’ve become less and less enamored with the language we use for “the cloud.” A cloud is pretty from the outside, but from inside it’s just fog. Cloud is a semantic abdication of responsibility. We understand our environment and how it works because of the seams between surfaces, the structures we can perceive. If the cloud were truly magical and just omnisciently knew how to behave and work based on our specific, unique context moment to moment, then fine. But the cloud actually has seams and surfaces, but we do a terrible job of exposing those structures to users so they can understand the way those structures work. We end up posting things where we shouldn’t, erasing files on a team dropbox account, or not realizing our private photos are being saved on Apple’s iCloud, where hackers can get to them.

CHANNELS STITCHED TOGETHER WITH LANGUAGE

35

My Recipe Box - Desktop + Mobile

Aisle Wayfinding +

Mobile Labeling

Alignment

ID number - semantic marker that ties everything together

The ways we enable and influence sensemaking and placemaking aren’t just about a single website or app either -- it has to do with all the contexts that we encounter in something like a grocery store service experience. At the Wegman’s grocery chain, there’s an architecture that uses language to stitch together the relationships between a physical ID card and a digital one, and how that identifies you to the infrastructure that tracks your purchases, helps you build a recipe plan and shopping list, and find a specific kind of peanut butter in your store.

INFORMATION ENVIRONMENTS

36

Consider the whole ecosystem of information environments that make up a retail customer experience. There’s the conversations a customer has with her family, with friends and neighbors, the social media discussions and customer reviews. There’s the published reviews by experts, price comparisons across the web, and the brand messaging coming from media advertising. Not to mention all the language one encounters in a retail store, from the labels on the aisles to the conversations with service people.

37

SYSTEMS

Language is both our interface with complex systems, and our material for making them.

THE STARING GAME A SEMANTIC FRAMEWORK FOR SOCIAL BEHAVIOR

38

*blink*

HA! I WIN!

“First one to blink loses”

Think about how the staring game works. If I just walked up to anyone and started staring, I’d likely be thought rude. But all it takes is asking “hey want to play the staring game?” or “I dare you to keep staring and not blink - whoever blinks first loses” and you’ve created a sort of shared map of structure between you that changes the place you’re in, if only for a few minutes. Architecture that’s invisible, made only of an agreement. (example adapted, with thanks, from part of Frederik van Amstel’s presentation at the 2012 Interaction conference in Dublin)

BASEBALL WOULDN’T WORK...... WITHOUT THE STRUCTURES OF LANGUAGE

39

http://en.wikipedia.org/wiki/File:Baseball_diamond.svg

All games work this way. They all have structures made of words. This is a diagram about baseball, but the same principles apply to cricket… just watch a game and think about all the action that happens on the field that is guided by a collective agreement to follow a set of rules. There are actually very few physical structures constraining action on a baseball field, no rails connecting bases, no apparatus taking a batter off the plate after three strikes. People just do it, because they agreed on the rules. Invisible structures mapping a sort of temporary shared hallucination of “Place” for the sake of the game.

SWEDEN, 1967, THE MORNING AFTER THE RULE CHANGEDFROM DRIVING ON THE LEFT SIDE TO DRIVING ON THE RIGHT

40

When rules change without everyone being on board, bad things can happen. This is Sweden in 1967 the day after a law passed that changed the legal side of the road for driving. The same streets were there, but just a change by a legislative body created structures in the environment that not everyone understood yet. All these things are just made of language, and yet they create, shape, and change Places for us all the time.This is basically what happens on Facebook every time they update their privacy controls.

picture: wikimedia commons

41

So, business rules, systems, maps, environments -- all dependent upon and are inseparable from language. That means, we have to take language seriously, and understand the nature of this material and the infrastructures it influences and creates. I foresee a near future where design schools and information management schools require students to study linguistics and semiotics, to have an understanding of all the layers that make up how language works, the same way we need to understand how CSS relates to HTML, or how a data stack relates to a presentation layer. Because, ultimately, all of those things are made with language too.

42

THREE THOUGHTS...

43

MODELING IS MAKING

In a previous article I wrote on this topic, I argued that planning is making -- and I still think that *good* planning is making. But good planning involves modeling: taking tacit abstractions and complex entities and dynamics and making them into objects that we can work with in time and space. Good modeling is a kind of prototyping -- but you’re prototyping your understanding of the problem and, eventually, of the solution. It’s not valuable in and of itself; it’s valuable as a way to achieve a collective understanding and a shared vocabulary. Scaffolding for collaborative knowing. This is hard work. It is design work. And whether it’s a month of workshops or a five minute conversation over a bar napkin, it is important work.

44

BUSINESS RULES ARE EVERYONE’S RESPONSIBILITY

All of these systems are built up from what companies call business rules. And I want to say clearly that these business rules are everybody’s responsibility. Never assume that just because something is a business rule that you don’t have something important or helpful to say about it. As our practice matures, we need to be able to participate in the discussions that determine the nature and overall direction of our work. Business rules are a story the business is telling itself; it’s often up to us to dig underneath those stories and figure out what’s driving them.

45

IT’S LANGUAGE ALL THE WAY DOWN

Third - there’s really nothing that matters to us as humans that isn’t somehow wrapped up in language. The ancients who first used the word “poet” were using a word that meant “maker” -- because they understood that the poet makes more than poems. The poet makes worlds. We are a linguistic species -- grit and gristle halos and horns -- immersed in symbol, and suffused with story. It’s language all the way down.

46

THANK YOUANDREW HINTON | @INKBLURTTHE UNDERSTANDING GROUP