making representations matter: understanding practitioner experience in participatory sensemaking

Making Representations Matter Understanding Practitioner Experience in Participatory Sensemaking

Al Selvin

Knowledge Media Institute The Open UniversityMilton Keynes, UKandVerizon Telecom ITValhalla, NY USA

http://people.kmi.open.ac.uk/selvin

8 June 2011, The Open University

What is practitioner sensemaking in participatory representations?

Other kinds of participatory representational experiences

3

Everyday participatory representational experiences

4:47

Background and motivation

• Development of the Compendium software, methodology, and practice

• Ten years of practice experience in business, research, and community settings

• Desire to find research that talked about the experience of such practice

� What practitioners encounter in the heat of the moment

� How they make sense of anomalies and shape representations of value to their participants

� Characterizing the practice as articulation work

Research context

Research questions

• RQ1: How to characterize and compare the interactions of specific representational situations and practitioner actions?

• RQ2: What kinds of obstacles, breaches, discontinuities, and anomalies occur that interfere with a representation's coherence, engagement, or usefulness?

• RQ3: How do practitioner actions at sensemaking moments serve to restore coherence, engagement, and usefulness?

• RQ4: What are the specific practices involved in making the hypermedia aspects of the representation coherent, engaging, and useful?

Conceptual framework

8

Conceptual framework

9

Conceptual framework

10

Conceptual framework

11

Other forms of participatory representation also map on

12

Initial literature

review

Grid and sensemaking

moment analyses of

two expert in situ sessions

Analysis: Need ways to characterize whole session and context

Subjects: Need to contrast with non-expert, non-in situ

sessions

Data: Need skill and experience profiles of

practitioners

1 2

3

4

5

Round 1: Pilot study

Exploratory qualitative approach

14

Initial literature

review

Grid and sensemaking

moment analyses of

two expert in situ sessions

Analysis: Need ways to characterize whole session and context

Subjects: Need to contrast with non-expert, non-in situ

sessions

Data: Need skill and experience profiles of

practitioners

1 2

3

4

5

Round 1: Pilot study

Develop CEU and Shaping analysis

tools

Conduct Ames and

Rutgers sessions

Develop subject questionnaire

Second literature

reviewAnalysis: Need way to

characterize ‘experience’ dimensions

6

7

8

9

10

Round 2: Expanded study

Exploratory qualitative approach

15

Initial literature

review

Grid and sensemaking

moment analyses of

two expert in situ sessions

Analysis: Need ways to characterize whole session and context

Subjects: Need to contrast with non-expert, non-in situ

sessions

Data: Need skill and experience profiles of

practitioners

1 2

3

4

5

Round 1: Pilot study

Develop CEU and Shaping analysis

tools

Conduct Ames and

Rutgers sessions

Develop subject questionnaire

Second literature

reviewAnalysis: Need way to

characterize ‘experience’ dimensions

6

7

8

9

10

Round 2: Expanded study

Develop Framing Model

analysis tool

Apply all five

analysis tools to all

eight sessions

Conduct comparative

analysis across

sessions11

12

13

Round 3: Comparative study

Exploratory qualitative approach

Practice /Experience Dimension

Computing research

Practitioner studies

Participatory design

Facilitation & mediation

Arts-based practices

Aesthetics

Ethics

Narrative

Sensemaking

Improvisation

Related work

16

Practice /Experience Dimension

Computing research

Practitioner studies

Participatory design

Facilitation & mediation

Arts-based practices

Aesthetics

Ethics

Narrative

Sensemaking

Improvisation

Related work

17

• Visual representations in communication and group work

• Engagement with such representations

• The importance of situation and context in studying practice

• Analysis at the move-by-move level

• Limitations of research focused on tools, methods, and outcomes

Research settings – Mobile Agents

Hab Crew

Remote Science Team

18

Research settings – Ames

Ames Group 1 Ames Group 2

Ames Group 4Ames Group 3 19

Research settings – Rutgers

Rutgers Group 1

Rutgers Group 2

20

Characteristics of practitioners

21

22

12

3

4

5

67

8

9

10

11

12

0

5

10

Ames Group 11

2

3

4

5

67

8

9

10

11

12

-4

1

6

Ames Group 21

2

3

4

5

67

8

9

10

11

12

-4.0

1.0

6.0

Ames Group 3

12

3

4

5

67

8

9

10

11

12

-4

1

6

Ames Group 41

2

3

4

5

67

8

9

10

11

12

0

5

10

Rutgers Group 11

2

3

4

5

67

8

9

10

11

12

0

5

10

Rutgers Group 2

12

3

4

5

67

8

9

10

11

12

0

5

10

RST1

2

3

4

5

67

8

9

10

11

12

0

5

10

Hab Crew

Practitioner skills and experience

Larger plot = greater levels of self-reported skill and experience

1How long have you been using Compendium?

2How long have you acted as a facilitator of groups?

3How long have you acted as a facilitator of groups using any kind of software?

4How long have you acted as a facilitator of groups using Compendium?

5How many times or sessions have you acted as a facilitator?

6How many times or sessions have you acted as a facilitator of groups using any kind of software?

7How many times or sessions have you acted as a facilitator of groups using Compendium?

8How would you describe your skill level with knowledge mapping / concept mapping software?

9How would you describe your skill level with Compendium?

10How would you describe your skill level as a group facilitator?

11How would you describe your level of technical proficiency with software?

12How familiar are you with hypermedia and hypertext concepts?

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Characterizing the representational characterof the whole session

What kind of shaping took place?

Analytical tools

23

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Characterizing the representational characterof the whole session

What kind of shaping took place?

Analytical tools

24

AG4 Example

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Mapping the coherence, engagement, and usefulness dimensions of each timeslot to build up a signature for the session

Aids in identifying sensemaking episodes

Analytical tools

25

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Mapping the coherence, engagement, and usefulness dimensions of each timeslot to build up a signature for the session

Aids in identifying sensemaking episodes

Analytical tools

26

AG4 Example

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Rich description of sensemaking episode

Analytical tools

27

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Rich description of sensemaking episode

Analytical tools

28

AG4 Example

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Micro-moment moves and choices during the episode

Analytical tools

29

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Micro-moment moves and choices during the episode

Analytical tools

30

AG4 Example

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Characterizing the practitioner actions during the episode in aesthetic, ethical, and experiential terms (informed by theoretical framework)

Analytical tools

31

Shaping form

CEU analysis

Narrative description

Grid analysis

Framing analysis

Characterizing the practitioner actions during the episode in aesthetic, ethical, and experiential terms (informed by theoretical framework)

Analytical tools

32

AG4 Example

33

Session

Context

(historical,technical,

social, etc.)

TimeslotChoices/Moves

Choices/Moves

Choices/Moves

Choices/Moves

Choices/Moves

Granularity

Shaping & FramingAnalysis

Sensemaking Moment & Grid Analysis

CEUAnalysis

Analytical tools

Timeslot

Timeslot

Session

TimeslotChoices/Moves

Choices/Moves

Choices/Moves

Choices/Moves

Choices/Moves

Timeslot

Timeslot

Session

TimeslotChoices/Moves

Choices/Moves

Choices/Moves

Choices/Moves

Choices/Moves

Timeslot

Timeslot

34

CEU heat maps showing sensemaking episodes

35

Sensemaking triggers and responses (AG4 example)

Trigger Response

Response type: Holding forward progress until new strategy is in place

Ethical dimension: Direct intervention for purpose of practitioner action

Aesthetic dimension: Creating space for remedial shaping to take place

Pertaining to volume or type of participant input (“Too much too fast”)

Facilitator: “But we had a question that says ‘how can the public become co-creators?’”Mapper creates Question node and facilitator narrates answers from previous discussion

Collaborative navigation to find item of interest

Negotiation/agreement on placement of an item

Direct collaboration between practitioner and participants

36Hab

Triggers (RQ2)

Pertaining to representational structure

Pertaining to volume or type of participant input

Pertaining to information/subject matter

Pertaining to intended process/plan

37

Sensemaking dimensions

Ethical Dimensions

Direct collaboration between practitioners and participants

Direct intervention aimed at participants

Direct intervention for purpose of practitioner action

Indirect intervention

Changing/blurring roles

Non-intervention

Aesthetic Dimensions

Direct contribution to shaping

Intended to help participant shaping

Creating space for remedial shaping to take place

Partially having to do with shaping

No aesthetic dimension

Practitioner responses (RQ1, 3)

Triggers (RQ2)

Pertaining to representational structure

Pertaining to volume or type of participant input

Pertaining to information/subject matter

Pertaining to intended process/plan

38

Sensemaking dimensions

Ethical Dimensions

Direct collaboration between practitioners and participants

Direct intervention aimed at participants

Direct intervention for purpose of practitioner action

Indirect intervention

Changing/blurring roles

Non-intervention

Aesthetic Dimensions

Direct contribution to shaping

Intended to help participant shaping

Creating space for remedial shaping to take place

Partially having to do with shaping

No aesthetic dimension

Practitioner responses (RQ1, 3)

AG4

• Overall context, tone, or characterof a session

• Initial plan and other pre-session factors• Dimensions

• Advance• As-played-out

• Interpersonal interactions and communicative styles

• Dimensions• Regulating• Bringing to the representation• Collaboration (style, force, purpose)

• Physical and conceptual shaping of the representations

• Relating the sessions to the normative model

Shaping and Framing categories

Category A – Conducting (RQ1)

Category B – Planning (RQ1, 3)

Category C – Relating (RQ1, 2, 3)

Category D – Shaping (RQ1, 4)

Category E – Framing (RQ1, 3)

How “good”/successful was the session?Multiplicity/heterogeneity of focus aspectsDegree of expressed participant resistance, disagreementDegree of ‘noise,’ chaos, boisterousness Degree of “meta” discussionSpectrum from “discussion-centric” to “map-centric”

Choice of methodHow much of shaping/process is "emergent" vs. pre-determinedGranularity of the pre-created structure (degree and complexity)Ambitiousness of the planned approachDegree of practitioner adherence to method during the sessionParticipant adherence/faithfulness to the intended plan

Density of practitioner verbal moves (frequent vs. infrequent)Practitioner willingness to interveneHigh practitioner “drive” of the session vs. high participant “drive” Practitioner-asked clarifying questions to participant inputPractitioner-requested validation of changesPractitioner “gating” of participant input Intervention to get participants to look at representationCollaboration between multiple practitioners Collaboration between practitioners and participants

Attention to textual refinement of shapingAttention to visual/spatial refinement of shapingAttention to hypertextual refinement of shapingDegree of ‘finishedness’ of the artifactsDensity of practitioner shaping movesComplexity of the software techniques in useDegree of de-linked interaction with representation

Narrative consistency and usefulness Inclusiveness of the narrative framingEvocativeness of the narrative framing Clarity of artifacts Openness and dialogicity of the mediated objects Resistance from participants and materials Addressing and incorporating participant impulses and desires

Shaping and Framing categories

Category A – Conducting (RQ1)

Category B – Planning (RQ1, 3)

Category C – Relating (RQ1, 2, 3)

Category D – Shaping (RQ1, 4)

Category E – Framing (RQ1, 3)

Comparative method

Using Compendium to rank order the sessions along eachqualitative dimension and capture rationale

(http://people.kmi.open.ac.uk/selvin/analysis)

Comparative method

Using Compendium to rank order the sessions along eachqualitative dimension and capture rationale

(http://people.kmi.open.ac.uk/selvin/analysis)

AG4

Shaping rankings and ratings

RG2

RST

Granularity of pre-created structureHigh Low

RST Hab AG2 RG1 AG4 AG3 AG1 RG2

44

Intervention to get participants to look at the representation

RG2

45

Intervention to get participants to look at the representation

AG4

High LowRST

High

AG3

High

Hab

Med High

RG2

Med

RG1

Med

AG1

Med Low

AG2

Low

AG4

Low

AG3

AG4

Visual/spatial refinement

47AG4

High LowRST

High

AG3

High

RG2

Med High

Hab

Med

AG1

Med Low

RG1

Low

AG4

Low

AG2

Low

RG2

Hab

Hypertextual refinement

48

49

1 23

45

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 11 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 21 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 3

1 23

45

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 41 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Rutgers Group 11 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Rutgers Group 2

1 23

45

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

RST1 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Hab Crew

It is now possible to compare sessions along the experiential dimensions

Larger plot = generally higher rankings

Shaping and framing dimensions

50

12

3

4

5

67

8

9

10

11

12

0

5

10

Ames Group 11

2

3

4

5

67

8

9

10

11

12

-4

1

6

Ames Group 21

2

3

4

5

67

8

9

10

11

12

-4.0

1.0

6.0

Ames Group 3

12

3

4

5

67

8

9

10

11

12

-4

1

6

Ames Group 41

2

3

4

5

67

8

9

10

11

12

0

5

10

Rutgers Group 11

2

3

4

5

67

8

9

10

11

12

0

5

10

Rutgers Group 2

12

3

4

5

67

8

9

10

11

12

0

5

10

RST1

2

3

4

5

67

8

9

10

11

12

0

5

10

Hab Crew

Practitioner skills and experience

51

1 23

45

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 11 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 21 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 3

1 23

45

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Ames Group 41 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Rutgers Group 11 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Rutgers Group 2

1 23

45

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

RST1 2

34

5

6

7

8

9

1011

12131415

1617

18

19

20

21

22

2324

2526

0

5

10

Hab Crew

It is now possible to compare sessions along the experiential dimensions

Larger plot = generally higher rankings

Shaping and framing dimensions

52

Facilitation skills were a stronger predictor than technical skills

SessionShapingIndex

Session goodness

rank

Software proficiency

rank

Facilitation proficiency

rank

Hab 83 1 1 1

RST 78 2 1 2

RG2 70 5 4 3

AG4 66 3 6 5

AG3 55 7 2 6

RG1 54 4 7 5

AG1 41 6 5 4

AG2 18 8 3 7

Shaping/framing dimension

A.1 How “good”/successful was the session?

B.5Practitioner adherence to the intended method during the session

B.6Participant adherence/faithfulness to the intended plan

C.2 Practitioner willingness to intervene

C.4 Practitioner-asked clarifying questions

C.5Practitioner-requested validation of changes to representation

C.7Intervention to get participants to look at the representation

C.8Collaboration between multiple practitioners (if applicable)

C.9Collaboration/co-construction between practitioners and participants

D.1 Attention to textual refinement of shaping

D.2 Attention to visual/spatial refinement of shaping

D.3 Attention to hypertextual refinement of shaping

D.4 Degree of ‘finishedness’ of the artifacts

Generalization to other genres of participatory representations

Graphic facilitation

Informal whiteboarding

54

Contributions

• Offers analytical tools

Contributions

• Offers analytical tools

• Provides a language to characterize and compare instances of representational practice

Contributions

• Offers analytical tools

• Provides a language to characterize and compare instances of representational practice

• Describes the types of sensemaking moments that practitioners encounter

57

Contributions

• Offers analytical tools

• Provides a language to characterize and compare instances of representational practice

• Describes the types of sensemaking moments that practitioners encounter

• Highlights the specific role of a hypermedia technology

58

Contributions

• Offers analytical tools

• Provides a language to characterize and compare instances of representational practice

• Describes the types of sensemaking moments that practitioners encounter

• Highlights the specific role of a hypermedia technology

• Contributes to reflective methods for practitioner and practice development

59

Future work

• Studies� Performing longitudinal studies and action research with

emphasis on artifactual sensemaking� Comparing other practices

• Tools and methods� Developing the analytical tools� Developing training, assessment, and reflective practice

methodologies

• Theory� Exploring the “recursive” nature of the experiential

dimensions as they relate to representational practices (as something you apply vs. something you live within)

60

For more

• Analysis artifacts http://people.kmi.open.ac.uk/selvin/analysis/

• Research blog http://knowledgeart.blogspot.com

• Recent journal articlehttp://oro.open.ac.uk/20948/1/Selvin-HumanTechnology2010.pdf