Multilevel Analysis of Socially Embedded Learning

Dan Suthers University of Hawaii

Supported by the National Science Foundation

“Do you go to the beach all the time?”

No. We do not always go to the beach.

We also go to the mountains.

“Health hazards on Mauna Kea:

Altitude sickness. At the summit

elevation of 13796 feet (4200 m), the

atmospheric pressure is 40

percent less than at sea level …”

Major Motivations and Ideas

!  Learning (particularly in socio-technical settings) is a complex and embedded phenomenon

!  Multiple theories and levels of analysis are needed

!  Distributed and multimediated nature of socio-technical systems present analytic challenges

!  Approaches illustrated with my work: !  Generalized concept of interaction and the contingency of acts on their setting

!  Abstract transcript and analytic hierarchy

Let’s start with Learning in Socio-Technical Networks …

… and the idea that Learning is “Embedded” in multiple ways.

Learning in Socio-Technical Networks

Agency Who or what is the agent

that learns? !  Individual !  Small groups !  Networks (communities,

cultures, societies)

Epistemologies What is the process of

learning? !  Acquisition !  Intersubjective meaning-

making !  Changes in participation

and Identity The correspondence is not strict. Epistemologies can be applied at local or network levels

How do social settings foster learning?

Based on ! Suthers (ijCSCL 2006)

Levels of Agency and Epistemologies !  Acquisition Epistemologies

Learning as acquisition of information, knowledge or skills !  Local: contribution theory, given/new contract, explanation,

conceptual change, practice of skills, etc. !  Network: weak ties, diffusion theories (contagion theory,

diffusion of innovations) !  Intersubjective epistemologies

Learning as intersubjective meaning-making !  Local: co-construction, collaborative inquiry, group cognition !  Network: knowledge building, communities of scientists

!  Participatory epistemologies Learning as changes in social participation and identity !  Local: identity, apprenticeship & mentoring (LPP) !  Network: expansive learning (CHAT)

A Complex Multilevel Phenomenon

Claim: individuals participate in the foregoing forms of learning simultaneously

!  One might choose to focus on one form, or !  Grapple with a fundamental question:

How does learning take place through the interplay between individual and collective agency in socio-technical networks?

!  Requires coordinated multi-level analysis !  Requires coordinated multi-level theorizing

Learning is Embedded

!  Interactionally embedded !  Learning accomplishments are contingent on their interactional setting

!  Socially embedded !  Social as source of resources !  Social entity as agent of learning

!  Technologically embedded !  Affordances influence processes !  Artifacts sustain practices and activity structures

Analytic Challenges

!  Embedded: need to say how activity is contingent on setting

!  Multimediated: need media independent unit of interaction, while being media aware

!  Distributed: need to unify diverse data streams

!  Hierarchical: need multiple levels of theory and analysis

Traces Analytic Hierarchy Addressing (some of) the needs

Activity is distributed across multiple media

"  Abstract transcript representation collects distributed events from multiple media into a single analytic artifact, reassembling fragmented record of activity

Local activity is hierarchically embedded in network settings, calling for coordinated multilevel analysis

"  Analytic hierarchy that supports multiple levels of description (interaction, mediated associations, ties) and analysis

! Suthers (HICSS 2011) ! Suthers & Rosen (LAK 2011)

Contingencies

Mediated Associations

Uptake Ties

Interaction Affiliations

Uptake and Contingencies

Thanks to NSF, Chris Hundhausen, Laura Girardeau, Nathan Dwyer, Richard

Medina and Ravi Vatrapu

How these ideas developed, picking

up where we left off in about 2003 ….

Example 1: First Encounter of Needs

Distributed Interaction

Motivated concept of “Uptake”

!  Needed a cross-media unit out of which to construct analytic accounts of interaction !  Media specific concepts (“adjacency pair”, “edit”, “reply”) are too specific

!  Not a new unit, but rather a name given to all such constructs taken collectively

!  Generalize beyond interaction as “reciprocal action or influence” to other forms of association

Uptake

Minimal requirement for two acts to form part of an interaction: that the existence of the first act is consequential in some way for the second act:

Uptake is present when an act takes some aspect of a prior act (or event) as relevant for ongoing activity.

Flexible and Broad: Opens up our thinking about how interaction might be accomplished

Example 2: Asynchronous Dyads

!  Asynchronously interacting dyads !  Public heath problem with hidden profile materials

!  Original study: representational guidance of evidence maps vs. threaded discussion

Example: Asynchronous Dyads

Closeup

Example 2: Interactional Pattern (“W”)

!  Information Sharing / Round Trip in Evidence Map !  Subsequent Negotiation in Threaded Discussion

Connecting Uptake to Evidence

Motivations !  “How do you know it’s really uptake?”

!  Problem of intentionality but also !  Separate evidence from claim

!  Manual analysis is slow !  Sufficiently “objective” evidence would also

be computable !  Action is contingent on its setting in

many (observable) ways: let’s use computational tools to leverage this!

Contingencies

Any observed relationship between events that may evidence how one event may have enabled or influenced other events (acts)

!  Include “many metaphysical shades between full causality and sheer inexistence” (Latour, 2005)

!  Contingencies record how each act is embedded in a history of interaction and a social and technological setting

Some Types of Contingencies

Media Dependency ei operates on object created or modified by ej

Temporal Proximity ei took place soon after ej

Spatial Organization ei takes place in configurational context created by ej

Inscriptional Similarity ei creates inscriptions similar to those created by ej

Semantic Relatedness The meaning of inscriptions created by ei and ej overlap

Contingencies of ei on ej

(! Suthers Dwyer, Medina & Vatrapu, ijCSCL 2010)

Example 3: Early Contingency Analysis

!  Analysis originally undertaken to explain convergence & divergence, but discovered emergence of representational practices

!  First automated construction and visualization of contingency graph

(# Medina & Suthers, RPTEL 2009)

Example 3: Asymmetry in Roles

Example 3: Representational Practices

Example 3: Episodic View of Interaction

Abstraction to uptake between episodes of specific acts

Example 3: Multi-level Analysis Lemke: "look at at least one organizational level below

the level we are most interested in (to understand the affordances of its constituents) and also one level above (to understand the enabling environmental stabilities)"

Traces Analytic Approach

Testbed: Tapped In SRI’s Network of education professionals: PD and peer

support (Mark Schlager, Patti Schank, Judi Fusco) 1997-2013: longest running educational online

community !  20K educators/year !  800 user spaces !  50 tenants !  40-60 volunteer-run

community-wide activities/month

!  Chats, discussions, wikis, resource sharing ... Good Testbed: Heterogeneous network of diverse small

groups interacting with multiple media

Automatic Discovery of Distributed “Sessions” and Influences Between Sessions

Overview of Analysis: Process Trace

Overview of Analysis: Events

Overview of Analysis: Contingencies

Overview of Analysis: Uptake

Overview of Analysis: Sessions

Process Model

XML Scripts driving Java, NLTK, iGraph <!-- ========== Content Preprocessing ========== -->!

<step bundlename="apps.analyzer" classname="apps.analyzer.script.PythonScriptStep" >!!<stepconfig scriptref="nltk/lancaster_stemmer.py" />!

</step>!

<!-- ========== Contingencies ========== --> !<step bundlename="apps.analyzer"

classname="apps.analyzer.script.ReadDiscussionMessageRule" />!

<step bundlename="apps.analyzer" classname="apps.analyzer.script.LexicalRule" />!<step bundlename="apps.analyzer" classname="apps.analyzer.script.ReplyRule" />!<step bundlename="apps.analyzer" classname="apps.analyzer.script.AddressRule" />!

<step bundlename="apps.analyzer" classname="apps.analyzer.script.SameActorRule">!!<stepconfig windowsize="300" tag="SA300" />!

</step>!<step bundlename="apps.analyzer" classname="apps.analyzer.script.TimeWindowRule">!

!<stepconfig windowsize="120" tag="TW120s" />!</step>!

<!-- ========== Activity Structure (Finding Sessions)========== -->!<step bundlename="apps.analyzer" classname="apps.analyzer.script.ActivityRule"> ! !<stepconfig graphName="activity">! ! !<weighter fileref="weights/activity_weights.xml" />! !</stepconfig>!</step>!. . . !

Case Study

Analyze 3 days of chat, centered on a session of

interest

Teaching Teachers Session 184 23:35 Mary: are all good teachers good mentors? 185 23:38 Amber: some people will take a while to get to that point 186 23:42 Amber: No..not all 187 23:51 Erica: definitely not 188 23:55 Lara: Training can help, but I think some is personality 189 24:09 Amy: some people are excellent teachers but are horrible mentors 190 24:09 Erica: some great teachers can not hold a decent conversation with an

adult 191 24:11 Amber: i had to co-ops who would be awful mentors 192 24:24 Lara: Nods 193 24:27 Dianne: That is an interesting question Maria, ... I would probably say

'yes' first off, and then wonder some more 194 24:42 Mary: it is something I have thought about often Lisa 195 24:47 Amber: I think its alot of personality 196 25:17 Dianne: one thing a mentor has to know is how to operate with a peer,

and how to be intentional about handing over, or encouraging greater independence

197 25:18 Mary: observation has made me think that it takes an extra “special ingredient” to tip the scales

198 25:34 Erica: I think if you have the passion for teaching you will want everyone else to feel the same

199 25:35 Amber: agree

Contingencies computed

!  Time Window (recency): all chats within 120 seconds

!  Last Contribution: last chats by same actor in 300 seconds

!  Address: Actor chats ... chat addresses actor !  Reply: Chat addresses actor ... actor chats !  Lexical Overlap: weighted count of overlapping lexical items (NLTK Lancaster Stemmer)

Weighted sum of counts of above $ estimate of uptake

Uptake Graph for 3 Days of Chat

Rooms

One session across

two rooms

Two sessions in one room

Sessions (Modularity Partitions)

One session across

two rooms

Two sessions in one room

Session Partitions Collapsed

Inspect group in Data Laboratory

Teaching Teachers Session Begins

Teaching Teachers Session Ends

Sociogram

Folding contributions by actor to expose actor-actor uptake

Session 74, Rooms

Session 74, Contributions Colored by Actor, ForceAtlas2 Layout

Can we characterize “good” sessions by structural patterns?

Nodes are contributions, Colors are actors, Node size is weighted indegree

Session 74 Sociogram

Nodes are actors Node size is weighted indegree

Selecting second group

Same actors in NTraining Session

How “Communities” are Embedded in Technological

Media Mediated Associations and

Community Detection

! Suthers, Fusco, Schank, Chu & Schlager (HICSS 2013)

Contingencies

Mediated Associations

Uptake Ties

Interaction Affiliations

Characterization of Community Structure

!  “I don’t know what communities are there” !  Organizational “tenants” and unsponsored !  Multiple, fluid forms of participation

!  An empirical matter !  Don’t assume that the network is one community

!  Don’t assume that external communities are replicated within the sociotechnical system

Communities: Technologically Embedded !  Multiple technologies for participation, each with their

own interactional and social affordances !  Choice of technologies reflect and reaffirm the

relationship between interlocutors (Licoppe and Smoreda, 2005)

!  Apply this idea to collective rather than dyadic level: Communities are embedded within and make use of technological media for interaction in ways that reflect and reaffirm their nature

!  Our approach identifies cohesive subgroups of actors and of actants (mediational means) simultaneously

! Suthers & Chu, LAK 2012

Intermediate level of representation

!  Actor-Actor ties: useful abstraction, but hide

how enacted

!  Intermediate granularity: mediated association

!  Interaction traces (e.g., contingency graphs): overwhelming detail!

Portion of an Associogram

actors

discussions

files

Cohesive subclusters in associogram

Modularity Partitioning •  234 Partitions •  Modularity: 0.828 Open Ord Layout in Gephi

Cohesive subgraphs of actors and artifacts via which they interact

Interpretations of Top 6 Partitions

After School Online Events

Associations via TI

Reception and other public

rooms

Chat-based CoP in a

Midwestern school district;

Discussion-based

professional development in the Southern

US

Chat-based Language

Arts in the US Midwest;

Pre-service program in Western US

Myriad of Small Clusters

Size distribution of Largest 86 partitions

Average weighted degree by actor size

(sample of every 10th partition)

Artifact/Actor ratios by actor size

Use of media in large and small partitions

!  Tenant and unsponsored are similar in large partitions

!  In small partitions, tenants are strongly chat based while unsponsored rely on asynchronous media

Summary & Comments !  Purely structural (graph theoretic) computations

identified cohesive subgroups that have interpretations as communities

!  Diversity demonstrates vibrancy of Tapped In as “transcendent community” (# Joseph et al., CSCL 2007)

!  Value to learning analytics: identify social units that are the setting or agent of learning

!  Can “dive in” to examine activity of high-degree actors, structure of chat sessions in rooms, etc.

!  Need algorithm for overlapping cohesive clusters !  Clique percolation fails on bipartite graphs !  Edge communities and flow compression promising

! Suthers, Fusco, Schank, Chu & Schlager (HICSS 2013)

Productive Multivocality

Bringing multiple theoretical and methodological traditions

to bear

Productive Multivocality Project

The complexity of learning requires multiple analytic “voices” (theories and methods): How to bring them into productive dialogue?

!  5 year project sharing/comparing approaches to analyzing collaborative learning

!  37+ researchers analyzed 5 corpora

! Suthers, Lund, Rosé, Teplovs & Law (Springer 2013)

Strategies for Productive Multivocality

!  Dialogue about the same data, from different perspectives

!  Share an analytic objective (e.g., “pivotal moments”)

!  Bring analytic representations into alignment with each other and the original data

!  Eliminate inconsequential differences and Iterate

!  Push the boundaries of traditions without betraying

!  Reflect on Practice: dialogue about methods as object-constituting, evidence-producing and argument-generating tools

Brief Comments on Design

Mediating between individual and group

Individual !" Small group

Representational affordances for intersubjective meaning-making:

!  (Im)Mutable Mobiles !  Negotiation Potentials !  Referential Resources !  Reflector of subjectivity (awareness) !  Persistence (reflection)

! Suthers & Hundhausen (JLS 2003) ! Suthers (ijCSCL 2006)

Individual !" Network

! Joseph, Lid & Suthers (CSCL 2007)

act

persist

find

care

care

act

persist

find

care

act

persist

find persist

find

care

act Thanks to Viil Lid

for diagrams

Key Ideas

!  Learning is interactionally embedded % Contingency and Uptake analysis of sequential structure

!  Learning is socially embedded % Empirically identify the social units in a STN

!  Learning is technologically embedded % Identify the mediational means (mediated associations)

!  Generalized concepts, abstract transcript, and analytic hierarchy help

Summary of Concepts !  Mediation and Associations

!  All interaction is mediated; actors are associated via media !  Understand how social phenomena are technologically

embedded (! Licoppe & Smoreda, SN 2005; ! Suthers & Chu, LAK 2012)

!  Uptake: (! Suthers, ijCSCL 2006) !  Taking some aspect of (the trace of) a prior act or event as

relevant for ongoing activity !  A generalized unit of analysis for “interaction” broadly

understood (multi/cross-media; inter/intra-subjective) !  Contingencies: (! Suthers Dwyer, Medina & Vatrapu, ijCSCL 2010)

!  Manifest relationships between acts and their setting (including other events)

!  Evidence for Uptake

Traces Analytic Hierarchy

" Abstract transcript representation that collects relevant events into a single analytic artifact

" Analytic hierarchy that supports multiple levels of analysis

! Suthers, HICSS 2011 ! Suthers & Rosen, LAK 2011

Contingencies

Mediated Associations

Uptake Ties

Interaction Affiliations

Mahalo!

Dan Suthers, Dept. of ICS suthers@hawaii.edu Lilt.ics.hawaii.edu