carolyn penstein rosé language technologies institute human-computer interaction institute school...

1

Carolyn Penstein RoséLanguage Technologies Institute

Human-Computer Interaction Institute

School of Computer Science

With funding from the National Science Foundation and the Office of Naval Research

Souflé: A Three Dimensional Framework for Analysis of Social Positioning

in Dyadic and Group Discussions

2

Developing technology capable of shaping conversation and supporting effective participation in conversation to achieve positive impact on…

Human learning

Health

Wellbeing

3

Developing technology capable of shaping conversation and supporting effective participation in conversation to achieve positive impact on…

Human learning

Health

Wellbeing

Human learning

4

• Introducing the Problem of Supporting Productive Discussion for Learning

• Discussion of Souflé• Transactivity• Engagement• Authoritativeness

• Application to Dynamic Support for Group Learning

• Conclusion and Current Directions

Outline

5

• Introducing the Problem of Supporting Productive Discussion for Learning

• Discussion of Souflé• Transactivity• Engagement• Authoritativeness

• Application to Dynamic Support for Group Learning

• Conclusion and Current Directions

Outline

Purpose of Interaction in Learning Contexts

Reward structures encourage students to focus on performance over learning

Well crafted instruction provides opportunities for learning

Opportunities only help if students take them

Take Home Message:

Introducing reflection points provides opportunities for students to take advantage of learning resources

CarefullyStructuredConceptualKnowledge

Important Ingredients for Learning

7

Reflectionthrough

RichInteraction

End of Fall Semester: Students learn about Rankine Cycles 1 Week of lectures Homework assignment on analysis of

Rankine Cycles Tutorial on using CyclePad software

package (Developed at Northwestern University (Forbes et. al. 1999) Allows students to construct and

analyze a variety of Thermodynamic Cycles)

Instructed on Effects of Changing System Variables (Temperature, Pressure) on System Output (Power, Waste Heat)

Second-Year Thermodynamics

Learning Goal: Encourage students to reflect on interactions between cycle parameters

Collaborative Task

Reduction in Steam Quality

Power

Waste Heat

Increasing heat increases power but also waste heat

Increasing pressure increases efficiency

Design Goal: Design a power plant based on the Rankine Cycle paradigm

Competing Student Goals: Power: Design a power plant that achieves

maximum power output Motivated by economic concerns

Green: Design a power plant that has the minimum impact on the environment Motivated by environmental concerns

Each pair turns in exactly one design

Computer Supported Collaborative Learning

11

12

• Introducing the Problem of Supporting Productive Discussion for Learning

• Discussion of Souflé• Transactivity• Engagement• Authoritativeness

• Application to Dynamic Support for Group Learning

• Conclusion and Current Directions

Outline

Souflé Framework(Howley et al., in press)

13

Person Person

3 Dimensions: Transactivity Engagement Authoritativeness

Souflé Framework(Howley et al., in press)

14

Sociolinguistics

Discourse Analysis

LanguageAnd

Identity

LanguageUse

MachineLearning

Multi-Level

Modeling

AppliedStatistics

ComputationalModels

OfDiscourseAnalysis

Souflé Framework(Howley et al., in press)

15

Transactive Knowledge Integration

Person Person

16

i

• Definition of Transactivity• building on an idea expressed earlier in a conversation • using a reasoning statement

We don't want tmax to be at 570 both for the material and [the Environment]

well, for power and efficiency, we want a high tmax, but environmentally, we want a lower one.

17

18

19

Findings

Moderating effect on learning (Joshi & Rosé, 2007; Russell, 2005; Kruger & Tomasello, 1986; Teasley, 1995)

Moderating effect on knowledge sharing in working groups (Gweon et al., 2011)

Computational Work

Can be automatically detected in:

Threaded group discussions (Kappa .69) (Rosé et al., 2008)

Transcribed classroom discussions (Kappa .69) (Ai et al., 2010)

Speech from dyadic discussions (R = .37) (Gweon et al., 2012)

Predictable from a measure of speech style accommodation computed by an unsupervised Dynamic Bayesian Network (Jain et al., 2012)

Transactivity (Berkowitz & Gibbs, 1983)

Engagement Engagement

Souflé Framework(Howley et al., in press)

21

Transactive Knowledge Integration

Person Person

Engagement(Martin & White, 2005, p117)

System of Engagement Showing openness to the

existence of other perspectives

Less final / Invites more discussion

Example: [M] Nuclear is a good choice [HE] I consider nuclear to be

a good choice [HC] There’s no denying that

nuclear is a superior choice [NA] Is nuclear a good

choice? 22

23

Findings

Correlational analysis: Strong correlation between displayed openness of group members and articulation of reasoning (R = .72) (Dyke et al., in press)

Intervention study: Causal effect on propensity to articulate ideas in group chats (effect size .6 standard deviations) (Kumar et al., 2011)

Mediating effect of idea contribution on learning in scientific inquiry (Wang et al., 2011)

Engagement (Martin & White, 2005)

Authority

Authority

Engagement Engagement

Souflé Framework(Howley et al., in press)

25

Transactive Knowledge Integration

Person Person

Analysis of Authoritativess

26

Water pipe analogy: Water = Knowledge or Action Source = Authoritative speaker Sink = Non-authoritative Speaker

Authoritativeness Ratio = Source Actions Actions

27

The Negotiation Framework(Martin & Rose, 2003)

Source

or Sink?

Primary

Se

condary

Type of Content?

Knowledge Action

K2requesting knowledge,

information, opinions, or facts

K1giving knowledge, information,

opinions, or facts

A2Instructing, suggesting, or

requesting non-verbal action

A1Narrating or performing your

own non-verbal action

Additionally…ch (direct challenge to previous utterance)

o (all other moves, backchannels, etc.)

28

The Negotiation Framework(Martin & Rose, 2003)

Source

or Sink?

Primary

Se

condary

Type of Content?

Knowledge Action

K2requesting knowledge,

information, opinions, or facts

K1giving knowledge, information,

opinions, or facts

A2Instructing, suggesting, or

requesting non-verbal action

A1Narrating or performing your

own non-verbal action

29

Source

or Sink?

Primary

Se

condary

Type of Content?

Knowledge Action

K2requesting knowledge,

information, opinions, or facts

K1giving knowledge, information,

opinions, or facts

A2Instructing, suggesting, or

requesting non-verbal action

A1Narrating or performing your

own non-verbal action

The Negotiation Framework(Martin & Rose, 2003)

Source

or Sink?

Primary

Se

condary

Type of Content?

Knowledge Action

K2requesting knowledge,

information, opinions, or facts

K1giving knowledge, information,

opinions, or facts

A2Instructing, suggesting, or

requesting non-verbal action

A1Narrating or performing your

own non-verbal action

The Negotiation Framework(Martin & Rose, 2003)

31

The Negotiation Framework(Martin & Rose, 2003)

Source

or Sink?

Primary

Se

condary

Type of Content?

Knowledge Action

K2requesting knowledge,

information, opinions, or facts

K1giving knowledge, information,

opinions, or facts

A2Instructing, suggesting, or

requesting non-verbal action

A1Narrating or performing your

own non-verbal action

Additionally…ch (direct challenge to previous utterance)

o (all other moves, backchannels, etc.)

32

The Negotiation Framework(Martin & Rose, 2003)

Source

or Sink?

Primary

Se

condary

Type of Content?

Knowledge Action

K2requesting knowledge,

information, opinions, or facts

K1giving knowledge, information,

opinions, or facts

A2Instructing, suggesting, or

requesting non-verbal action

A1Narrating or performing your

own non-verbal action

Additionally…ch (direct challenge to previous utterance)

o (all other moves, backchannels, etc.)

K1 + A2

K1 + K2 + A1 + A2

Authoritativeness:

33

K2?

34

Set up!

K1

K2

35

Findings Authoritativeness measures display how students respond to

aggressive behavior in groups (Howley et al., in press) Authoritativeness predicts learning (R = .64) and self-efficacy (R

= .35) (Howley et al., 2011) Authoritativeness predicts trust in doctor-patient interactions (R

values between .25 and .35) (Mayfield et al., under review)Computational Work

Detectable in collaborative learning chat logs (R = .86)

Detectable in transcribed dyadic discussions in a knowledge sharing task (R = .95) (Mayfield & Rosé, 2011)

Detectable in transcribed doctor-patient interactions (R = .96) (Mayfield et al., under review)

Authoritativeness (Martin & Rose, 2003)

37

• Introducing the Problem of Supporting Productive Discussion for Learning

• Discussion of Souflé• Transactivity• Engagement• Authoritativeness

• Application to Dynamic Support for Group Learning

• Conclusion and Current Directions

Outline

Computer Supported Collaborative Learning

39

AutomaticAnalysis

Of Conversation

ConversationalInterventions

PositiveLearning

Outcomes

6 Years of Positive Results

Foundational study: students work with a partner and dialogue agent for support Learn 1.24 s.d. more than individuals without support (Kumar et al., 2007a)

Results inform iterative design of agent behavior Personalized agents increase supportiveness and help exchange between students (Kumar et al.,

2007b) Agents are more effective when students have control over timing of the interaction (Chaudhuri

et al., 2008; Chaudhuri et al., 2009) Agents that employ Balesian social strategies are more effective than those that do not (Kumar

et al., 2010; Ai et al., 2010) Students are sensitive to agent rhetorical strategies such as displayed bias (Ai et al., 2010),

displayed openness to alternative perspectives (Kumar et al., 2011), and targeted elicitation (Howley et al., 2012)

Bazaar architecture enables efficient, principle based agent development (Kumar & Rosé, 2011; Adamson & Rosé, 2012)

41

• Introducing the Problem of Supporting Productive Discussion for Learning

• Discussion of Souflé• Transactivity• Engagement• Authoritativeness

• Application to Dynamic Support for Group Learning

• Conclusion and Current Directions

Outline

42

• Transactivity is a conversational behavior that is important for learning

• Authoritativeness and Engagement are dimensions of conversation that play a supporting role

• Positioning students to exchange Transactive contributions

• We have made progress at automating analysis of Transactivity and Authoritativeness

•Automated analysis enables dynamic triggering of supportive interventions for online group learning

Conclusions and Current Directions

43

•In the future, CSCL activities will be part of societies of online learners

• Vision began with Virtual Math Teams/ The Math Forum• Now we’re already seeing the shift through companies

like Coursera and Udacity

•We are taking steps towards this future• Fully distance learning studies (UCSB, Drexel)• Sustainable CSCL (online office hours agent)

•As we look to the future: we must understand the emergent effects of our local interventions in order to maximize positive benefit on a grand scale

Conclusions and Current Directions

44

Thank You!

Questions?

Revoicing Agent

45

46

Experimental Paradigm Day 1: Pretest on conceptual questions related to the unit (Diffusion or

Punnett Squares) Day 2: Online collaborative activity + Immediate Posttest isomorphic to

pretest Day 3: Whole class teacher led discussion + Delayed Posttest isomorphic to

pretestParticipants: consenting 9th grade biology students, randomly

assigned to groups of 3Experimental Design: Simple between subjects design

Groups randomly assigned to Revoicing condition or Control Condition

Revoicing Agent Studies

Revoicing Agent

47

48

Study 1: Year 1, Diffusion Lab (50 students) Students learn more on explanation questions in supported conditions

(F(1,46) = 4.3, p < .05, effect size 1 standard deviation) Students in supported conditions more active in whole group discussion

(F(2,26) = 4.2, p < .05, effect size .75 standard deviations)

Study 2: Year 2, Diffusion Lab (78 students) Students learn more on immediate post test in Revoicing Agent

condition (F(1,74) = 4.3, p < .05, effect size .51 standard deviations)

Study 3: Year 2, Punnett Square Lab (78 students) Students learned more on delayed post-test in Revoicing Agent

condition

Results of CSCL Studies