Temporal patterns of knowledge construction:

Statistical discourse analysis of a role-based online discussion

To appear in the International Journal of Computer-Supported Collaborative Learning

Alyssa WiseSimon Fraser University

alyssa_wise@sfu.cas

Ming Ming ChiuState University of New York –Buffalo

mingchiu@buffalo.edu

I appreciate the research assistance of Choi Yik Ting

Online, asynchronous forums• Can participate anywhere – no geographic limits• Can share ideas at any time – more time to think• But often disconnected, only lists of isolated ideas

Guzdial & Turns, 2000; Herring, 1999; Thomas, 2002Summaries• Connect previous ideas and develop them• But often occur at end of discussion

& Do not benefit other members De Wever et al., 2007; Schellens et al. 2005; 2007

Encourage summaries in the middle of discussions?

Motivation for the Study

Knowledge Construction (KC) FrameworkGunawardena et al.’s (1997) Five-Phase Model

Research Context for the Study Emerging Themes in Collaborative Learning Research

(e.g. Chiu & Khoo, 2005; Kapur, 2001; Reimann, 2009)

(e.g. Cress, 2008; Suthers & Teplovs, 2011)

(e.g. Arvaja, 2007; Stahl, 2004; Strijbos et al., 2004)

Possible KC Patterns

0 5 10 15 201

2

3

4

51a

0 5 10 15 201

2

3

4

51b

0 5 10 15 201

2

3

4

52a

0 5 10 15 201

2

3

4

52b

0 5 10 15 201

2

3

4

53

0 5 10 15 201

2

3

4

54

Knowledge Construction Phase

Post Number

Research Questions• What patterns characterize knowledge

construction processes during an online discussion?

• What characterizes pivotal posts that divide a discussion into distinct segments? Summaries?

• Which characteristics of a post influence the knowledge construction phase of the next post?

 

PivotalPost

FunctionsSummary (+)…

RolesSynthesizer (+)…

Individual Control variablesGenderAge…

PostControl variables# of wordsTime of post…

Time contextWeek

 

KnowledgeConstruction

FunctionsSummary (+)…

RolesSynthesizer (+)…

Individual Control variablesGenderAge…

PostControl variables# of wordsTime of post…

Time contextWeekSegment

Methods

Function

Role

Give Direction

New Idea

BringSource

Use Theory Respond Summarize

Starter X X        Inventor   X        Importer   X X      Mini-me       X    Questioner         X  Elaborator         X  Devil’s Advocate         X  

Traffic Director X          

Synthesizer X         XWrapper           X

Content Analysis

Variable Inter-rater reliability ()Knowledge construction .84New Idea .65Bring in Source .92Use Theory .73Respond .98Give Direction .76Summarize .88

Unit of analysis: Post / Note / Message Objectively identified unit that its author defines

Rourke, Anderson, Garrison, & Archer, 2001Inter-rater reliability Krippendorf’s (range: -1 … 1; desired: > .67)

4 types of Analytical Difficulties

• Time

• Outcomes

• Explanatory variables

• Dataset

- No missing data

Statistical Discourse Analysis

Statistical Discourse AnalysisDifficulties regarding Time

Segments differ (S2 S4)

Serial correlation (p8 → p9)

Branches of notes

Strategies

Breakpoint analysis + Model Multilevel analysis (MLn, HLM)

Test with I2 index of Q-statistics Model with lag outcomes, KC (-1)

Store path: Identify prior turn

1

2

3

8

4

5 6

7

ID Action Turn # Valid?Previous

TurnValid (-1)

Ana Do three times four. 1 – –Ben Three times four is seven 2 X 1 Eva Three times four is nine. 3 X 2 XJay Three times four is twelve. 4 3 X

ID Action Turn # Valid?Respondto post?

Valid (-1)

Ana Do three times four. 1 – –Ben Three times four is seven 2 X 1 Eva Three times four is nine. 3 X 1 Jay Three times four is twelve. 4 3 X

Statistical Discourse AnalysisDifficulties regarding Time

Segments differ (S2 S4)

Serial correlation (p8 → p9) Multiple topics

Branches of notes (→→ )

Strategies

Breakpoint analysis + Model Multilevel analysis (MLn, HLM)

Test with I2 index of Q-statistics Model with lag outcomes, KC (-1)

Store path: Identify prior turn Vector Auto-Regression

Lag explanatory variablese.g., Valid (-1), Girl (-1) Valid (-2)

1

2

3

8

4

5 6

7

Statistical Discourse AnalysisOutcome Difficulties

Ordered outcome (KC 1-5) Infrequent outcomes (00010)

Strategies

Ordered Logit / Probit Logit bias estimator

Statistical Discourse AnalysisExplanatory model Difficulties

People, Groups & Topics differ

Mediation effects (X→M→Y)

False positives (+ + + +)

Strategies

Multilevel analysis Multilevel mediation tests

2-stage linear step-up procedure

Results – KC PhasesKC Phase % of Posts1) Sharing Information 60

2) Exploring Dissonance

3

3) Negotiating Meaning 16

4) Testing / Modifying 4

5) Agreeing / Applying 17

Results: Summaries as Pivotal PostsEach discussion averaged

1 pivotal post (2 time periods)

Results - KC Patterns

0 5 10 15 201

2

3

4

51a

0 5 10 15 201

2

3

4

51b

0 5 10 15 201

2

3

4

52a

0 5 10 15 201

2

3

4

52b

0 5 10 15 201

2

3

4

53

0 5 10 15 201

2

3

4

54

Knowledge Construction Phase

Post Number

No Regressive Segments

Pivotal Posts → Distinct Segments

No Regressive Segments

Segments Skipped

KC phases

Predicting Pivotal Posts

SynthesizerPivotal

PostExtensive SummaryWrapper

Role Current Post

Time 2 posts ago Previous post Role Current post

Knowledge Construction

Summary

After 1st

pivotal post

New Idea (-1)after 1st pivotal

post

Respond (-2) after 1st pivotal post

Wrapper

Synthesizer

Predict Knowledge Construction

KC pattern KC phase 1 KC phase 3 or 5

(Share) (Negotiate Meaning or Agree/Apply) Few KC phases 2 or 4 (Dissonance, Testing)

Pivotal post Extensive Summary often By Synthesizer or Wrapper usually

Extensive Summary Showed higher KC Elevated KC of subsequent posts

Summary of Results

Teacher / Designer Assign Synthesizer Role

- Increase midway summaries and elevate KC- Simple, effective intervention

Productive online discussions do not require all phases

Researcher Empirically test Gunawardena et al’s KC model New method for analyzing online discussion

- Statistically identifies pivotal posts & segments- Test hypotheses about relationships among posts - Examine variables at multiple levels - Examine differences over Time

Implications

Further Questions• With many choices of dimensions for the

breakpoints, which one(s) should we use?

• What do identification of same vs. different breakpoints across different dimensions tell us?

• How can we do meta-analyses of multiple data sets with somewhat different codes?

• Which analyses (qualitative and/or quantitative) might be fruitful on the same data set?