what they did really do with the social software and mobile devices? a temporal analysis of...

"Exploring methodological approaches to conduct process analysis

in fragmented, long-duration collaboration mediated by multiple

artifacts."

Jari LaruLearning & Educational Technology Research Unit

University of Oulu10/2009

Pedagogical design..

Syndication service via RSS

Blog

Theoretical grounding

Groups’ grounding

Reflection,Constructing




Elaborating Constructing

Media-sharing services

Course wiki & Blog (announcements; schedules; lecture materials; workspaces; helpdesk)

Wiki

Media

Media

Media

Media

= Face-to-Face sessions

Meaning-making sessionCollaborative virtual work

Individual virtual work

Grounding sessionLecture

Monitoring


led to messy data..







Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media




Elaborating

time

Constructing

video recordings

media artefacts

media artefacts

video recordings

Blog content Wiki historylecturetopics

pre-test (conceptual knowledge)

+ questionnaire

post test SRI interviews

video recordings

RSS Feeds RSS Feeds

monitored content

conversationanalysis

Conversationanalysis

Content analysis

statisticalmethods


Content analysis

Content analysis

Content analysis

Content analysis

statisticalmethods

statistical methods used to calculate dependencies or probabilities between variables

Variable level

Event level

Ouput: a lot of different kind of frequency data a la coding categories, log data, conceptual test etc

Research problem was and is uh..challenging

What they did really do with the social software and mobile devices? A temporal analysis of long-term technology mediated collaboration.

Methodological skills oriented my interests towards variable

level approach

• Content Analysis

• Social Network Analysis

• Conversation Analysis

• Mann-Whitney U-test

• etc..

Variable level

Well, what’s the problem with variable data?

We werent’ only using MULTIPLE tools..

Blog Wiki

but we also had different kind of F2F settings in different phases

Variable level

Actually results lived in data archipelago, were small (or

big)islands there

Having a holistic approach into analysis and design, it was little

bit too tricky to start..

Ishikawa diagrams(also called fishbone diagrams or cause-and-effect diagrams)

First event-level try-out. No success. No methodological bacground. FAIL

Event level

FAIL

A Framework for Analyzing Interactional Processes in Online Learning

Daniel Suthers, Nathan Dwyer, Richard Medina, Ravi Vatrapu (2007). Paper presented at the Annual Meeting of the American Educational Research Association (AERA), Chicago, April

9-13, 2007.

FAIL

Too instrumental & multi-widget-chat tool orientation (several presentations at CSCL’09)

Event level

What’s next?• Didn’t found appropriate methodological

literature of event level analysis

• wrong terminology used (didn’t know)

Event level

WEB2.0 / Social WEB• this terminology didn’t help neither

• Methodologically exteremely heterogenous area

Variable level approachwas chosen..

Appropriate coding categories?

FAIL OK

Blog Wiki

Cress & Kimmerle (2008) A systemic and cognitive view on collaborative knowledge building with wikis. International Journal of Computer-Supported Collaborative Learning, 3(2), 105–122.

OK

Basic categories:“Besides assimilating information the wiki can also accommodate. This happens when new information is not only attached to the existing information, but the information in the wiki is organized in a new way.”

Problem remains

Data

Scheme

Variables

Data

Scheme

Variables

Data

Scheme

Variables

Blog Wiki OtherFAIL FAIL

Analytical toolkit for time-series data

MISSINGWiki: timetags are essential. No way to import into

NVIVO8Blogs: cumulative nature and “metadata” essential. No

way to import into NVIVO8

WHERE IS A TOOL FOR ANALYSING HETEROGENOUS, MULTITOOL TIME-SERIES

DATA? Answer: Tatiana! (CSCL 09)

TIME IS RUNNING!New tryout with variable

level approach

•How to measure dependencies between (all) variables??!

•Petri Nokelainen visited our group: IDEA! Bling!

Bayesian dependency modeling• SRL Self-report questionnaire (Pintrich, 1993; Wolters, 2001)

• Short-answer (paper & pencil) conceptual knowledge test

– 3 x 6 concepts (about lectures), same in pre/post tests

• Basic activity measures as “log data”

– # of images/videos (file-sharing services)

– # of articles/edits (blog, wiki)

– # of wordcount (blog, wiki, face-to-face sessions)

– # of items (rss)

• Qualitative data collection for other analyses (stimulated recall interviews, group interviews, content produced by learners, wiki history)

• Statistical dependencies between the variables were studied with Bayesian Dependency Modeling (BDM), which predicts the most probable statistical dependency structure between the observed variables (Myllymäki, Silander, Tirri & Uronen, 2002).

• B-Course: A Web-Based Tool for Bayesian and Causal Data Analysis was used as a analysis toolkit

• Bayesian analysis is based on propabilities instead of frequencies (Nokelainen, Silander, Ruohotie, Tirri, 2007)

var 1 var 2B-Course

LET - Learning and Educational Technology Research UnitJari Laru. 27.8.09 ERLI09, Amsterdam

Are learners’ actions in the web2.0 tools and F2F sessions related to their learning outcomes?

19

“monitor”

“reflector”

Some arcs are pretty strong, so that removing any of them would result in a model with probability less:* than one millionth of that of the original model** than one thousandth of that of the original model*** smaller than one thousandth (weak connection)

blog

mediasharing

Syndication service

wiki

wiki

direct dependencies:Learning gain:- RSS_Read (# of read RSS items)=> Monitor- WP_Entries (# of blog entries)=> Reflector

Learning outcome

FAIL

http://edufeed.wikispaces.com/















LET - Learning and Educational Technology Research UnitJari Laru. 27.8.09 ERLI09, Amsterdam

WHY IT WAS DOOMED TO BE A FAIL?

• Bayesian approach provided only surface level analysis:–deep insight into collaborative interaction in F2F groups, wikis & blogs missing

–order, duration and quality of events: eg. lecture > F2F > image/video > blog reflection > F2F > Wiki

• Interesting question “How learning unfolds over time” wasn’t answered at all..

• Didn’t support design of the study!

20

What’s next?

What I had on my hands after Bayesian?• Analytical toolkit: Tatiana

• Some coding schemes which didn’t stretch over the whole design

• Some methodological hints about event based approach..

• Intuition that event based approach is “my way”

Narrative and learning with web2.0 technologies: towards a research agenda.

Pachler & Daly (2009). Journal of Computer Assisted Learning. 25, 6-18

•Narrative trails (+)•Sense-making (+)•Blog centric (-)•vague understanding of collaboration (-)(commenting in blogs)•Holistic approach?..

Well.. hmm

Task level

Missing piece is ..

Missing piece

Time is precious: Variable- and event-centred approaches to process analysis in CSCL research.

(2009) Computer-Supported Collaborative Learning (2009). 4:239-257

Methodological link

Suther et. al (2007)graphical techniques

Mercer (2008) et. alcommunication &

learning in classroom

Muukkonen et. al (2008)

Time series analysis

Central tenet: “methods developed in conversation analysis and ethnomethodology...do not carry over to interactions that are

fragmented over time, stretch over longer durations, or are mediated by artifacts rather than talk” pp.241

Process method should.. (Reimann, 2009)

• a process method should, in addition to efficient cause (typical for event approach), be able to deal with at least two other kinds of causes:

• formal cause, referring to the patterns of which things are made

• final cause, the end for which things are made [goal, in our study: wiki]

One coding scheme to evaluate interpersonal exchange

• Mercer, N. (2008). The seeds of Time: Why Classroom Dialogue Needs a Temporal Analysis. The Journal of Learning Sciences, 17: 33-59

• Focus of article is on examining long-term classroom talk and how it’s used to:

• represent past experiences

• carry ideas forward from one occasion to another

• approach future activities

• achieve learning outcomes

I propose an applied approach for Mercer

• To evaluate interactions that are mediated by artifacts or than talk and can be categorized according Mercer:

• represent past experiences [for example: in personal blog, or F2F discussions]

• carry ideas forward from one occasion to another [photo => blog, or peer blog => own blog etc.]

• approach future activities [design groups’ / individual activities]

• achieve learning outcomes [troughout the design]

Unit of analysis in my research

• Main entities are individuals and groups

• events are constrained to those incidents in which either individuals or groups can participate

• Focus will be put on the sequences of activities, incidents, crises, or stages that unfold in the groups over the duration of the course

led to messy data..







Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media

Media




Elaborating

time

Constructing

video recordings

media artefacts

media artefacts

video recordings

Blog content Wiki historylecturetopics

pre-test (conceptual knowledge)

+ questionnaire

post test SRI interviews

video recordings

RSS Feeds RSS Feeds

monitored content

conversationanalysis


Content analysis

statisticalmethods


Content analysis

Content analysis

Content analysis

Content analysis

statisticalmethods

statistical methods used to calculate dependencies or probabilities between variables

Variable level

Event level

Ouput: a lot of different kind of frequency data a la coding categories, log data, conceptual test etc