overview and preliminary results of using polycafe for collaboration analysis and feedback...

Barcelona, 30th September 2010 5th European Conference on Technology Enhanced Learning, EC-TEL 2010

Overview and Preliminary Results of Using PolyCAFe for Collaboration

Analysis and Feedback Generation

Traian Rebedea, Mihai Dascalu, Stefan Trausan-Matu et al.

“Politehnica” University of Bucharest, Department of Computer Science and Engineering

Overview and Preliminary Results of Using PolyCAFe for Collaboration Analysis and Feedback Generation

EC-TEL 2010, Barcelona30th September 2010

Overview

• Context: CSCL • Learning scenario: problems and solution• Our solution: PolyCAFe

– System architecture & Functionality– Words, voices & threads– Services & Widgets

• Validation Experiment• Transferability• Conclusions



Context: CSCL

• Collaborative learning through dialogue• Most popular web technologies: chats and forums• Take into account the content of the discussion• But collaboration is also important

• No tools for analyzing these online discussions• The need for automatic assessment and feedback• Due to the high number of participants, parallel

discussion threads may appear!



Learning Scenario

• Formal context– Undergraduate students in Computer Science– HCI course

• Students divided in small groups– Provided a topic for the discussion– First phase: document the topic, read materials– Second phase: participate in a chat conversation– Third phase (optional): receive feedback from tutors

• Similar scenario might arise naturally in an informal context (students or CoP)



Problems

• Tutors find difficult the analysis of the chat conversations– Multiple participants– Multiple discussion threads– Difficult to remember who said what and when (what was

the context of the conversation)

• Very time consuming for tutors• The feedback provided by different tutors is very

heterogeneous• Students receive feedback very late



Solution

• A software solution that:– Provides preliminary feedback and support to the

learner– Helps the tutors understand better and assess

quicker a conversation– The preliminary feedback is used by the tutor to

provide the final feedback to the learner– Offers improved search facilities to the users



Proposed Solution - PolyCAFe

• Chat and Forum Analysis and Feedback System• PolyCAFe = Polyphony-based Collaboration Analysis

and Feedback generation• Provides feedback to learners, tutors and teachers

related to the interaction of students in online discussions

• Takes into account both:– Content of the conversation (related to a domain or topic)– Collaboration (related to the conversation, participation,

etc.)



System Architecture



Technologies & Techniques

• Natural Language Processing– NLP pipe: spelling correction, phrase extraction,

stemming, POS tagger, NP-chuncker– Discourse analysis

• General elements: coreferences, argumentation• Conversation-related: speech acts

– Semantics• Linguistic ontologies (e.g. WordNet)• Tagged LSA (Latent Semantic Analysis)• Domain ontologies



Words, Voices and Discussion Threads

• The topic of the discussion defines a list of important concepts (words/stems/…) => initial voices

• Other voices appear naturally during the conversation– Some may be related to the initial ones

• Voices can influence each other through explicit or implicit links between utterances

• One or more similar voices can define a discussion thread• Discussion threads = a set of utterances that are linked

implicitly or explicitly and correspond to a specific voice or more similar voices



Functional Architecture

Social Network Analysis



Utterance Evaluation



Services & Widgets

• PolyCAFe is an online platform:– Web services– Web widgets using W3C Widgets1.0 standard

• The widgets can be integrated into any web platform that has a W3C widget container (e.g. Wookie)

• There are services for maintenance tasks and analysis tasks (process discussion, search, etc.)

• The widgets have been deployed in Elgg (PLE)



Presentation of Widgets

• Conversation Feedback (content + collaboration)• Conversation Visualization (collaboration)• Utterance Feedback (content + collaboration)• Participant Feedback (content + collaboration)• Search Conversation (content)

• Prepare chat/forum assignment• Add conversations for assignment



Conversation Feedback



Conversation Visualization



Search Conversation



More Examples?

• Interested to find more?• Click to watch tutorial (screencast)• http://augur.wu.ac.at/wp5/wp51/ltfll-student.

htm



Validation Experiment

• Validation of PolyCAFe 1.0– 9x students– 5x tutors, 1x teacher

• Two chats with 4-5 students in a team• All the students used PolyCAFe for the first time• Only two tutors have previously used the system• Students used the automatic feedback to get insight

about their discussion• Tutors used PolyCAFe to offer manual feedback



Validation Experiment (2)

• Brief information about the system• Watch screencast designed as a tutorial• Use PolyCAFe for 40-90 minutes + think-aloud• Each user wrote down observations while using the

system• Questionnaire with more than 30 Likert questions• Focus groups – 60-90 minutes



Validation Experiment (3)

• Monitored by two investigators that could only answer questions for usability issues– In order not to influence the users

• Most of the students were interested about the functionality of the system– How does it work? – What technologies?



Validation Results – Tutors

• The tutors validated all the instruments with suggestions to change part of them– Quiz with 35 questions – all of them passed with an

average score between 3.50-5.00 / 5.00

• The system is relevant and useful for their activity• The time for providing final feedback to the students

is definitively reduced (between 30-50%)• The quality of the feedback is improved



Validation Results – Tutors (2)

Category Average Perc. agreePedagogic effectiveness 4.11 83%Efficiency 5.00 100%Cognitive load 4.60 100%Usability 4.36 93%Satisfaction 4.57 91% Total 4.53 93%

1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree



Validation Results – Students

• The students have validated most of the instruments– Quiz with 32 questions – 5 where not validated; all others

have scores over 3.66/5.00

• In the focus group, they reported that several misleads have been found using this widgets

• These errors or misleads were reported to be only minor without influencing the overall feedback

• It has been suggested to try and fix them in order to gain the full trust of the users



Validation Results – Students (2)

• For example, the students did not validate:• “Overall, I believe that the support for my learning

PolyCAFe (Chat Analysis and Feedback Service) provides is close enough to the current support provided by humans. ”– Average = 3.11– Perc. agree = 33%

• However, we do not want to provide a substitute for human evaluation!



Validation Results – Students (3)

Category Average Perc. agreePedagogic effectiveness 3.94 77%Efficiency 4.22 78%Cognitive load 3.56 56%Usability 4.11 81%Satisfaction 3.89 72% Total 3.94 73%

1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree



Validation Results - Widgets

Validation Statement Learners Agreement

Tutors Agreement

PolyCAFe feedback is useful 100% 100%

PolyCAFe feedback is relevant 63% 80%

Conversation feedback is useful 78% 80%

Conversation visualisation is useful 89% 100%

Utterance feedback is useful 83% 100%

Participant feedback is useful 78% 100%

Search conversation is useful 61% 100%



Transferability Issues

• Domain– The topic of the conversation should be easily solved using

discussions, no graphics or formulas• Language

– Need for the components of the NLP pipe– Corpus for training the LSA– Maybe, a domain ontology

• Activity– Collaborative activity– Teams of 4-15 students (in the current design)



Conclusions

• PolyCAFe is a solution under development that uses a mixture of techniques:– NLP & SNA & Polyphony

• Adapted them to web conversations analysis• PolyCAFe 1.0 was found useful by students and

tutors• Tutors seem to agree more with the relevance of

PolyCAFe• Students might need some more time to

accommodate with the software



Conclusions (2)

• PolyCAFe 2.0 focuses on:– Improving the usability especially for students– Fixing some of the errors discovered in this

validation round

• PolyCAFe can be transferred to other domains and languages– Easier for English at this point



Open Points

• Maybe tutors are biased due to the time reduction for providing final feedback– Everyone loves to have more time for something

else

• The next validation is done for HCI and Professional Behavior in Medicine

• Looking for people interested to use PolyCAFe!



Test it!

• http://ltfll-lin.code.ro/ltfll/wp5/• Follow link to test PolyCAFe version 1.5• Still working for version 2

– release November 2010



More Information

• Polyphony theory of discourse for chats with multiple participants: – Trausan-Matu & Rebedea, CICLING 2010

• A more detailed technological and functional description of PolyCAFe– Dascalu, Rebedea & Trausan-Matu, AIMSA 2010

• Screencasts (video tutorials )for LTfLL services:– http://augur.wu.ac.at/screencasts/v1/



THANK YOU!

Questions &

Feedback

overview and preliminary results of using polycafe for collaboration analysis and feedback...

Education