HAL Id: hal-00190427https://telearn.archives-ouvertes.fr/hal-00190427

Submitted on 23 Nov 2007

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.

Combining qualitative evaluation and social networkanalysis for the study of classroom social interactionsAlejandra Martinez, Yannis Dimitriadis, Bartolomé Rubia-Avi, Eduardo

Gomez-Sanchez, Pablo De La Fuente

To cite this version:Alejandra Martinez, Yannis Dimitriadis, Bartolomé Rubia-Avi, Eduardo Gomez-Sanchez, PabloDe La Fuente. Combining qualitative evaluation and social network analysis for the study ofclassroom social interactions. Computers and Education, Elsevier, 2003, 41 (2003), pp.353-368.<10.1016/j.compedu.2003.06.001>. <hal-00190427>

Combining qualitative evaluation and social networkanalysis for the study of classroom social interactions

A. Martıneza Y. Dimitriadisb B. Rubiac E. Gomezb

P. dela Fuentea

aDpt. of ComputerScience, University of Valladolid, SpainbDpt. of SignalTheory, CommunicationsandTelematicsEngineering, University of

Valladolid, SpaincDpt. of DidacticsandSchool Organization,University of Valladolid, Spain

Abstract

Studyingandevaluatingrealexperiencesthatpromoteactiveandcollaborative learningis acrucialfield in CSCL.Major issuesthatremainunsolveddealwith themerging of qualita-tive andquantitative methodsanddata,especiallyin educationalsettingsthatinvolve bothphysicalandcomputer-supportedcollaboration.In this paperwe presenta methodologythatcombinestraditionalsourcesof datawith computerlogs,andintegratessocialnetworkanalysisin an overall qualitative evaluationapproach.Several computertools have beendevelopedto assistin thisprocess,integratedwith genericsoftwarefor qualitative analysis.We presentthemethodin thecontext of aneducationalandresearchprojectthathasbeengoingon for threeyears,to which we have incrementallyappliedandvalidatedtheevalu-ationdesignandtools.Our proposalandthepresentedcasestudyaim at giving ananswerto theneedof innovative techniquesfor thestudyof new formsof interactionemerging inCSCL;at increasingtheefficiency of the traditionallydemandingqualitative methods,sothatthey canbeusedby teachersin curriculum-basedexperiences;andat thedefinitionofasetof guidelinesfor bridgingdifferentdatasourcesandmethodologicalperspectives.

Key words: cooperative/collaborative learning,evaluationmethodologies,learningcommunities,post-secondaryeducation

1 Introduction

Designanddevelopmentof ComputerSupportedCollaborative Learning(CSCL)systemsis very complex, due to the diversity of implied actorsand the varietyof issuesto consider:learningimprovement,schoolorganization,cultural prob-lems, software design,distributed systemsmanagement,human-computerinter-action,etc. This complexity demandsappropriatemethodsof evaluationthat let

Preprintsubmittedto Elsevier Science 3 September2002

researchersandpractitionerslearnby applyinginnovativeexperiencesandreflect-ing onthem(Neale& Carroll,1999).Theapplicationof computersto collaborativelearninghasbeenconsideredasanew resourcefor researchin thefield,dueto theircapabilityof logging interactionsandprocessingthemautomatically. However, italsopresentsnew challenges,mainly relatedto the appearanceof new collabora-tive situationswith new forms of interactionand to problemsof automaticdatamanagementandinterpretation(Guribye& Wasson,2002).

In orderto supportthedevelopmentof CSCLsituationswe proposeda conceptualframework namedDELFOS(“a Descriptionof a tele-EducationalLayeredFrame-work orientedto LearningSituations”)(Osuna& Dimitriadis, 1999).It definesanarchitecturefor thedesignof CSCLapplicationsanda developmentmethodologybasedon the ideasof participatoryanalysisanddesign(Chin, Rosson,& Carroll,1997),which emphasizesthe role of formative evaluationin the developmentofinformationsystems.We arecurrentlyworking on the refinementof the methodsandtechniquesdefinedin DELFOSfor this formativeevaluation.For thispurpose,wedraw onasituativeapproachthatconsidersbothindividualaswell associalandparticipatoryaspectsof learning(Wilson & Myers,2000).In thepreviousversionof the framework, evaluationwasmainly orientedto the constructivist aspectsoflearning,focusingontheindividualratherthanonthesocialperspective.Therefore,wearenow completingtheevaluationmethodin DELFOSby definingamethodol-ogy andtoolsfor evaluatingsocialaspectsrelatedwith theparticipationin a com-munityof learners.

A discipline that has showed to be appropriatefor the efficient study of thesesocial and participatoryaspectsof learning is Social Network Analysis (SNA)(Wasserman& Faust,1994).Severalstudieshavedemonstratedits valuewithin theCSCLfield for thestudyof structuralpropertiesof individualslearningin groups(Nurmela,Lehtinen,& Palonen,1999;Cho,Stefanone,& Gay, 2002).Thesestud-iesusuallytake computerlogsasaninput,andprocessthemwith a SNA softwarepackage,suchasUCINET (Borgatti,Everett,& Freeman,1999).However, SNA byitself is not enoughfor achieving a full understandingof theproblems,andneedsto be complementedwith othermethods,like qualitative contentanalysis.Unfor-tunately, no guidelineshave beenprovided in the CSCL literatureregardingtheintegrationof qualitativedataandmethodswith SNA. On theotherhand,existingSNA tools requirea high level of expertiseandthey useproprietarydataformats.An approachbasedon XML (W3C, 2000)for the representationof collaborativeinteractionscould thenoffer a meansto solve the problemsof heterogeneityandintegrationin asystematicway.

We considerthat the principlesof qualitative casestudy(Stake, 1995)constitutea good framework towardsthe integrationof SNA methodsin the evaluationofCSCL experiences.This approachdraws on naturalisticresearchmethodsabletodealwith thesubjectiveandcomplex natureof thestudiedphenomenon.Casestud-ies performedunderthis perspective arebasedon the analysisof interactionsof

2

the participantsin the contexts wheretheseeducationalactionstake place.Thesestudiesuseethnographicsourcesof data,suchasobservations,questionnairesandinterviews,ableto capturetheperceptionsof theparticipants.Quantitativedatacanbeusedto accountfor theoccurrenceof actionsor events,andrelatethemwith thequalitative categories.This way, thequantitative databecomesanadditionalinputin theoverall qualitativeapproach.

In this paperwepresenta methodthatfacesthenew requirementsposedby CSCLsituations,enablingthe integrationof differentsourcesof dataandmethodsintoqualitative casestudiesorientedto formative evaluation.Part of the datacomesfrom event logs of computer-basedtools that studentsuseto fulfill the courseas-signments,while otherdataarecollectedby traditionalmeans(formalobservations,questionnaires,groupinterviews).As anintegralpartof thismethodwepresentthetools we have developedin order to increasethe efficiency and usability of theevaluationprocedures.In orderto exemplify thediscussion,wepresentthemethodin the context of the classroom-basedresearchprojectnamedLAO (Dimitriadis,Martınez,Rubia,& Gallego, 2001),on which we have beenworking during thelast threeyears.This projecthasbeentheplatformfor theconceptionandvalida-tion of theevaluationmethodpresentedin this paper. Weaimat providing answersto theneedof consideringnew formsof interactionthatarisewhenworking withnetworkedcomputers,andthe needof developingan effective evaluationmethodableto dealwith differentdata,methodsandtools.

The restof the paperis structuredasfollows: next sectionoutlinesthe projecttowhich the evaluationhasbeenapplied.Then,the researchmethodand tools de-velopedfor its supportarepresented.The third sectionillustratesthe useof themethodfor thestudyof onecategory extractedfrom thecasestudy. We alsoelab-orateon theadvantagesandlimitationsof theproposedmethodology. Finally, wedraw someconclusionsandoutlinefuturelinesof research.

2 Case study description: The LAO project

During the last threeyearswe have beeninvolved in a classroom-basedresearchanddevelopmentprojectaimedat the introductionof project-basedlearningwithcasestudiesandcollaborative learningin acourseof ComputerArchitecturein thestudiesof TelecommunicationsEngineering.Thedetailsof thisprojectandits evo-lution canbefoundin (Dimitriadiset al., 2001).Here,wepresentits mainfeaturesasthey constitutethecasestudyin which our evaluationmethodandtoolsevolvedandwerevalidated.

Following theprinciplesof theeducationalmodelof DELFOSaswell asthedirec-tivesof the IEEE/ACM ComputingCurricula,theprojectaimsto provide contex-tualized,integratedandmeaningfulknowledge;promotingactive, intentionaland

3

collaborative learning.Besidestheselearningobjectives,the LAO projectservedasaplatformfor educationalresearch,whereseveralissuesrelatedto theimpactofthepedagogicaldesignandtoolsonattitudestowardscollaborationhavebeenstud-ied. Theexperimentalwork hastakenplaceduringthe last threeyears,during thefall semester(Septemberto February)of the academicyears1999-2000to 2001-2002.The generaldesignwasvalidatedduring the fist year. The revisedprojectwasextensively andsystematicallyevaluated,in orderto assessits effectivenessatfulfilling theaforementionedlearningobjective.Generalfindingsof thisevaluationcanbereadin (Dimitriadis et al., 2001).Herewe presentthemain featuresof theprojectasthey constitutethesettingwherewehaveappliedour evaluationdesign.

The wholecourseis definedasa projectthatdevelopsalongthesemester, whoseobjective is thedesignandevaluationof computersystems.In orderto enabledis-tinct perspectivesof thesubjectwithin theclassroom,fivecasestudies(clients) aredefined,covering differentmarket sectorsandsystemrequirements.The teachertakes the rolesof the differentclientsandthe director of the manufacturer com-panies. Studentswork in pairs,and assumethe roles of a consultingfirm andacomputermanufacturer. Eachpair is assignedoneout of thefive casestudiesforthewholecourse,i.e. they serveonly oneof thefiveclients. This way, in eachlab-oratorygroupof atmostforty students,differentclientsarebeingstudiedalongthecourse.Theeducationaldesignaimedatpromotinginteractionwithin andbetweenthepairsassignedto differentclients.

Theprojectis dividedinto threesubprojectsthatstudydifferentspecificissuesofthewholeproblem.Everysubprojectpresentstwo milestones.In thefirst one,basicdecisionsaretaken,andin thesecondmilestone,eachpair hasto submita formaltechnicalreport to the client (teacher).In every milestone,eachlaboratorygroupholds a debate,designedas a collaborative review of the work of the students,andwherethe problemsof the differentclientscanbe sharedanddiscussedat alaboratorygrouplevel. At the endof the whole project,a technicalreport is col-laboratively producedamongall pairsthat dealwith the samecasestudyin eachlaboratorygroup.

Severaltoolsareusedtosupporttheproject.BSCW(BasicSupportfor Co-operativeWork) (Appelt,1999),awell-known sharedworkspacesystembasedonwebinter-face,wasusedfor asynchronousdocumentsharingandthreadeddiscussions.Ofspecialinteresthere is the fact that BSCW logs every action performedon thesharedworkspace,providing datathatwereusedasasourceof theanalysis,asex-plainedin the following section.Other tools, like e-mail for communicationandsimulatorsfor theassignmentsarealsoused.

Next sectionpresentsthemethodthatwasappliedfor theevaluationof theresearchobjective of the LAO project,while specificfindings in the LAO casestudy arepresentedin section4.

4

3 Mixed evaluation methodology for the study of social aspects of learning

Many featuresincludedin the evaluationmethodproposedin this sectiontry togiveananswerto thenew requirementsposedby CSCLwith respectto thisgeneralproblem.We outlinebriefly theseissuesbeforeproceedingwith thedescriptionofthemethod,asthey areimportantto understandthemotivationsof theproposal.

An importantconsequenceof theuseof computersto supportcollaborativelearningis thefactthatmany researcherssaw in it anopportunityfor evaluation,dueto theirstorageandprocessingcapabilities.This way, log files providedby the computerarenowadaysa commonsourceof data,normallycombinedwith moretraditionalones.However, not everythingis anadvantage.Computermediationof collabora-tivelearninghasintroducednew challengesto evaluation.Someof themarerelatedto new featuresof the situation,like the needto dealwith asynchronousanddis-tant settings(Neale& Carroll, 1999)or new forms of interaction(Crook,1994).Otherproblemsrefer to datamanagement,interpretationandprivacy (Guribye&Wasson,2002).Many studiesreportedin the literatureusead hoc datalogs notspecificallytailoredfor evaluation.This leadsto largeamountsof unnecessaryandredundantdata.Furthermore,the format is not suitablefor direct processing,andthey do not provide thesemanticinformationneededfor automaticprocessing.Itis not commonto find customizationfunctionsthatallow theteacheror researcherto selectthe kind of study shewantsto perform,or to dealwith privacy issues.Therefore,challengesin CSCLresearcharehow to make log dataeasilyavailableto researchers,allowing themto configuretheevaluation;how to performdatapro-cessingby automaticmeans;andhow to presentresultsin anintuitiveformat.

Schema of categories �

(concepts, procedures, attitudes) �

Questionnaires �

Observations �

Automatic �

data �

Event logs � At appropriate

milestones

Face to face

interactions

Final

(individual)

Previous

concepts (Individual)

Previous

social

relationships (in pairs)

Final social

relationships

(in pairs)

Research Phases

Qualitative analysis �

(NUD* IST) �

Social Network �

Analysis �(SAMSA)

�

Descriptive statistics �

Students' opinion of educational

project

Group interviews �

Initial

At appropriate

milestones

Final

Analysis methods

End of project

Along the

project

Preparation

Daily activities

Use of tools

Sources �of data �

Quantitative data �Qualitative data �Relationships data �

Fig. 1. The proposedmixed evaluationscheme:Data sources,methodology, timing andanalysistools.Arrows show informationflow paths.

5

In orderto facetheseproblems,we have defineda mixedevaluationmethod,de-picted in figure 1. It usesseveral sourcesof dataandanalyticalmethods,and issupportedby automatictoolsto increasetheefficiency of theoverallprocess.

3.1 Datasources

Theresearchmethodusesethnographicdatafrom avarietyor sources.It combinesdifferent questionnaires:general questionnairesat the beginning and end of thecourse,with openandclosedquestionsregardingtheresearchobjective; students’posthoccommentsto gettheirshort-termimpressionsaftermeaningfulactivitiesorevents;students’critics abouttheeducationalproject,submittedasappendicestothereports,in orderto gettheirsubjectiveview of theproject.Groupinterviewsareheldwith a groupof volunteersat appropriatemilestones,includingthebeginningandtheendof thecourse,in orderto gaininsight into thestudents’point of view;classroomnon-participantobservationswhereanobserver takesnoteof thediffer-entinteractionsandattitudestowardsparticipationin thestudents’daily work atthelaboratory. Finally, log files register interactionsheld throughthe CSCL tool thatis beingused.While interview andquestionnairesaremoresuitablefor acquiringa participant’s perspective of theproblem,datacollectedautomaticallyandobser-vationsarebetterfor measuringactualuseof thetoolsandtheinteractionsarisingfrom it. This varietyof sourcesaimsat supportingadatatriangulationschemethatconsidersthenew requirementsof CSCLsettings.

Partof thisdatais processedby softwaretools,asexplainedin section3.3.It is im-portantto notethattheuseof thesetoolsrequiresthedatato bedescribedatanap-propriatelevel of abstraction,suitablefor beingprocessedby acomputer. WehavechosenXML (W3C, 2000) as the datarepresentationformat. The main reasonsfor this choiceareself-descriptiveness,standardization,andinteroperability. XMLfiles areeasyto understandandproduce,as they follow a syntaxthat is definedby meansof a DTD (DocumentTypeDefinition). In our case,this DTD describesin anabstractmannerthedifferenttypesof interactionthatcanbeencounteredindifferentCSCL scenarios.By definingthis DTD, we areproviding a unified rep-resentationof thedistinctsourcesof data,andthusavoiding thecumbersomedatatransformationprocessestypical in scenarioswith differentdataandtools.More-over, beingagenerallyacceptedstandard,developersof toolscantakeadvantageoftheincreasingnumberof technologiesbasedon XML; while final users(teachers,researchers)might benefitfrom the integrationof theproposedtoolswith genericwebtechnologiesor anothersystembasedin XML. FurtherdetailsabouttheDTDandits useto supportevaluationcanbe found in (Martınez,Dimitriadis, & de laFuente,2002).

6

3.2 Descriptionof themethod

The evaluationmethodis basedon the principlesof casestudy research.As ex-plainedbeforehand,the studyof CSCL situationsneedsto be donefrom a qual-itative standpoint,which aimsto understandeachexperienceconsideringits con-text andevolution.Amongthequalitativemethods,casestudiesareappropriateforevaluations,asthey dealwith theintensivestudyof oneor few examplesof certainphenomenon.Quantitativedatacanbeusedin thestudyto accountfor occurrencesof actionsandrelatethemwith thegeneralscheme.

As shown in figure1, thestudystartswith thedefinitionof aschemeof categories,whichcanbedoneempirically, basedontheresultsof pastexperiences,or theoreti-cally, accordingto thespecificevaluationobjectives.Theschemeevolvesalongthestudy, by specializationof existing categoriesor additionof new onesthatemergefrom theanalysis.This qualitativeanalysisis fed by qualitativedatasources(openquestionnaires,observations,groupinterviews) in a traditionalmanner. Addition-ally, quantitativedataandSNA resultsarealsosubmittedto thequalitativeanalysis,in thewayexplainedbelow.

Quantitativeanalysisof closedquestionsis performedin orderto accountfor occur-rencesof facts,possibleproblematicpoints,etc.,in anefficient manner. It consistsof simpledescriptive statisticalanalysisassistedby any of thecurrentlyavailablestatisticalpackages.We “qualitize” thesequantitative databy including them asnew inputsto thequalitativeevaluation.

On theotherhand,integrationof SNA in theoverallevaluationschemeis achievedby identifyingasetof SNA measurementsthatrelateto oneor moreof thequalita-tive categories.As mentionedbeforehand,we have introducedSNA in theevalua-tion processdueto its potentialfor thestudyof socialrolesandstructuralpatternsrelatedto participationin thecommunity. Givena socialnetwork, in which actorsarelinkedby meansof oneor severalrelationships,SNA providesstructuralmea-surementsthatdescribethenetwork asa whole,aswell asactors’measurements,yielding informationabouttheparticipationof eachactorin thenetwork. We haveidentifiedasetof SNA indicatorsasa startingpoint of our proposal:Networkden-sity (D), actor’sdegreecentrality (CD

ni � ), andnetworkdegreecentralization(CD)

(Wasserman& Faust,1994).D measureshow muchknittedis anetwork, with val-uesrangingfrom 0 (mostscarce)to 1 (mostdense).Degreecentrality is an indexof actor’s prestige.Given an actorni , CD

ni � is the proportionof actorsthat are

adjacentto ni. It givesan ideaof theactivity of theactors.In thecaseof directedrelationships,that considerthe direction of the link, two degreeindexesare de-fined: indegree, or thenumberof links terminatingat thenode;andoutdegree, orthenumberof links originatingwith thenode.Finally, networkdegreecentraliza-tion (CD), is a group-level measurebasedon actor’s degree,which givesan ideaaboutthedependency of thenetwork on theactivity of asmallgroupof actors.Di-

7

rectednetworksdefinethecorrespondingindexesof indegreecentralization(DID)andoutdegreecentralization(DOD).

A relevantfeatureof socialnetworksis thatthey canbevisualizedasgraphscalledsociograms,which representtheactorsasnodesof thegraphsandthelinks amongthemaslines in thegraph.A convenientrepresentationof sociogramsis producedby theuseof MultidimensionalScaling(MDS). MDS mapsthesimilaritiesamongactors,sothatthosethataresimilar to eachotherin theinput dataappearcloserinthegraph,andviceversa.Usinggeodesicdistancesasameasureof dissimilarity, asociogramwill show in an intuitive mannersubgroupsof inter-relatedactors,andsomerelevantpositions,suchasthemoreandlessprominentactors(Wasserman&Faust,1994).

In orderto performsocialnetwork analysis,we needto definethesetof networksandrelationshipsto which thestudyis to beapplied.This taskdependson thepar-ticularcontext beingstudied.However, it is possibleto proposeasetof genericnet-works,definedfrom differentdatasources,namely:questionnaires,observations,logsof interactionsthroughthecomputer. We will show examplesof themin sec-tion 4.2.Weshouldnotethattheuseof thesedifferentdatasources,aswell asin theoverall evaluationscheme,complementsthe informationandprovidesfor a morethoroughandreliableunderstandingof theprocesses.

3.3 Supportingevaluationtools

Ourglobalevaluationschemeissupportedby threesoftwaretools:QUEST, SAMSA(Systemfor Adjacency Matrix andSociogram-basedAnalysis),andNud*IST. Ad-ditionally, any statisticalpackagecanbeusedfor thequantitative analysis.Figure2 shows their usein the overall analysismethod,which will be outlined in thissection.

QUEST (Gomez,Dimitriadis, Rubia,& Martınez,2002)allows for the designofquestionnairesbyateacheror researcher, andtheirpresentationasweb-basedformsto the students.QUEST collectsautomaticallythe resultsandconvertsthemintodifferentformats:RTF filesfor its usewith Nud*IST, spreadsheetfilesfor quantita-tivedataanalysis,andXML files representingtheinteractionsfor SNA. Weshouldpoint out that theseautomaticconversionsof dataaresimple,but very necessaryfor improving theefficiency of thewholeprocess.

SAMSA1 supportsthe socialnetwork analysisproceduresof the evaluation.Asshown in figure2, it containsseveral input modules(obs2xml, el2xml), that takedatafromdifferentsources(observationsandeventlogsrespectively)andtransformtheminto theXML file representingtheinteractions.Then,SAMSA allows there-

1 Namedafterthesubjectto Kafka’s Metamorphosis.

8

SAMSA

SNA �

processor

Teacher / Researcher

Configuration parameters

DL file (UCINET format)

BSCW

QUEST

obs2xml �

Relationships answers

Students

Answers Shared workspace

actions

el2xml �Event logs

Questionnaire definitions

Interaction maps NUD*IST

Sociograms & Relational

measurements

Numerical answers

New categories

face to face interactions

XML file

Qualitative categories

Educational tool

Evaluation tool & module

File

RTF files (open

answers)

STATISTICAL �

PACKAGE

Descriptive statistics

& inferences

Scheme of

qualitative

categories

Concepts

Attitudes &

Procedures

towards

collaboration

Formal observations

Fig. 2. Theproposedanalysistools,asrelatedto thepotentialactors(students,researcher,teacher),showing informationflow duringeducationalresearch.

searcherto selectandconfigurethe network shewantsto study(selectingdates,actors,andrelationship).The tool builds the matrix that representsthe network,known associomatrix, andcomputesthe indexeschosenby theresearcher. It alsoshowsthesociogramrepresentingthenetwork, andallows for actors’attributesvi-sualization.SAMSA supportsall theaforementionedSNA measurementsthathavebeenidentifiedfor ourgeneralevaluationpurposes.However, deepernetwork anal-ysismight benecessaryto find out new featuresof thenetworks.In orderto allowfor this deeperanalysis,SAMSA producesa DL (DataLanguage)file, thatcanbereadby UCINET (Borgattietal.,1999),agenericSNA softwarepackage.Throughthis feature,SAMSA supportsgenericsocialnetwork research,by automaticallyconvertingdifferentsourcesof data,andthusavoiding thesecostly datatransfor-mationprocedures.

The overall evaluationprocedureis supportedby a third tool, Nud*IST (QSR,1997),a well known qualitative dataanalysispackage.As explainedin the pre-vious section,the qualitative analysisis at the coreof the process,andtherefore,this tool receivesdatafrom the restof the elementsof the system(figures1 and2). It takesdirect input from QUEST(free-formquestionnairesin RTF format),aswell as the transcriptionsof the observationsandthe group interviews. As men-tionedbeforehand,both thestatisticalanalysisandtheSNA resultsareprocessedwith Nud*IST to fulfill thequalitativeanalysis.

Thedatasources,proceduresandtoolsdescribedin this sectionconstituteour pro-posalof a mixedmethodfor evaluation,in which SNA methodsandquantitativestatisticshave beenintegratedin anoverall qualitative scheme.SNA hasbeenin-troduceddueto our interestin the studyof participatoryaspectsof learning,andquantitativedatais usedin orderto detectgeneraltendencies.All themethodsarefedwith datacomingfrom differentsources,facingtheneedof capturingthediffer-entformsof interactionthatarisein computer-network supportedclassrooms.This

9

way, we provide for methodaswell asdatatriangulationthatwill leadto increasethe evaluationreliability. The needof unifying the differentdatasourceshasledusto proposetheuseof XML andthesubsequentdefinitionof a DTD thatmodelsdifferent typesof collaborative interaction.Next sectionwill illustrate the useofthesystemwith anexamplerelatedto theLAO project.

4 Collaboration as sharing information: an example of the method in theLAO project

The methodproposedin the previous sectionhasbeenappliedto theeducationalandresearchprojectdescribedin section2. In thepresentsection,we illustratetheproposedschemeusinga specificexample,that focuseson a category (collabo-ration as sharinginformation) that is speciallyrelevant within the context of theeducationalresearchobjectiveof theproject.

4.1 Evaluationdataandproceduresof theexperience

Theexamplepresentedin thissectionhasbeenextractedfrom thegeneralresultsoffall 2001casestudy, thatwasappliedto a coursein thefourth (out of five) yearofthestudiescarriedout in theTelecommunicationsEngineeringSchoolat theUni-versityof Valladolid,Spain.Thewholeclassof 100-120studentswasdividedintothreesessionsof forty students(maximum),in which theelementaryunit consistsof groupsof two students(pairs).The fifteen-weeklong semestercorrespondstothreesubprojectsof four weekseach,wherethereviews(synchronousdebates)tookplaceevery two weeks.Elaborationof thefinal reportstartedin thesixthweek.Fi-nal reportwasto besubmittedamonthafterthecoursehadfinished,andtherefore,duringthis period,therewereno lectureswherestudentscouldmeetfaceto face.

Themain issueof evaluationwasto know if theseinnovativemethodswould suc-ceedin developingnew concepts,attitudesandprocedurestowardscollaboration,in thecontext of thepassiveandindividualisticcultureof Spanishuniversities.Wewerealsointerestedin therole of computer-basedtoolsin this potentialchangeofperspective.An initial schemeof categorieswasdefinedaccordingto theseevalua-tion objectives,andtakinginto accountthecategoriesidentifiedin fall 2000study.

One of thosecategorieswas “collaboration as sharing information”. This con-cept is of specialinterestin our educationalsettingfor several reasons.First, theemergenceof a communityof learnersrequiresthefreeflow of informationin thegroup,whereeverybodyhasthefeeling(andthecertainty)thatthey canaccessim-portantdataavailablein thecommunity. Thiswasamajorchallengein ourproject,as the previous experienceof the studentswascloserto that found in traditional

10

classrooms,whereinformationis expectedto flow mainlybetweenteacherandstu-dents.Second,computersupportedcollaborationin theprojecttakesplacemainlythroughsharinginformation:BSCWcanbeusedin orderto sharereports,papers,URLs andotherinformationresources,andstudentswerestronglyencouragedtodosoby theteacher. And finally, theeducationalprojectwasalsodesignedin orderto promotethesesharinginteractions.The studentswereasked to write the finaltechnicalreportamongall thepairsof eachlaboratorygroupdealingwith thesameclient, thusrequiringthemto share(at least)their subprojectreports.They werealsoaskedto comparetheir solutionswith therestof theclients, whichwasitself ameansof encouragingthestudentsto accessother’spairsinformation.

Theevaluationfollowedtheschemepresentedin section3, adaptedto thepartic-ular characteristicsof the project.Formal observationstook placeevery week inoneof the laboratorygroups,andgroupinterviews wereheldwith a groupof tenvolunteers.Questionnairesweresubmittedto thestudentsfollowing theevaluationdesign,andBSCWeventswereloggedduringthewholecourse.Severalsocialnet-workswereconstructedto inspectthe interactions.Theexamplepresentedin thissectionaimsat showing how thedifferentsourcesof data,methodsandtechniquesof analysisprovideabetterunderstandingof theconceptunderstudy.

4.2 Analysisprocedures

Right beforethe coursestarted,previous conceptsrelatedto collaborationwereinspectedthroughtwo questionnaires,onedevotedto establishaninitial socialnet-work of collaboration,while the other wonderedfor their perceptionof it. Theresultsof theformerwereprocessedwith SAMSA, yielding theindexesshown inthe first columnof table1. They indicatethat their previous experiencesdid notincludesharing informationasa main form of collaboration(this conceptis theleastdense,with Di=0.29%),whereassolving doubtsand creating a product incommonwerethemostdenserelationships.In addition,in thelatterquestionnaireonly a few studentsregardedtheclassroomenvironmentascollaborative (16.5%),andsomeothers(30.6%)saidthatcollaborationhappenedonly with friends.Sur-prisingly, mostpairs (75 vs. 10) declarethemselvesmotivatedto work in group.This pointsout an apparentcontradiction:they appreciatecollaborationasanab-stractvalue,but their previousexperienceis ratherpooranddoesnot actuallytendtowardscollaborativeattitudes.

Our evaluationsettingallows us to observe the evolution of this concept,sincethesametwo questionnaireswererepeatedright afterthecoursefinished.Regard-ing the relationshipsquestionnaire,the densityof all activities increasedsignifi-cantly, notablysharingof information, thatalmostdoublesitsvalue.Positiveresultsalsoappearedin the final general questionnaire, wherethe samequestionabouttheclassroomcollaborationenvironmentgot thefollowing distribution: 6.8%very

11

competitive;18.2%competitive;9.1%indifference;40.9%only with friends.Thisinitial observationreflectsthepositiveimpactof theeducationalproject,but shouldbefurtherconfirmedby therestof theanalysistoolsin our methodology.

Table1Densitymeasurementsfrom the networksbuilt from questionnaires (Di: initial question-naire networkdensity;D f : final questionnaire networkdensity)

Relationship Di D f

Discussing 0.34% 0.48%

Solvingdoubts 0.43% 1.11%

Sharinginformation 0.29% 0.53%

Createaproductin common 0.77% 0.82%

With respectto themeansfor sharinginformation,BSCWoffersasharedworkspacewherestudentswereencouragedto publishtheirdocuments,piecesof information,andnotes,while theteacherusedit to deliverdocumentsandcommentsto theclass.An obvious questionthat emergesnow is whetherBSCW helpedthe aforemen-tioneddevelopmentof collaborative attitudeswith respectto sharinginformation.In orderto answerthisquestion,asocialnetwork wasconstructedonthebasisof in-directrelationshipsthatstemfrom postingandreadingotherspeople’sdocuments,sothata link betweenni andn j representsn j accessinganobjectcreatedby ni. Thetwo networks,oneincluding the teacherandother leaving him out, will allow totesthis overall influencein theuseof BSCW. Table2 shows densityandcentral-izationindexesof thenetworks,dividedin four periods:thethreesubprojects(sp1,sp2, sp3) andthefinal project(spf). Thetwo columnsto theleft show resultsof thenetwork includingtheteacher, while columnsto theright, resultswithouthim.

Bothnetworksshow thatdensitydecreasesalongtime,duringthethreesubprojects.The low useof BSCWin this periodof thecoursewasconfirmedby mid-course,througha questionnaireperformedin oneof themilestones(post-review question-naire), in which studentswereasked if they intendedto post their notesand in-termediatedocumentsin BSCW, anda latterquestionnaireaskingif they actuallydid soandwhy. Surprisingly, twentypairsexpressedtheir intentionof sharingtheirdocuments,but just threeof themdid it. Whenasked in the following milestonethroughanopen-endedquestionnaireaboutthe reasonsof this contradiction,theyarguedlackof timeor confidencein their contributions.

Nevertheless,the table shows a sharppeakof density in the period of the finalproject.This increaseof activity is explainedby severalreasons:at this timeof theyearthey wereon examinationperiodwithout lectures,andtherefore,they couldnot meetfaceto face.They startedto postdrafts,notes,comments,andnumericalresultsin BSCW andreadalmostall the documentspostedby otherpairshavingthe sameclient (i.e. the samecasestudy),which supportsthe idea that collabo-ration as sharinginformationdevelopspartially influencedby the relationshipof

12

collaborationasgeneratinga commonproduct.

On the otherhand,outdegreecentralization(COD) givesan ideaof how balancedwasthe creationof documents,which canbe regardedasactive sharingof infor-mationin BSCW. By observingtheresultsit is possibleto seethattheteacherhasa stronginfluence,with a peakin the secondsubproject(sp2). By inspectingtheXML translationof theBSCWlogs,wecouldeasilyfind outthatduringthisperiod,theteacherpublishedarticlesandcommentsregardingthefirst subprojectreports,thatwerereadby all thestudents.COD is alsovery high in thefinal project(spf),whentheteacherpublishedcommentsto thesuccessiveversionsof thefinal reportssubmittedby thestudents.Thesecommentsweremassively readby thestudents.

Table2IndexesfromtheBSCWindirect interactionsnetwork.D: normalizeddensity;COD: outde-greecentralization

Without teacher With teacher

Period D COD D COD

Sp1 17.65% 115.57% 21.93% 374.38%

Sp2 13.73% 88.93% 17.54% 1668.21%

Sp3 10.13% 137.02% 14.33% 226.85%

Spf 31.05% 157.79% 35.38% 775.31%

Sociogramscomplementtheinformationof thetablesabove,showingatfirst glanceanideaof how thedifferentactors(pairs,in this case),aresituatedwith respecttothe relationship.Figure3 shows the sociogramsof the indirect relationshipsnet-work during the fourth period of the course,correspondingto the final projectwriting, with the teacher(x00) includedin the network. We canobserve severalinterestingfeatures.First, thenetwork shows a completegraph(all nodesarecon-nectedto eachother),which wasnot the casewith othernetworks built from thequestionnairesandfrom themapsof interactionstakenin theobservations.There-fore,a first conclusionis thatBSCWsucceedat eliminatingobstaclesbetweenthestudents.Second,we seethat, in spiteof the teacherbeing a most centralactorasexplainedbeforehand,somepairs(x26,x34,x23,x39) occupy centralpositionsinthesociogram.Inspectingthelogs,we couldseethatsomeof thesepairshadpub-lishednotesandcommentsalongthecourse,whichweremassively readby therestof thestudents.Thus,thesecentralstudentscouldbeidentifiedasbeingthe infor-mation“sharers”of theclass.Third, pairsareplacednearbyothersthat sharethesameclient, dueto thefactthatin this periodBSCWwasusedmainly to exchangeinformationandmessagesrelatedto thefinal report(thathadto bewritten amongseveralpairswith thesameclient). Sincethisclient-centeredorganizationcannotbeobservedin thesociogramscorrespondingto therestof thesubprojects(notshownhere),we may concludeagainthat writing a joint report (i.e. having a commonproduct)significantlyincreasedcollaborationthroughsharinginformation.This isconfirmedby thefollowing statementsfrom thefinal groupinterview: “[...] in this

13

projectwedid useBSCWquitea lot to make appointments,to publishthe things,to..., in this onewedid, in theothers wealmostdid not” ; “We usedBSCWsothatthepersonin chargeof puttingthepiecestogether, coulddoit at homein a moment,andthenthenext daywemetwith theprintedversionandon it wewere modifyingit.”

Concludingthissection,wecanresumesomeof thefindingsthatreferto theevalu-ationobjectiveandpoint out how theproposedschemehelpedto achieve theeval-uationgoal.

Resultsyieldedby the quantitative analysisof the pre- andpost-coursequestion-nairesreflectedan improvementin the student’s perceptionof their own collabo-rativeattitudestowardssharinginformation.Thispositiveresultwaspartially con-firmed by the rest of the methods,which helpedto uncover somenew aspects.Comparingthe differencesbetweenthe social networks of the threesubprojectsandthefinal one,wecanconcludethatBSCWwasmainly usedto getinformationfrom the teacher, andonly in the last periodstudentsstartedto shareinformationthroughit. A deeperqualitative analysisof student’s answersto open-endedques-tions,andof the groupinterviews confirmedthat this increasein the interchangeof informationwasmainly dueto practicalreasons,andnot so muchto the aimof sharingtheir own knowledge.On the otherhand,quantitative analysisshowedthat the generalperceptionof the classroomcollaborative environmentimproveddramaticallyafterthecourse.In conclusion,wecansaythattheeducationaldesignalmostobliged the studentsto practicenew forms of collaborationthat, in spiteof not beingasconstructive asintended,helpedto improve theoverall classroomcollaborativeenvironment.

Regardingtheevaluationmethoditself, wehavealsoobservedthatsociogramscanintuitively unveil interactionpatterns,both throughthecomputer(usingcomputerlogs)or directly (usingtheobservernotesassourcedata).Thediscoveredpatternscanbeconfirmedby densityandcentralizationscores,providedby theSNA tools.Whenit comesto explainmore,purequalitativetoolslikeNud*IST helpcategorizetheinformationcomingfrom thequalitativedatasources,asquestionnnaires,inter-viewsor theobserverannotations.Therefore,thecombinationof severalsourcesofdataresultsextremelyconvenientto avoid falseor incompleteconclusions.How-ever, we have seenthat triangulationin mixedevaluationschemesmay imposeanunaffordableburdenon researchers,in termsof time efficiency andusability. Ourexperienceat theLAO projectcansuggestthattheproposedtoolstogetherwith theguidelinesmaysignificantlyalleviatethis problemin CSCLsystemsevaluation.

14

Fig. 3. Sociogramof theindirectrelationshipsthroughBSCWnetwork duringthelastpe-riod (writing of the final report).The teacherand the differentclientsarerepresentedindifferentstyles.

5 Conclusions and future work

We have proposeda mixed evaluationmethodthat aims at the study of partici-patoryaspectsof CSCLenvironments,by includingSNA techniques,quantitativestatistics,andcomputerdatalogsinto anoverall qualitativecasestudydesign.Theproposedschemehasbeenextensively testedduringa threeyearperiodin a semi-presentialuniversityenvironmentwith considerablesuccessandits mainobserva-tionshave beenpresentedin this paper. We canarguethattheproposedevaluationdesignis generalenough,andits ideascanbeadoptedin CSCLenvironmentsdif-ferentfrom theonethathasbeenconsidered.Thisstatementhasbeenvalidatedbyourown experiencein theevaluationof asignificantlydifferentenvironmentbasedon a synchronouscollaborative puzzlefor kindergartenchildren(Martınezet al.,2002).

Oneof the major educationalresearchissuestackledby our designrefersto us-ability andtime efficiency. Whenworking on formativeevaluation,time limits arevery important,ascorrective actionscouldbeof no valueif resultscometoo late.In thissense,QUESTprovedveryefficientandflexible atdesigning,collectingandconverting questionnairedatainto adequateformats(both qualitative andquanti-tative). Besidesthat,SAMSA allows for an easyandefficient processingof datacomingfrom varioussources(interactionmapsfrom observations,computerlogs,questionnaires)andprovidesconfigurableandeasyto visualizeSNA results.Theenhancementin usabilityandefficiency hasbeenclearlyobservedalongthe threeyearsof field work within the LAO project.As the proposedmethod,guidelinesandtools evolved,our effort couldmainly focuson theevaluationobjectives,be-

15

ing alsoableto reactwith corrective actionsandto supportpartially theformativeevaluationprocessof thecourseteacher.

Theuseof XML to representtheinteractionsprovidesfor theconceptualandoper-ationalintegrationof thedifferentdataundera commondescription.It is thecoreof a looselycoupledarchitecture,in whichnew modulescanbeaddedto dealwithnew sourcesof data.Additionally, developerscantakeadvantageof thelargeofferof XML-basedtechnologiesfound in the market. All thesefactsare increasinglyimportantif we considerthata multifacetedevaluationschemerequirestheuseofmultipledatasources,andthatCSCLsettingstendto becomposedby anumberofdifferentandindependenttools.

Severalissuesneedstill to beaddressed.Oneof themostimportantrefersto thefactthattriangulationdependsheavily ontheexpertiseof theresearcher/teacher. A deepknowledgeof the context, a preciseandcarefuldesignof the researchobjectivesandcategories,aswell asa gooduseof thetoolsarerequiredfor a successfulap-proach.Althoughseveralof theseproblemsareclearlyrelatedto naturalisticeval-uationapproachesin general,we aim at providing a morerefinedsetof guidelinesaswell astoolsbasedon techniquesof Artificial Intelligence.

Ongoingresearchdealswith the integrationof this perspective with the existingconstructivist evaluationof DELFOS,whichwill allow usto reflecton theintegra-tion of individual andsocialaspectsof learning.This could leadto considernewSNA techniquesrelatedto therolesof actorsandtheirpositionswithin thenetwork.

Acknowledgements

Theauthorswouldliketo acknowledgethecontributionsof OlgaVelloso,JuditTar-dajos,InmaculadaGarrachon andJoseMarıa Marcos(developmentof SAMSA),RaquelDıazandM. Jose Gallego (setupof the initial experimentalenvironment),Dr. CesarOsuna(DELFOS framework), the rest of the transdisciplinarygroupEMIC, andlastbut not leastall thestudentsthateagerlyparticipatedin theseaca-demicexperiences.Partial financialsupportfor this researchprojectwasgivenbythe AutonomousGovernmentof CastillaandLeon, Spain(VA18/99,VA117/01),the Ministry of ScienceandTechnology, Spainand the EuropeanFundsfor Re-gionalDevelopment(TIC2000-1054)andtheEuropeanUnionLeonardoProgramme.

References

Appelt,W. (1999).WWW BasedCollaborationwith theBSCWSystem.Proceed-ingsof SOFSEM’99,Springer Lecture Notesin ComputerScience1725(p.

16

66-78).Milovy.Borgatti,S.,Everett,M., & Freeman,L. (Eds.).(1999).UCINET5.0Version1.00.

Natick:Analytic Technologies.Chin, G., Rosson,M., & Carroll, J. (1997). Participatoryanalysis:Shareddevel-

opmentof requirementsfrom scenarios.Proceedingsof HumanFactors inComputingSystems,CHI’97 Conference(p. 162-197).N.Y.: ACM.

Cho,H., Stefanone,M., & Gay, G. (2002).SocialInformationSharingin a CSCLCommunity.Proceedingsof ComputerSupportfor CollaborativeLearning,CSCL2002(p. 43-50).Boulder, CO.

Crook,C. (1994). Computers andthecollaborativeexperienceof learning. Lon-don:Routeledge.

Dimitriadis,Y., Martınez,A., Rubia,B., & Gallego,M. (2001).CooperativeLearn-ing in ComputerArchitecture:an educationalprojectandits telematicsup-port. Proceedingsof the Conferenceof Frontiers in Education,FIE 2001.Reno,NE.

W3C (2000).Extensiblemarkuplanguage(XML).(http://www.w3.org/XML/)Gomez,E., Dimitriadis, Y., Rubia,B., & Martınez,A. (2002). Quest,a telem-

atic tool for automaticmanagementof studentquestionnairesin educationalresearch.Proceedingsof 2nd EuropeanConferenceon Technology, Informa-tion, EducationandCitizenship,TIEC 2002.Barcelona.

Guribye, F., & Wasson,B. (2002). The ethnographyof distributed collabora-tive learning.Proceedingsof ComputerSupportfor CollaborativeLearning,CSCL2002(pp.637–638).Boulder, CO.

Martınez,A., Dimitriadis,Y., & dela Fuente,P. (2002).CSCLformativeevaluationsupportby meansof analysisof interactions(in Spanish). ProceedingsofIV SimposioInternacionalde Informatica Educativa,IE 2002. Vigo. (Toappear)

Neale,D., & Carroll, J. (1999). Multi-f acetedevaluationof complex, distributedactivities. Proceedingsof ComputerSupportfor Collaborative Learning,CSCL’99. Stanford,PaloAlto, CA.

Nurmela,K., Lehtinen,E.,& Palonen,T. (1999).EvaluatingCSCLlog filesby So-cial Network Analysis. Proceedingsof ComputerSupportfor CollaborativeLearning, CSCL’99. Stanford,PaloAlto, CA.

Osuna,C.,& Dimitriadis,Y. (1999).A Framework for theDevelopmentof Educa-tional CollaborativeApplicationsbasedon SocialConstructivism. Proceed-ings of CYTEDRITOSInternationalWorkshopon Groupware, CRIWG’99(p. 71-80).Cancun,QuintanaRoo.

QSR (1997). NUD*IST. Software for qualitativedata analysis. ThousandOaks,CA: Scolari.

Stake,R. (Ed.). (1995).Theart of casestudyresearch. ThousandOaks,CA: SAGEPublications,Inc.

Wasserman,S.,& Faust,K. (1994).SocialNetworkAnalysis:MethodsandAppli-cations.Cambridge:CambridgeUniversityPress.

Wilson, B., & Myers,K. (2000). SituatedCognitionin TheoreticalandPracticalContext. In D. Jonassen& S.Land(Eds.),Theoretical foundationsof learn-

17

ing environments(pp.57–88).Mahwah,N.J.:LawrenceErlbaumAssociates,publishers.

18