Using Time-Anchored Peer Comments to Enhance SocialInteraction in Online Educational Videos

Yi-Chieh Lee1, Wen-Chieh Lin1, Fu-Yin Cherng1, Hao-Chuan Wang2, Ching-Ying Sung1,Jung-Tai King1

1National Chiao Tung University, Taiwan, 2National Tsing Hua University, Taiwanbe341341@gmail.com, wclin@cs.nctu.edu.tw, fufu22710@gmail.com, haochuan@cs.nthu.edu.tw,

{jtchin2, eva350053}@gmail.com

ABSTRACTOnline learning is increasingly prevalent as an option for self-learning and as a resource for instructional design. Prere-corded video is currently the main medium of online ed-ucation content delivery and instruction; this affords asyn-chronicity and flexibility, and enables the dissemination oflecture content in a distributed and scalable manner. How-ever, the same properties may impede learners’ engagementdue to the lack of social interaction and peer support. Inthis paper, we propose a time-anchored commenting inter-face to allow online learners who watch the same video clipsto exchange comments on them. Comments left by previ-ous learners at specific time points of a video are displayedto new learners when they watch the same video and reachthose time points. We investigated how the display of time-anchored comments (dynamic or static) and type of com-ments (content-related or social-oriented) influenced users’perceived engagement, perceived social interactivity, andlearning outcomes. Our results show that dynamically dis-playing time-anchored comments can indeed enhance learn-ers’ perceived social interactivity. Moreover, the content ofcomments would further affect learners’ intention of com-menting. Based on our findings, we make various recommen-dations for the improvement of social interaction and learningexperience in online education.

Author KeywordsOnline Educational Interface; E-learning; SynchronousCommunication; Asynchronous Communication.

ACM Classification KeywordsK.3.1. Computer and Education: Computer Uses in Educa-tion, Distance Learning.

INTRODUCTIONOnline learning has become one of the most influential me-dia of instruction. Online courses not only provide a new

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from Permissions@acm.org.CHI 2015, April 18-23 2015, Seoul, Republic of Korea.Copyright c© 2015 ACM 978-1-4503-3145-6/15/04..$15.00.http://dx.doi.org/10.1145/2702123.2702349

approach for self-education but also show potential in trans-forming education, such as the introduction of flipped class-rooms, where students are expected to learn from a combi-nation of prerecorded video lectures and in-class non-lectureactivities.

Online course platforms such as Massive Open OnlineCourses (MOOCs) and Open Course Ware (OCW) are nowwidely used, with the proliferation of diverse curricula at-tracting hundreds of thousands of learners to join them. On-line educational platforms provide course materials to learn-ers using a variety of methods. However, prerecorded onlinevideos appear to be the universal main medium for contentand instruction delivery across all online learning systems.

While it is widely agreed that the quality of video content isan important factor in the quality of online education, exist-ing analytics seem to suggest that videos with high-qualitycontent alone are insufficient to ensure this. Some studies[8, 10] have explored online learners’ learning outcome andengagement with videos in MOOCs [15], but there is still lim-ited clue showing that videos alone are the only best designchoice in the design space of online education. In fact, pre-vious studies indicated that many online learners reported thesense of isolation and that can hinder their ability and perfor-mance of learning [21]. It is especially noteworthy that thereare seldom works focusing on supporting social interaction inonline education.

Social interaction, such as asking and answering questions orcollaboration among peers, may have not only important cog-nitive consequences, but also the potential to enhance learn-ers’ engagement with online courses. Previous work [10, 15]has indicated that engagement is one of the most importantcomponents of effective learning; Dixson [10] further sug-gests that rich communication using multi-modal interactionsmay be related to higher engagement, with learners spendingmore time on the topics at hand and being more motivatedto learn about them [1, 27]. Moreover, highly interactivecommunications provide online course attendees with oppor-tunities to practice different aspects of content processing andlearning, such as remembering and retrieving the course con-tent, which could also help online course attendees to con-struct their knowledge more efficiently [1]. Although there isa consensus in the learning science literature that social inter-actions are crucial to meaningful and productive learning [1,

27], it remains unclear how to effectively support and lever-age social interactions to enhance online learning.

Existing designs for communication in online education sys-tems can be categorized into two distinct types: synchronousand asynchronous. Asynchronous communication, such asthe discussion forums that are often integrated with on-line course platforms, can better afford in-depth and morethoughtful discussion [20]; however, prior research [4, 20]also indicated some limitations of asynchronous designs, e.g.,a lack of immediate feedback, and learners’ perception of so-cial disconnection etc. Huang et al. [18] indicated there arevery few learners engaged in forums, and only about 5% ofregistered users contribute to the discussions in forums. Syn-chronous communication such as live chat affords immediateinformation exchange and discussion, with learners typicallyproducing a greater volume of messages [20]; social pres-ence is also much better supported by this mode of communi-cation [26]. However, synchronous communication also hasvarious drawbacks. Two major constraints, in the context ofMOOCs, are that it can be technically very difficult to me-diate and support large-scale conversations, and that onlinelearners may not have enough time for deep reflection wheninteracting with others via synchronous communication [4].Besides these technical difficulties, the global learner pop-ulation from various time zones is also a challenge to syn-chronous communication.

To address these problems, we propose a design that allowsshared commenting and annotation of video content, namelytime-anchored commenting to support comment exchangeand social interaction for online course videos. The time-anchored commenting design combines certain characteris-tics of both synchronous and asynchronous communicationtechnologies. It partially resembles asynchronous discussionforums, in that users can leave their comments when watch-ing a course video, but in addition to its content, each com-ment is time-stamped with the corresponding playing time ofthe video that users were watching. In other words, the pro-posed design will explicitly connect or anchor the content ofthe comment to the content of the video along its timeline.As a result, when other learners subsequently watch the samecourse material, existing comments can be displayed dynami-cally when the playback time matches the time of annotation.In this way, the system provides learners a way to exchangeinformation and interact with previous learners around thevideo content, despite that learners reading and writing thecomments are neither co-present nor watching the video atthe same time.

Previous work [12, 14] in a different context also proposedsimilar techniques. The time-anchored design can be used tosynchronize comments around a specific event in the video,allowing audiences to follow it more easily. However, similartechniques have mostly been evaluated in the context of digi-tal entertainment and there is still a gap in our understandingof how these techniques might influence online learning.

This paper intends to make three contributions: (1) To build aprototype system for time-anchored commenting and demon-strate how it can be integrated with online educational videos;

(2) To conduct a lab experiment to determine whether andhow the time-anchored design influences learners’ perceivedengagement, perceived social interactivity, and learning out-comes in the context of online learning; and (3) To exam-ine the influence of comment type (content-related or social-oriented) on users’ learning and interactions, by performingan initial content analysis of comments.

RELATED WORKSThe number of users of online courses has increased dramat-ically in recent years. Some online course platforms evenprovide verified certificates to learners who fulfill course re-quirements; as such, online learning has become a signifi-cant instructional avenue. Online course platforms supplyvarious curricula and are intended to improve users’ learn-ing outcomes and engagement [15]. One of the importantissues confronting online learning is social interaction, in par-ticular learner-learner social interaction, which has been un-derstated. Vygotsky [29] has implied that learners can worktogether to create knowledge collaboratively and support oneanother’s learning, while Johnson [20] points out that learner-learner or learner-instructor interactions are prerequisites forcourse satisfaction. Currently, online learning platforms’ ma-jor social-interaction approaches comprise discussion forums(asynchronous) and chatrooms (synchronous). We discusstheir advantages and disadvantages in the following sections.

Asynchronous CommunicationThe effects of the use of asynchronous discussion in onlinelearning have been investigated extensively [4, 8, 26, 28, 16].Some studies [19, 22] have compared face-to-face discussionwith online discussion to examine the latter’s effectivenessas an aid to learning, and the results indicate that online dis-cussion could be superior to face-to-face discussion in somecases. In addition, Koory [22] found that learners reportedhigher levels of course satisfaction when online discussionswere used, and that the text-based communication in onlinecourses reinforced learners’ literacy skills. Other previousstudies [8, 2] also found that integrating a reputation systemwith MOOC forum could increase the forum engagement [2]or induce more responses, but it had no impact on the users’sense of community [8]. A number of studies [19, 20] statethat asynchronous communication in online courses enrichesusers’ learning experiences [30] and increases the number ofcomments with high-quality contents. However, a clear limi-tation of asynchronous discussion is its lack of immediate so-cial interaction, with one prior study [20] noting specificallythat asynchronous postings were slow and emotionless.

Synchronous CommunicationChatrooms are less frequently adopted in online courses thandiscussion forums, because the former are seen as essentiallyrecreational, and therefore likely to distract users from theirlearning tasks [6]. Nevertheless, Coetzee et al. [7] comparedthe influence of chat-tab interfaces (dedicated pages for chat)with that of embedded-chat interfaces (chatrooms placed inevery pages) on a MOOC platform. They found no evidencethat chat adversely impacts online learning. Other research[11, 20] has suggested that synchronous online discussion candevelop a sense of community among learners.

Figure 1. The interface of niconico

Conversation is one of life’s enjoyable experiences [31],and integrating real-time communication with online video-watching can make users feel they are together despite beingphysically dispersed. Weisz et al. [31] found that chat duringvideo viewing has a positive influence on social relationshipsand mutual engagement. However, real-time chat encoun-ters problems when the number of active users becomes large[17]. Hamilton et al.’s study [17] using live streams of videoindicated that users can become frustrated due to the difficultyof interacting with an over-abundance of streams. A similarproblem also exists in MOOCs [20, 4].

Other Interaction Methods for Online VideosOnline courses attract large numbers of learners every day[23]. This characteristic can help teachers improve the qual-ity of online course videos as well as teaching strategies, andlearners can potentially obtain help from other learners online[9]. Cross et al. [9] proposed an online platform to help im-prove online courses’ presentation quality, operating on simi-lar lines to Wikipedia. Some other recent researches [12, 14]proposed annotation systems to enrich online videos, eitherby annotating specific object [12] or text-based comments[14] on a video, thus helping users share their comments withother audience members and motivating social interaction.

The video-sharing website niconico allows users to leave text-based comments in online videos (Figure 1). An importantfeature of niconico is that users’ comments will be dynami-cally displayed on videos. These comments are shown on thetop half of each video according to the playing time. This in-terface design enables the user to easily realize which eventsin the video have been focused on by other viewers, and thushas the potential to enhance social interaction.

Online learning videos can be accessed worldwide, whichmakes it relatively convenient for learners to revisit specificparts of an online course, and to receive feedback from otherlearners. Nevertheless, the methods of learner-learner inter-action in current online learning are probably too limited tobe effectively exploited. Inspired by niconico, this paper in-troduces a time-anchored commenting method and examinesits effect on a selection of online learning videos. Given asufficiently strong connection with the video content, we an-ticipate that this approach to learner-learner communicationwill bring significant benefits to online learning.

Although both niconico and our method align comments withthe video time, we do not directly embed/overlap comments

in the lecture video like niconico does. We display commentsbeside the video (Figure 2CD). Our design could avoid dis-tracting the learners. Besides, the display style in niconicois also more diversified and sometimes exaggerated. Besidesintroducing a new interaction design for online educationalvideos, this paper focuses on exploring the influences of time-anchored commenting on learners. As niconico has neverbeen applied to online educational videos, our study is criticalfor understanding the impact of time-anchored commentingin this domain.

RESEARCH HYPOTHESESTo explore the effects of the integration of time-anchoredcommenting into online course videos, we developed a com-munication platform broadly similar to the annotation sys-tems in niconico. We then conducted an experiment andface-to-face interviews to investigate how learners interactedwith others on this new platform, and measured its influenceon perceived engagement, perceived social interactivity, andlearning outcomes.

Engagement and social interaction play important roles inany online learning environment, motivating learners to keeplearning online and engaging them with course content.As time-anchored commenting directly connects learner-generated comments with video course content, it seemslikely that peers’ comments could influence a learner’s per-spective and social interactions. Due to social interactions,learners will probably leave not only content-related com-ments but also social-oriented comments such as off-topicchats and jokes. However, as the number of peer-generated,content-related comments increases, we expect that onlinelearners would become more engaged in the content of theircourse. Therefore, our first research hypothesis is:

H1. Time-anchored commenting will enhance online learn-ers’ perceived engagement with their course, and this effectwill be more marked as the number of content-related com-ments increases.

As we mentioned that one of important capabilities of time-anchored commenting is dynamically displaying users’ com-ments, thus, how to display the collected comments is a cru-cial issue. If comments left by previous learners are dis-played dynamically when the current playback time matchesthe playback time at which they were added, it will resultin an immediate impact on how learners interact with oth-ers’ feedback. Indeed, this may make learners feel as if theyare viewing the video synchronously. Consequently, we hy-pothesize that dynamically displaying peers’ time-anchoredcomments would motivate (perceived) social interactivity andprobably encourage learners to write comments in the sameclip. Thus, our next two hypotheses are:

H2. Dynamically displaying time-anchored comments willenhance learners perceived social interactivity.

H3. Dynamically displaying time-anchored comments willencourage learners to leave more comments.

Finally, when introducing any new technology into learningenvironments, it is essential to assess its effect on learning

Figure 2. The interface of our system used in the experiment.

outcomes. As comments left by previous learners may in-clude both content-related and social-oriented comments, weassume that content-related comments would provide subse-quent learners with useful supplementary ideas and informa-tion that may improve their learning outcomes. Our final re-search hypothesis is therefore,

H4. Content-related comments will help achieve better learn-ing outcomes in online courses than social-oriented com-ments.

METHOD

Interface DeploymentWe embedded an online video player in the interface (Fig-ure 2A). Participants could use the video player to control thecourse videos, and press the ’CREATE COMMENT’ buttonshown in Figure 2A to leave their comments. When the but-ton was pressed, the course video would be paused automat-ically and the participants were allowed to leave their com-ments (Figure 2B). The system also recorded the timestampof the video. After the participants pressed the ’CREATE’button shown in Figure 2B, or ’Enter’ on the keyboard, theircomments would be added to the area shown in Figure 2D,which statically displayed all the time-anchored commentscollected from other participants. If participants wanted tobrowse other comments, they could scroll the bar manually.Figure 2C indicates the area that showed all time-anchoredcomments dynamically along with the playing of the video.

Experimental Design

Stage AThe main purpose of this stage was to collect learners’ com-ments. Although many comments can be found on onlinelearning websites, these sources were not deemed appropri-ate for our experiments, because few, if any, online coursesoffered to date have recorded users’ comments using a time-anchored method. Thus, it was necessary as part of our ex-periment to collect users’ comments in a time-anchored way.

We selected three online courses from online learning web-sites. In order to reduce the subject bias, we chose them fromthree different fields: neural science, economics, and philos-ophy. Each offered a basic curriculum aimed at undergradu-ates or novices; this made it less likely that participants wouldabandon their courses due to the difficulty of the subject mat-ter. We chose a video clip, with a length of approximately 15minutes, from each course video. The neural science video”Reward and Addiction” was obtained from the OCW of Na-tional Tsing Hua University; The economics video ”Introduc-tion to Economics” was obtained from the OCW of NationalChiao Tung University; The philosophy video ”Understand-ing the Greek Philosophy” was from the Coursera at NationalTaiwan University.

Participants enrolled in Stage A watched all three video clipsand were allowed to leave comments in our system. Werecorded the comments and their corresponding time stampsin the videos. Each participant’s comments were added to thedatabase immediately, so that at any given point in the exper-iment, all comments made by previous participants could beseen by those who followed them.

Stage BIn this stage we examined two factors, display method andcomment type, to investigate our research hypotheses. Dy-namically displaying users’ comments is an important ca-pability of time-anchored commenting, thus, we examinedtwo display methods in our experiments. The first was time-anchored dynamic display, in which each comment appearedin Figure 2C at the time its time stamp indicated until the fol-lowing comments appeared. The second method (Figure 2D)displayed the same time-anchored comments, but the partici-pants had to scroll down the bar and correlate the comments’timing with the video themselves.

In order to explore how the characteristics of comments in-fluenced users’ learning and social behaviors, we categorizedthe comments collected in Stage A into two groups, content-related and social-oriented comments. The former group con-sisted of those comments deemed relevant to the curriculumcontent of the course videos. For example, learners mightsummarize what they saw as the important parts of coursevideos, leave their opinions of the course content, or ask ques-tions about it, such as ”Law of demand: There is an inverserelationship between quantity demand and its price.” (eco-nomics course); ”According to these three properties, I thinkthat Pre-Socratic philosophy was concerned with the featuresof objects on earth.” (philosophy course); ”Was bipolar dis-order caused by an excess of dopamine? I thought it wouldmake people feel happier.” (neural science course).

In contrast, comments comprising jokes, off-topic conver-sation, and other matters irrelevant to the curriculum wereclassified as social-oriented comments. For example, ”It’sa lively description which is easy to remember! However, Istill got a bad score on the test. XDD” (economics course);”The teacher could speak slowly. He usually makes slips oftongue!” (neural science course)

Based on the two factors investigated in this work, we probed

2 (dynamic or static) × 2 (content-related or social-oriented)experimental conditions in this stage. We added an additionalcondition, which did not include any comments, as the base-line. Consequently, there were a total of five experimentalconditions (C0-C4) in Stage B:

Condition 0 (C0): Baseline, No Comments. This condi-tion was a control group, so there were no previous users’comments shown on the web page. However, the participantscould still leave their own comments as they watched the on-line course content.

Condition 1 (C1): Dynamic and Content-related Com-ments. This condition primarily included comments fromthe content-related group. To avoid making the participantsfeel the comments were unnatural, we manipulated the pro-portion of comments of each type, as 80% content-relatedand 20% social-oriented. The system dynamically displayedthese time-anchored comments as the video was played.

Condition 2 (C2): Dynamic and Social-oriented Com-ments. The interface and the method used to display thecomments in this condition were the same as those of C1.However, the proportions of the two comment types were re-versed, with content-related comments making up 20% of thetotal and social-oriented comments, 80%.

Condition 3 (C3): Static and Content-related Comments.In this condition, we removed the function of the time-anchored dynamic display. The participants could see eachcomment as well as the time-stamp relating it to the video,but they needed to map the time of each comment themselvesvis-a-vis the videos. Content-related comments accounted for80% of the total, and social-oriented comments for 20% inthis condition.

Condition 4 (C4): Static and Social-oriented Comments.The methods for displaying the comments in this conditionwere the same as those of C3, but the proportion of content-related comments was 20% and of social-oriented commentswas 80% in this condition.

ParticipantsStage AA total of 50 participants (21 females) were involved in theexperiment in Stage A. All participants were undergraduateor graduate students (M=20.6 years old). In this stage, partic-ipants enrolled in the three selected online courses, and wererequired to complete all of them.

Stage BThere were 52 participants (16 females) who participated instage B. All of them were undergraduate or graduate students(M=21.3 years old). None of them had taken part in Stage A.In this stage, all participants took part in the experiments inour laboratory and were interviewed afterwards.

ProcedureIn Stage A, participants were required to watch a three-minutetutorial video before initiating the main tasks. The video in-troduced the basic rules of the experiment and demonstratedhow to use our system to leave comments on online course

videos. Each participant was given three Internet links forthe three selected online courses and asked to complete all ofthem within 20 days after receiving the links. Participantscould take the same course as many times as they wishedwithin the 20-day time limit. After collecting the participants’comments in Stage A, we invited three experts separately tocategorize them into content-related comments and social-oriented comments, as required by the experimental factorsin stage B. If a comment could not be classified conformably,three experts would discuss to find the category that they wereall agree for the comment. While the comment still could notbe classified, it was discarded from the experiment.

Stage B was a lab-based experiment. Participants needed tocomplete a pretest before watching the online courses. Thepurpose of pretesting was to examine the background knowl-edge of each participant. They were also required to fill inposttests after finishing each online course. The participantswere informed that they would sequentially watch three on-line course videos, and take a survey as well as a posttestat the end of each course video. The three videos were thesame as those in Stage A, and the order in which they werewatched was randomly assigned to each participant. The fiveexperimental conditions described above (C0-C4) were ran-domly deployed with equal probability; in other words, everyparticipant watched three online courses with three differentconditions. We had 24, 34, 33, 33, and 32 participants in C0,C1, C2, C3, and C4, respectively.

We introduced the components of the two different interfacesat the beginning of each course without informing partici-pants that the displayed comments had been edited. Partic-ipants could also leave comments of their own in this stage.The lengths of all course videos were about 15 minutes. Par-ticipants were asked to finish studying and commenting onthe online course videos within 25 minutes, and were allowedto rewind and replay parts of the videos as many times as theywanted within the overall time limit.

MeasureSurveyAfter watching each course video, the participants were re-quired to fill in surveys. Each survey contained a total of 11items, and used 5-point Likert scales (1-strongly disagree, 5-strongly agree). The items measuring perceived engagementwere modified from [24, 25] and the items measuring per-ceived social interactivity were adopted from [13, 32].

Learning OutcomesTo investigate the influence of the different experimental con-ditions on learners’ performance, we used the knowledge-oriented recall test as measures and used pretest and posttestscores to evaluate participants’ learning outcomes. Althoughboth tests were almost the same, the order and wording ofthe questions were modified. We designed the tests based onthe course slides and other course content, such that partici-pants should have been able to answer these questions if theyhad focused on the online material. Some example questionsof test are: ”What is Reward System? Please give a briefexplanation.” (neural science course); ”What is Pre-Socraticphilosophy and its alias?” (philosophy course).

CourseTotal Numberof Comments

Content-related

Social-oriented

Neural Science 430 40% 60%Economics 420 50% 50%Philosophy 427 40% 60%

Table 1. The number of classifiable comments collected in Stage A

C0 C1 C2 C3 C40

1

2

3

4

*2.74

3.45 3.38 3.143.38

Figure 3. The result of perceived engagement. Error bars denote onestandard error.

InterviewTo supplement quantitative analysis, we conducted face-to-face interviews to help us understand each participant’s learn-ing experiences on the online courses. These interviews tookplace after each participant had completed the Stage B exper-iments. Each interview lasted about 20 minutes. Interviewswere audio-recorded and transcribed before analysis. Thequestions asked in the interviews addressed three issues: 1)the influence of others’ comments on the interviewee’s own,2) user experiences with the different interfaces in our sys-tem, and 3) the participants’ past experiences of using onlinelearning websites.

RESULTS

Comment CollectionAfter excluding comments that could not be successfully clas-sified by the three experts, there were total of 1,277 commentsclassified as either social-oriented or content-related (Table1), as previously explained; and these two categories wereused in Stage B to explore effects on perceived engagement,perceived social interactivity, and learning outcomes, as de-tailed in the sections that follow.

Perceived EngagementWe computed a one-way ANOVA with condition (C0-C4) asthe independent variable. The dependent variable was the per-ceived engagement score derived from the survey. The userID was set as the random variable. A Tukey HSD test wasused for post-hoc analysis. Figure 3 shows the results for per-ceived engagement in each condition.

There was a main effect of experimental conditions on per-ceived engagement (F (4, 124.3)=8.16, p<.0001, η2=.20).Furthermore, C0 (M=2.74, SD=.43) had a significantlylower perceived engagement score (p<0.05) than all the otherconditions (C1:M=3.45, SD=.43; C2:M=3.38, SD=.42;C3:M=3.14, SD=.49; C4:M=3.38, SD=.43). Although C3had a notably smaller mean score than C1, C2, and C4, thisdifference was not statistically significant. These findings in-dicate that the time-anchored commenting system we devised

55

DY ST0

1

2

3

4

5

Figure 4. The result of perceived social interactivity

NO RE URE0%

20%

40%

60%

80%

DIS NO ST0%

20%

40%

60%

80%

100%

Mea

n Sc

ore

Pos

ttes

t * *p<.05

Bas

elin

e,

No

Com

men

t

Soci

al

Con

tent

67.8%

Stat

ic

52.4%66.8%

Dyn

amic

Bas

elin

e,

No

Com

men

t

p<.05

Display Type

52.4%68.3% 66.3%

Comment Type

( )

C0 C1 C2 C3 C40%

20%

40%

60%

80%

100%

Mea

n Sc

ore

Pos

ttes

t

*p<.001

Bas

elin

e,

No

Com

men

t

Dyn

amic

&

C

onte

nt

Stat

ic

&

Con

tent

Stat

ic

&

Soci

al

Dyn

amic

&

So

cial

C0 C1 C2 C3 C4

Learning Outcomes

52.4%66.7% 67.0% 70.0%

65.5%

%

Figure 5. The adjusted results of the posttest of learning outcomes

could enhance learners’ perceived engagement with the on-line courses, regardless of the method of display or the typeof comment.

Perceived Social InteractivityBecause no comments were presented in C0 condition, theC0 condition was excluded and a two-way ANOVA (2 typesof display × 2 types of comments) was used to analyze thescores for perceived social interactivity. User ID was set asthe random variable. A Tukey HSD was used to analyze therelationships between the conditions.

All results are shown in Figure 4, where error bars representone standard error. The type of comments was non-significant(F (1, 85.56)=.36, p=.55, η2=.01). However, there was a maineffect of the methods of display (F (1, 128)=12.07, p<.001,η2=.10). These results reflect that the display method in-fluenced participants’ perceived social interactivity. We alsofound that the score of perceived social interactivity in C3(M=3.15, SD=.52) was significantly lower than that in C1(M=3.55, SD=.50, p<.05). In other words, it seems that dy-namic display significantly enhanced the level of perceivedsocial interactivity associated with content-related comments.Although C2 (M=3.59) had a higher mean score than C4(M=3.3), there was no statistically significant difference be-tween them. This implies that, in the case of social-orientedcomments, the effect of display method to the learners’ per-ceived social interactivity would not be as significant as thatfor content-related comments.

Learning OutcomesWe used mixed-model ANCOVA to examine the effects ofdifferent conditions on learning. We used ANCOVA herein order to treat pre-test scores as a covariate and to revealthe real impact of our conditions on post-test scores and thuslearning. To analyze the data, we treated user as a randomvariable, experimental condition and course as fixed vari-ables, pretest score as a covariate, and posttest score as thedependent variable. A Tukey HSD was used to analyze therelationships between the conditions.

The mean accuracy of posttests in each condition is shown inFigure 5. The main effect for condition was non-significant(F (4, 122.2)=2.056, p=.08, η2=.04). Further post-hoc analy-ses using Tukey HSD show that C3 had a significantly betterlearning outcome than C0 (p<.001). We also analyzed theeffects of display method and comment type with the samemethod, respectively. A significant main effect for displaymethod was found (F (2, 128)=3.52, p<.05, η2=.03). Thestatic display method (C3 and C4) resulted in a significantlyhigher posttest score than C0, but the dynamic display method(C1 and C2) did not. In addition, we found a significant maineffect for comment type (F (2, 127.7)=3.30, p<.05, η2=.03).The content-related comments (C1 and C3) led to signifi-cantly better posttest score than C0, but no significant differ-ence was found between social-oriented comments (C2 andC4) and C0. These findings suggest that statically displayingprior users’ content-related comments might enhance learn-ing outcomes.

Effect of CommentsTo further explore the effects of comment types, we inves-tigated the impact of comments from the qualitative data.More than half (51%) of the participants reported that theypreferred the content-related comments when learning on ourplatform, while 42% of them liked the social-oriented com-ments more. The remaining 7% of participants did not likeany type of comment to be displayed when they were learn-ing. Two of the interviewees who preferred the content-related comments described their experience as follows:

”Some users noted the important or complex contents of thecourses, and I can read these notes after watching the coursevideo.” (S2)

”There were several technical terms in the neural sciencecourse and philosophy course. I noticed that some learn-ers kindly commented the detailed information about thoseterms.” (S23)

A participant who liked the social-oriented comments re-ported his experience as follows:

”There are lots of interesting comments in the video. I’d liketo read and respond to those comments when I feel somethingis boring in the courses. And I thought those interesting com-ments can help me revive.” (S15)

Finally, those participants who did not like any comments hada fairly consistent reason:

P<.005P<.05

Co C1 C2 C3 C4

Total comments Course-Related Course-UnrelaTotal Comment Content-related Social-oriented

Dynamic & Unrelated

Dynamic & Related

Static & Related

Static & Unrelated

Baseline , No Comment

Baseline ,No Comment

Dynamic &Social

Dynamic &Content

Static &Content

Static &Social

Figure 6. The quantity of comments in five different experimental condi-tions. The unit is the average number of comments for each participantin different conditions

”Reading those comments with a course video would distractme, and I thought I am used to focusing on the courses.” (S43)

According to the interviews, the 93% of participants who pre-ferred displaying the comments wanted to find facilitating in-formation or interesting comments to help them engage withthe video courses. In contrast, those participants who did notlike the online comments thought that these would distractthem from learning the content of the video itself.

Quantity of Comments in Stage BFigure 6 shows the average number of comments posted byeach participant in each experimental condition, and indi-cates that C2 and C4 yielded the largest number of comments,trailed by C3 and C1, and lastly by C0. This result indicatesthat the participants would not necessarily leave more com-ments in the conditions that feature dynamic display.

This finding could be explained by the participants’ responsesto the two of the questions that we summarized from the in-terviews. The first was ”Do you perceive more social inter-action with others when their comments are replayed dynam-ically?”, to which 80% of participants reported ”Yes”, and20% said ”No”. The second question was ”Does the dy-namic display motivate you to leave your own comments?”,to which 37% of participants said ”Yes”, while the other 63%said ”No”. One of interviewees made the following remark:

”The factor that motivated me to leave a comment is the con-tents of comments such as questions and interesting topics.”(S18)

A one-way ANOVA was performed and the result indi-cated a significant main effect for experimental condition(F (4, 109.77)=5.155, p<.005, η2=.06). As can be seen fromFigure 6, C4 yielded more comments than C3 (p<.05), whileC2 generated more comments than C1 (p<.005). Therefore,it appears that social-oriented comments tended to increaselearners’ intentions to leave comments. It can also be seenthat approximately half of the comments were content-relatedin C1 and C3, whereas in C2 (p<.005) and C4 (p<.05),social-oriented comments outnumbered content-related com-ments by approximately double. We can also observe fromFigure 6 that the quantity of content-related comments wasabout the same in four conditions (C1, C2, C3, and C4).

These findings imply that social-oriented comments mightencourage subsequent participants to leave additional social-oriented comments of their own.

DISCUSSIONSWe summarize the findings and contributions of the study,and then discuss the implications to design and research aswell as the limitations of the current study as following.

Summary of FindingsFrom the results of the study, we confirmed that onlinelearners experienced higher perceived engagement and leftmore comments when prior learners’ time-anchored com-ments were displayed (H1 supported). This matched the find-ings in previous studies [3, 5] suggesting that peers could sup-ply excellent feedback, and demonstrating that the contentof this feedback could prompt learners to actively engage inlearning and reflect on their learning processes. In addition,dynamically displaying the time-anchored comments tendedto make learners perceive more social interactivity, as com-pared to static display of the comments; and this differencewas especially salient in the case of content-related comments(H2 supported).

Also, dynamically displaying the time-anchored commentsmight not always encourage learners to leave more commentsthan displaying them statically (H3 rejected). This findingindicated that higher levels of perceived social interactivitydo not imply that users would definitely or necessarily engagemore in commenting behavior. It is necessary to treat theperception of social interactivity and commenting behaviorseparately, which will need further analyses in the future tounderstand the underlying mechanisms.

In terms of learning achievement, we found that content-related comments could enhance learners’ learning outcomesrelative to the baseline, but did not reach a significantly bet-ter level than social-oriented comments (H4 rejected). Ourresults also indicated that time-anchored commenting did notdegrade learning outcomes when compared with the baselinecondition (in which new comments could be made, but noprior users’ comments were displayed). These findings ap-peared to be consistent with prior research [7] that suggestedembedding chatrooms in online course material would not ad-versely affect learning outcomes. Moreover, users’ learningoutcomes were actually enhanced under the condition of stati-cally displayed content-related comments. This is in keepingwith a prior finding [5] that collaborative experiences helplearners understand complex systems and concepts. Learnersare given the opportunity to interact with the comments bothto gain information they need, and to receive guidance thatmay improve their learning and reasoning.

We also noted that social-oriented comments could stimulatelearners to leave more comments while maintaining the samelevel of participation in content-related discussion. Figure6 shows that all of the experimental conditions with time-anchored comments (C1-C4) collected more learner com-ments than the baseline condition (C0). This suggests that

time-anchored commenting in general could encourage learn-ers to participate in more course-related discussions. More-over, we found that the numbers of content-related com-ments in the content-related conditions (C1 and C3) andsocial-oriented conditions (C2 and C4) were similar. Thisis a particularly interesting finding because one might thinkthat social-oriented comments could distract learners, and/orprompt them to leave fewer content-related comments. How-ever, we found that although the proportion of social-orientedcomments increased in the conditions with social-orientedcomments, the learners still left a similar amount of content-related comments in these conditions.

Design ImplicationsTime-anchored commenting retains the flexibility of asyn-chronous communication that is commonly adopted in onlineeducation (e.g., web forum), while at the same time increasesindirect information exchange and interactivity between peerlearners. Although similar techniques have been used in thedesign of niconico and other video streaming sites, the use ofthe method in the context of online education for enhancingengagement and interactivity is still limited. Our study repre-sents an initial step toward this direction. Below we present anumber of design implications for instructors and researchersinterested in incorporating time-anchored commenting in on-line courses and education.

1. Dynamic display tends to work better than statisticdisplay. Our findings indicate that neither a dynamic nora static display method would decrease participants’ learn-ing outcomes, but participants perceived more social inter-activity when the comments were dynamically displayed.Hence, we suggest that designers consider displaying time-anchored comments dynamically to improve learners’ per-ceived social interactivity; however, it is also beneficial toconsider individual differences in preference and learningstyle, and provide different display options for them, sincesome interviewees reported feeling slightly annoyed whencomments were actively shown on the screen. Furthermore,the dynamic display method could particularly enhance per-ceived interactivity of content-related comments. Becausethe content-related comments are usually more intense thansocial-oriented comments and are often time-sensitive, show-ing these time-anchored comments at the correct time pointscould be more beneficial to the learners.

2. Social-oriented comments are desirable. If there aremore social-oriented comments to start with, our results showthat the number of new social-oriented comments would in-crease but new content-related comments would not be inhib-ited. It should be borne in mind, however, that content-relatedcomments could provide supplemental information for learn-ers and probably enhance their learning outcomes. There-fore, depending on the purposes of the online courses, instruc-tors may find it useful to modulate the proportion of content-related and social-oriented comments to channel the directionof learners’ discussions online. If we could further detectlearners’ status in an online course, this method could alsobe used to automatically modulate the constitution of com-ments: for example, we may choose to decrease the propor-

tion of content-related comments relative to social-orientedcomments if we detect that learners’ perceived social interac-tivity or perceived engagement is falling.

3. Video-centered, time-anchored comment exchangesupports collaborative learning. The biggest difference be-tween this time-anchored design and regular chatrooms is theconnection with video. Learners can leave their commentsat the exact moment the course content confuses them, mak-ing it easier for peer learners to help them. Several inter-viewees mentioned in the post-study interviews that seeingother learners noting the important or complex contents ofthe online courses helped their own learning. The design al-lows learners to use a large body of auxiliary messages gener-ated by the learning community to understand difficult infor-mation and form collaborative relationships with their peers.There is a potential to extend the time-anchored commentingmethod as a full-fledged design for large-scale collaborativelearning in the future.

LIMITATIONS AND FUTURE WORKThis study has some limitations. First, our experiments con-ducted in a lab setting cannot account for contextual factors,which are certainly important for the research of online learn-ing, e.g., characteristics of learners, number of learners, andlength of study. The current results are more adequate forapplications of online learning videos where the number oflearners watching the same video and characteristics of thelearners would approximately match the structure of the cur-rent study, such as flipped classrooms.

As a lab study, the number of sample learners examinedis smaller than the number of learners present in a regularMOOC course. The current study is intended to explore theutilities of the time-anchored peer commenting mechanism,and thus it is reasonable to test our hypotheses with the ex-perimental design and the sample size. In addition, the im-pact of the design on learning was evaluated in a short periodof time. We also need a solution to deal with hundreds orthousands of comments present in real MOOCs, which mayinvolve automatic analyses of language content and personal-ized sampling of comments.

To generalize our findings and to increase the external valid-ity of the time-anchored method, it would be complementaryto scale up the work, conducting deployment studies to un-derstand the impact of the method on actual online learners,especially given that the current study has obtained promisingresults.

Second, our findings suggest that dynamically displayingtime-anchored comments can enhance learners’ perceivedengagement and perceived social interactivity. The designtherefore may be able to improve the completion rates of on-line courses, which is known to be one major challenge toMOOCs. A future deployment study with real usage data willhelp verify this possible benefit.

Third, in terms of data analytics, we decided to treat thecourse as a fixed effect in the linear model analysis of learn-ing outcomes. The decision of treating the course either asa random or fixed effect will lead to different interpretations.

Because we plan to replicate the study with other courses inthe future, we see the feasibility to treat course as a fixed ef-fect, and replications will help generalize the results.

Fourth, instructors also play an important role in the learn-ing process and social interaction, regardless of whether thecourse is delivered in face-to-face or online settings. Kizlicecet al. [21] investigated how adding the instructor’s face to alearning video affects learners. They found whether show-ing or hiding the instructor’s face did not affect recall ability.Our result, on the other hand, shows that utilizing the learn-ers’ social-oriented comments could increase their learningoutcomes. It is worth further exploring the effects of differenttypes of social interactions and social cues. The contributionsthat may be made by instructors are also worth considering infuture work.

CONCLUSIONIn this study, we have presented a time-anchored comment-ing method for online educational videos, along with the re-sults of an experiment we conducted to investigate the im-pact of this method on learners. We focused on two mainfactors, display methods and comment types, and their re-spective influences on learners’ perceived engagement, per-ceived social interactivity, and learning outcomes in video-based learning. Based on the findings of the experiments, wefound that dynamically displaying time-anchored commentsenhanced learners’ perceived social interactivity. Moreover,previous learners’ comments had positive influences on laterlearners’ perceived engagement. We hope this work providessome new perspectives that will help improve social interac-tion and learning experiences in online video-based educa-tion.

ACKNOWLEDGMENTSWe thank the anonymous reviewers for their insightful com-ments. This work was supported in part by Taiwan Ministryof Science and Technology (MOST) under grants 101-2628-E-009-021-MY3, 102-2221-E-009-082-MY3, 102-2221-E-007-073-MY3, and UST-UCSD International Center of Ex-cellence in Advanced Bioengineering sponsored by TaiwanMOST I-RiCE Program under grant 103-2911-I-009-101.

REFERENCES1. Abrami, P. C., Bernard, R. M., Bures, E. M.,

Borokhovski, E., and Tamim, R. M. Interaction indistance education and online learning: Using evidenceand theory to improve practice. Journal of Computing inHigher Education 23, 2-3 (2011), 82–103.

2. Anderson, A., Huttenlocher, D., Kleinberg, J., andLeskovec, J. Engaging with massive online courses. InWWW’14 (2014), 687–698.

3. Anderson, T., and Elloumi, F. The theory and practice ofonline learning. Athabasca University Press, 2008.

4. Branon, R. F., and Essex, C. Synchronous andasynchronous communication tools in distanceeducation. TechTrends 45, 1 (2001), 36–36.

5. Bransford, J. D., Brown, A. L., and Cocking, R. R. HowPeople Learn: Brain, Mind, Experience, and School.The National Academies Press, 2000.

6. Burnett, C. Learning to chat: Tutor participation insynchronous online chat. Teaching in higher education8, 2 (2003), 247–261.

7. Coetzee, D., Fox, A., Hearst, M. A., and Hartmann, B.Chatrooms in MOOCs: All talk and no action. In L@S(2014).

8. Coetzee, D., Fox, A., Hearst, M. A., and Hartmann, B.Should your MOOC forum use a reputation system? InCSCW (2014), 1176–1187.

9. Cross, A., Bayyapunedi, M., Ravindran, D., Cutrell, E.,and Thies, W. VidWiki: Enabling the crowd to improvethe legibility of online educational videos. In CSCW(2014).

10. Dixson, M. D. Creating effective student engagement inonline courses: What do students find engaging?JoSoTL 10, 2 (2010).

11. Duemer, L., Fontenot, D., Gumfory, K., Kallus, M.,Larsen, J., Schafer, S., and Shaw, B. The use of onlinesynchronous discussion groups to enhance communityformation and professional identity development.Journal of Interactive Online Learning 1, 2 (2002),1–12.

12. Goldman, D. B., Gonterman, C., Curless, B., Salesin,D., and Seitz, S. M. Video object annotation, navigation,and composition. In UIST (2008), 3–12.

13. Granovetter, M. The strength of weak ties: A networktheory revisited. Sociological theory 1 (1983), 201–233.

14. Guimaraes, R. L., Cesar, P., and Bulterman, D. C. Letme comment on your video: Supporting personalizedend-user comments within third-party online videos. InProceedings of the 18th Brazilian symposium onMultimedia and the web (2012), 253–260.

15. Guo, P. J., Kim, J., and Rubin, R. How video productionaffects student engagement: An empirical study of moocvideos. In L@S (2014), 41–50.

16. Guzdial, M., and Turns, J. Effective discussion through acomputer-mediated anchored forum. The journal of thelearning sciences 9, 4 (2000), 437–469.

17. Hamilton, W., Garretson, O., and Kerne, A. Streamingon twitch: Fostering participatory communities of playwithin live mixed media. In CHI (2014).

18. Huang, J., Dasgupta, A., Ghosh, A., Manning, J., andSanders, M. Superposter behavior in mooc forums. InL@S (2014), 117–126.

19. Johnson, G., and Howell, A. Attitude towardinstructional technology following required versusoptional WebCT usage. Journal of Technology andTeacher Education 13, 4 (2005), 643–654.

20. Johnson, G. M. Synchronous and asynchronoustext-based CMC in educational contexts: A review ofrecent research. TechTrends 50, 4 (2006), 46–53.

21. Kizilcec, R. F., Papadopoulos, K., and Sritanyaratana, L.Showing face in video instruction: effects oninformation retention, visual attention, and affect. InCHI (2014).

22. Koory, M. A. Differences in learning outcomes for theonline and F2F versions of ”an introduction toshakespeare”. Journal of Asynchronous LearningNetworks 7, 2 (2003), 18–35.

23. Martin, F. G. Will massive open online courses changehow we teach? Commun. ACM 55, 8 (2012), 26–28.

24. O’Brien, H. L., and Toms, E. G. The development andevaluation of a survey to measure user engagement.Journal of the American Society for Information Scienceand Technology 61, 1 (2010), 50–69.

25. O’Brien, H. L., and Toms, E. G. Is there a universalinstrument for measuring interactive informationretrieval?: the case of the user engagement scale. InProceedings of the third symposium on Informationinteraction in context (2010), 335–340.

26. Oztok, M., Zingaro, D., Brett, C., and Hewitt, J.Exploring asynchronous and synchronous tool use inonline courses. Computers & Education 60, 1 (2013),87–94.

27. Romiszowski, A., and Mason, R. Handbook of researchfor educational communications and technology, 2 ed.Taylor & Francis, 2004, ch. Computer mediatedcommunication, 397–431.

28. van der Pol, J., Admiraal, W., and Simons, P. R.-J. Theaffordance of anchored discussion for the collaborativeprocessing of academic texts. International Journal ofComputer-Supported Collaborative Learning 1, 3(2006), 339–357.

29. Vygotsky, L. S. Mind in society: The development ofhigher psychological processes. Harvard universitypress, 1980.

30. Walker, R., and Arnold, I. Introducing group-basedasynchronous learning to business education. reflectionson effective course design and delivery. Educationalmedia international 41, 3 (2004), 253–265.

31. Weisz, J. D., Kiesler, S., Zhang, H., Ren, Y., Kraut,R. E., and Konstan, J. A. Watching together: integratingtext chat with video. In CHI (2007), 877–886.

32. Zhang, S., Jiang, H., and Carroll, J. M. Integratingonline and offline community through facebook. InIEEE International Conference on CollaborationTechnologies and Systems (2011), 569–578.