Purdue UniversityPurdue e-Pubs

Computer Science Technical Reports Department of Computer Science

2007

Virtual Classroom Extension for Effective DistanceEducationRadu Dondera

Chun Jia

Voicu PopescuPurdue University, popescu@cs.purdue.edu

Cristina Nita-RotaruPurdue University, crisn@cs.purdue.edu

Melissa DarkPurdue University, dark@cs.purdue.edu

Report Number:07-001

This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact epubs@purdue.edu foradditional information.

Dondera, Radu; Jia, Chun; Popescu, Voicu; Nita-Rotaru, Cristina; and Dark, Melissa, "Virtual Classroom Extension for EffectiveDistance Education" (2007). Computer Science Technical Reports. Paper 1666.http://docs.lib.purdue.edu/cstech/1666

VIRTUAL CLASSROOM EXTENSION FOR EFFECTIVE DISTANCE EDUCATION

Radu Dondera Chun Jia

Voicu Popescu Cristina Nita-Rotaru

Melissa Dark Cynthia York

CSD TR #07-001 January 2007

VIRTUAL CLASSROOM EXTENSION FOREFFECTIVE DISTANCE EDUCATION

Radu DonderaChun Jia

Voicu PopescuCristina Nita-Rotaru

Melissa DarkCynthia York

CSD TR #07-001January 2007

Virtual Classroom Extension for Effective Distance Education Radu Dondera, Chun Jia, Voicu Popescu,

Cristina Nita-Rotaru, Melissa Dark, and Cynthia York

Purdue University

Abstract We present the design, implementation, and initial results of a system for remote lecture attendance based on extending on-campus classrooms to accommodate remotely located students. A remote student is modeled with a real-time video sprite. The sprites are integrated into a geometric model that ~rovides a virtual extension of the classroom. The

view of the instructor, who can conveniently get a sense of their body language and of their facial expression. The system has been deployed in a first classroom and a pilot study indicates that the system promises to deliver quality education remotely. The system relies exclusively on commodity components, therefore it can be deployed in any classroom to allow any course to offer distance education seats.

A -

virtual extension is rendered and projected onto the Keywords I.3.6.d Interaction techniques, I.3.7.g back wall of the classroom. The remote students are Virtual reality, I.3.2.a Distributed/network graphics, displayed at a natural location within the field of 1.3.8 Applications.

Figure 1. Distance education system deployed in first classroom (top), and photographs of the back-wall screen showing remote students integrated into a virtual extension of the classroom (botrom).

Virtual Classroom Extension for Effective Distance EducationRadu Dondera, Chun Jia, Voicu Popescu,

Cristina Nita-Rotaru, Melissa Dark, and Cynthia York

Purdue University

AbstractWe present the design, implementation, and initialresults of a system for remote lecture attendancebased on extending on-campus classrooms toaccommodate remotely located students. A remotestudent is modeled with a real-time video sprite. Thesprites are integrated into a geometric model thatprovides a virtual extension of the classroom. Thevirtual extension is rendered and projected onto theback wall of the classroom. The remote students aredisplayed at a natural location within the field of

view of the instructor, who can conveniently get asense of their body language and of their facialexpression. The system has been deployed in a firstclassroom and a pilot study indicates that the systempromises to deliver quality education remotely. Thesystem relies exclusively on commodity components,therefore it can be deployed in any classroom toallow any course to offer distance education seats.

Keywords 1.3.6.d Interaction techniques, I.3.7.gVirtual reality, 1.3.2.a Distributed/network graphics,1.3.8 Applications.

- . - - ---~------~'

,-

~~~~~-~'-_.

;',;~~~~:'-~:-~::::,- .. ,2,."<.>.:_~-. '....;':;'-';~';"; . :-:1

:. i",,",.);;

Figure I. Distance education system deployed in first classroom (top), and photographs of the back-wallscreen showing remote students integrated into a virtual extension of the classroom (bottom).

Figure 2. Hardware components for classroom system.

1. Introduction Distance education services provide access to education for students living in remote areas as well as for working adults, and allow field-authority experts to reach an increasing number of learners. The latest report from the National Center for Educational Statistics indicates that more than half of the higher education institutions in the United States offer distance education services, with enrollment doubling every three years [NCES 20021. However, current distance education systems fail to match the effectiveness of conventional on-campus education, because of several shortcomings.

First, students engaged in distance education (or for short remote students) feel isolated due to lack of interactive communication with the instructor and the on-campus students. Most systems only support asynchronous remote delivery of lectures, and existing synchronous systems provide a low level of interactivity.

Second, remote students do not have access to other proven on-campus education activities such as office hours and study groups.

Finally, the availability of distance education services is limited by the reliance on expensive specialized infrastructure (e.g. teleconferencing-enabled classrooms with attached broadcasting rooms), which requires a large technical support staff. Moreover, the technology places a substantial burden on instructors, who are asked to deviate considerably from their usual approach to educational activity preparation

and delivery. Higher-education institution surveys indicate that cost (program development, equipment acquisition and maintenance), faculty workload, and course quality are factors that prevent to a "major extent" starting or expanding distance education course offerings [NCES 20021.

Research advances in areas such as computer networking, information security, computer graphics, and education science, as well as an almost vertical progress in the performance of commodity audio, video, general computing, networking, and graphics hardware provide new opportunities to reassess the role played by technology in education, and even to develop a new approach to education in general. A debate on whether the current lecture-attendance- based approach to education should be abandoned in favor of a radically new approach, nonetheless interesting, is beyond the goal of this work. Our approach is to leverage technological and research advances in order to increase the effectiveness of distance education to a level comparable to that of conventional on-campus education.

1 .I. Contribution We are developing a distance education system that has unique characteristics that address limitations of current distance education systems. Specifically, our system:

I . relies on an education-science-inclusive design process by integrating at all stages expertise and feedback from education science experts, members of our team;

Figure 2. Hardware components for classroom system.

1. IntroductionDistance education services provide access toeducation for students living in remote areas as wellas for working adults, and allow field-authorityexperts to reach an increasing number of learners.The latest report from the National Center forEducational Statistics indicates that more than half ofthe higher education institutions in the United Statesoffer distance education services, with enrollmentdoubling every three years [NCES 2002]. However,current distance education systems fail to match theeffectiveness of conventional on-campus education,because of several shortcomings.

First, students engaged in distance education (or forshort remote students) feel isolated due to lack ofinteractive communication with the instructor and theon-campus students. Most systems only supportasynchronous remote delivery of lectures, andexisting synchronous systems provide a low level ofinteractivity.

Second, remote students do not have access to otherproven on-campus education activities such as officehours and study groups.

Finally, the availability of distance education servicesis limited by the reliance on expensive specializedinfrastructure (e.g. teleconferencing-enabledclassrooms with attached broadcasting rooms), whichrequires a large technical support staff. Moreover, thetechnology places a substantial burden on instructors,who are asked to deviate considerably from theirusual approach to educational activity preparation

and delivery. Higher-education institution surveysindicate that cost (program development, equipmentacquisition and maintenance), faculty workload, andcourse quality are factors that prevent to a "majorextent" starting or expanding distance educationcourse offerings [NCES 2002].

Research advances in areas such as computernetworking, information security, computer graphics,and education science, as well as an almost verticalprogress in the performance of commodity audio,video, general computing, networking, and graphicshardware provide new opportunities to reassess therole played by technology in education, and even todevelop a new approach to education in general. Adebate on whether the current lecture-attendance­based approach to education should be abandoned infavor of a radically new approach, nonethelessinteresting, is beyond the goal of this work. Ourapproach is to leverage technological and researchadvances in order to increase the effectiveness ofdistance education to a level comparable to that ofconventional on-campus education.

1.1. ContributionWeare developing a distance education system thathas unique characteristics that address limitations ofcurrent distance education systems. Specifically, oursystem:

I. relies on an education-science-inclusive designprocess by integrating at all stages expertise andfeedback from education science experts,members of our team;

2

2. provides transparent extension of on-campus educational activities such as lectures, office hours, and study groups to support distance education, which ensures that remote students are included without significantly altering the preparation or the normal course of the educational activity;

provides support for effective instructor-student and student-student interactions by using a group communication system as communication platform and by integrating the remote students into a unified virtual environment; this enables the instructor to maintain high-quality visual contact with all remote students simultaneously and thereby to be aware of their facial expression, body language, or desire to interject, in real-time;

4. can be deployed in any classroom or ofice by relying exclusively on inexpensive off-the-shelf components, making distance education an integral part of conventional on-campus education rather than a parallel activity that drains considerable resources;

5. is evaluated by assessing performance from both the technical and educational standpoints.

In this paper we report on the design, implementation, and first results of a major component of our distance education system: the remote lecture attendance system. The system has recently been deployed in the first classroom, see top of Figure 1. The operation of the system is illustrated in a video which we ask the reviewers to download from w~~~'.c~.purd~~e.eduic~vlabidl/.

Remote students are integrated into a virtual extension of the classroom, which is projected onto the back wall of the classroom. A remote student is acquired with a webcam and is modeled as a real- time video sprite. The sprite is inserted into the geometric model of the classroom extension. Even though each remote student can potentially be located at a different site, the remote students are integrated into a unified virtual environment, which is displayed at a natural location within the field of view of the instructor. The instructor gets a sense of the body language and facial expression of remote students and sees if a remote student raises hisher hand in real time (see bottom row of Figure 1). Audio is captured continually for each remote student and played back

in the classroom. Instructor audio and video is continually provided to each remote student.

Our system strives to make distance education an integral but unobtrusive part of conventional on- campus education. This is achieved by relying exclusively on commodity components; the only notable piece of additional equipment is the rear- facing projector (Figure 2). Our system can be deployed in any classroom with minimal cost to allow any course to offer distance education seats. Our system does not imply substantial modifications to the way the instructor prepares and delivers the lecture. The instructor requires little or no training in order to be able to use the system effectively. We foresee that our line of work will ultimately lead to campuses abandoning the current approach of maintaining one or a few distance education classrooms with specialized staff and specially trained instructors, and moving towards merging support for distance education with general audio, video, and IT support.

We conducted a preliminary evaluation of the system through a pilot study involving 5 remote and 15 local students. The goal of the pilot was to measure the effectiveness of the design, independently of networking bottlenecks. Pre- and post-test scores reveal no significant difference between remote and on-campus students. The students also completed a comprehensive survey and participated in focus groups. These indicate that the remote students perceived the ability to naturally interact with the instructor as an important strength of the system, that the remote students would have liked to be able to better interact with the on-campus students, and that the on-campus students were little affected by the distance learning aspect of the class.

We discuss prior work next. Section 3 gives an overview of the system, and sections 4 and 5 describe the classroom and remote student subsystems. Section 6 describes the implementation. Section 7 discusses system assessment. Section 8 concludes and sketches directions for future work.

2. Background Several systems have been developed that employ audio, video, and communication technology to deliver education remotely (Section 2.1). However, distance education is far from being a solved

2. provides transparent extension of on-campuseducational activities such as lectures, officehours, and study groups to support distanceeducation, which ensures that remote studentsare included without significantly altering thepreparation or the normal course of theeducational activity;

3. provides support for effective instructor-studentand student-student interactions by using agroup communication system as communicationplatform and by integrating the remote studentsinto a unified virtual environment; this enablesthe instructor to maintain high-quality visualcontact with all remote students simultaneouslyand thereby to be aware of their facialexpression, body language, or desire to interject,in real-time;

4. can be deployed in any classroom or office byrelying exclusively on inexpensive off-the-shelfcomponents, making distance education anintegral part of conventional on-campuseducation rather than a parallel activity thatdrains considerable resources;

5. is evaluated by assessing performance from boththe technical and educational standpoints.

In this paper we report on the design, .implementation, and first results of a majorcomponent of our distance education system: theremote lecture attendance system. The system hasrecently been deployed in the first classroom, see topof Figure I. The operation of the system is illustratedin a video which we ask the reviewers to downloadfrom www.cs.purdue.edu/cgvlab/dl/.

Remote students are integrated into a virtualextension of the classroom, which is projected ontothe back wall of the classroom. A remote student isacquired with a webcam and is modeled as a real­time video sprite. The sprite is inserted into thegeometric model of the classroom extension. Eventhough each remote student can potentially be locatedat a different site, the remote students are integratedinto a unified virtual environment, which is displayed·at a natural location within the field of view of theinstructor. The instructor gets a sense of the bodylanguage and facial expression of remote studentsand sees if a remote student raises hislher hand in realtime (see bottom row of Figure 1). Audio is capturedcontinually for each remote student and played back

in the classroom. Instructor audio and video IS

continually provided to each remote student.

Our system strives to make distance education anintegral but unobtrusive part of conventional on­campus education. This is achieved by relyingexclusively on commodity components; the onlynotable piece of additional equipment is the rear­facing projector (Figure 2). Our system can bedeployed in any classroom with minimal cost toallow any course to offer distance education seats.Our system does not imply substantial modificationsto the way the instructor prepares and delivers thelecture. The instructor requires little or no training inorder to be able to use the system effectively. Weforesee that our line of work will ultimately lead tocampuses abandoning the current approach ofmaintaining one or a few distance educationclassrooms with specialized staff and speciallytrained instructors, and moving towards mergingsupport for distance education with general audio,video, and IT support.

We conducted a preliminary evaluation of the systemthrough a pilot study involving 5 remote and 15 localstudents. The goal of the pilot was to measure theeffectiveness of the design, independently ofnetworking bottlenecks. Pre- and post-test scoresreveal no significant difference between remote andon-campus students. The students also completed acomprehensive survey and participated in focusgroups. These indicate that the remote studentsperceived the ability to naturally interact with theinstructor as an important strength of the system, thatthe remote students would have liked to be able tobetter interact with the on-campus students, and thatthe on-campus students were little affected by thedistance learning aspect of the class.

We discuss prior work next. Section 3 gives anoverview of the system, and sections 4 and 5 describethe classroom and remote student subsystems.Section 6 describes the implementation. Section 7discusses system assessment. Section 8 concludesand sketches directions for future work.

2. BackgroundSeveral systems have been developed that employaudio, video, and communication technology todeliver education remotely (Section 2.1). However,distance education is far from being a solved

3

problem, as attested by education science research (Section 2.2).

2.1. Prior Distance Education Systems Several distance education systems have been implemented to date. A virtual classroom multimedia distance learning system [Deshpande and Hwang 20011 extends streaming and conferencing tools by supporting both unicast and multicast networks, by employing a specialized compression algorithm for handwritten text, by improved synchronization and resource allocation for media streaming, and by providing the ability to record the live classroom sessions. Interactivity is limited by rigid communication rules. A single audio and video feed is transmitted at any time, originating either from the instructor or from a remote student. The instructor must grant permission for students to ask questions, which places a strong limitation on interactivity and keeps the focus on the use of technology rather than on the actual educational activity.

A step toward increased interactivity and realism is taken by the Smart Classroom system [Shi et a1 20021 which provides a "face-to-face" interactive classroom environment by projecting the images of the remote students onto the wall of a real classroom and by allowing the instructor to move freely in the room. Special network technologies for large-scale access and for heterogeneous network architectures are proposed for real time transmission of data between the instructor and the remote students, but no results are given regarding the effectiveness of these technologies. The system achieves limited interactivity since, as with the previous system, only one student can send hisher audio and video at a time and the instructor has to grant permission. The system projects only the image of the remote student that has the token to speak which does not confer the instructor the ability to maintain contact with all remote students simultaneously, in real time.

A system in which audio and video feeds are sent simultaneously from multiple sites is IRI-h [Maly et al. 20011. The distinguishing feature of the system is the "shared view", a window with reconfigurable content that is viewed by all participants. The window displays simultaneously a dynamic subset of the participants. However, each participant is shown in a separate sub-window within the shared window, and the lack of a unified visualization reduces the

effectiveness of the visual communication. Moreover, participants are displayed at variable locations within the virtual environment according to the current subset visualized in the shared view, which precludes taking advantage of seating consistency that is known to be beneficial in conventional on-campus learning.

The Virtual Blackboard [Sagias 20021 is a classroom learning environment that allows the instructor and the remote students to manipulate video, audio, text, whiteboard images and 3D graphics simultaneously. The system aims to provide a wide range of modalities for interaction by granting both the remote students and the instructor the opportunity to participate in an online lesson and at the same time exchange information and share applications. Nevertheless, the degree of interaction is limited in that the remote students and the instructor can only communicate via text, whiteboard, email and shared applications, yielding an unnatural interaction.

2.2. Distance vs. On-Campus Education As distance education has been growing in terms of institutional participation and student enrollment, education science research has been targeted towards assessing current systems and towards drafting guiding principles for putting the effectiveness of distance education at par with that of conventional on-campus education.

Frequently, in distance education remote students do not have the opportunity to interact with other participants directly, or interaction occurs through delayed, asynchronous communication, which precludes in-depth, continuous discussions. Such experiences have been shown to lead to a feeling of alienation and isolation due to the lack of face-to face interaction and support from peers [Galusha 19971. Numerous research studies that experiment with different strategies for augmenting communication, collaboration, participation, and peer feedback in distance education systems have shown that the level of interactivity is a major factor for increasing learning in distance education environments [Lavooy and Newlin 20031.

Distance education systems are deficient regarding both student-to-student and student-to-instructor interaction [Foshay and Bergeron 20001. Student-to- instructor interaction is regarded as essential by many educators and students. Research studies on

problem, as attested by education science research(Section 2.2).

2.1. Prior Distance Education Systems

Several distance education systems have beenimplemented to date. A virtual classroom multimediadistance learning system [Deshpande and Hwang2001] extends streaming and conferencing tools bysupporting both unicast and multicast networks, byemploying a specialized compression algorithm forhandwritten text, by improved synchronization andresource allocation for media streaming, and byproviding the ability to record the live classroomsessions. lnteractivity is limited by rigidcommunication rules. A single audio and video feedis transmitted at any time, originating either from theinstructor or from a remote student. The instructormust grant permission for students to ask questions,which places a strong limitation on interactivity andkeeps the focus on the use of technology rather thanon the actual educational activity.

A step toward increased interactivity and realism istaken by the Smart Classroom system [Shi et al 2002]which provides a "face-to-face" interactive classroomenvironment by projecting the images of the remotestudents onto the wall of a real classroom and byallowing the instructor to move freely in the room.Special network technologies for large-scale accessand for heterogeneous network architectures areproposed for real time transmission of data betweenthe instructor and the remote students, but no resultsare given regarding the effectiveness of thesetechnologies. The system achieves limitedinteractivity since, as with the previous system, onlyone student can send his/her audio and video at atime and the instructor has to grant permission. Thesystem projects only the image of the remote studentthat has the token to speak which does not confer theinstructor the ability to maintain contact with allremote students simultaneously, in real time.

A system in which audio and video feeds are sentsimultaneously from multiple sites is IRI-h [Maly etal. 2001]. The distinguishing feature of the system isthe "shared view", a window with reconfigurablecontent that is viewed by all participants. Thewindow displays simultaneously a dynamic subset ofthe participants. However, each participant is shownin a separate sub-window within the shared window,and the lack of a unified visualization reduces the

effectiveness of the visual communication. Moreover,participants are displayed at variable locations withinthe virtual environment according to the currentsubset visualized in the shared view, which precludestaking advantage of seating consistency that is knownto be beneficial in conventional on-campus learning.

The Virtual Blackboard [Sagias 2002] is a classroomlearning environment that allows the instructor andthe remote students to manipulate video, audio, text,whiteboard images and 3D graphics simultaneously.The system aims to provide a wide range ofmodalities for interaction by granting both the remotestudents and the instructor the opportunity toparticipate in an online lesson and at the same timeexchange information and share applications.Nevertheless, the degree of interaction is limited inthat the remote students and the instructor can onlycommunicate via text, whiteboard, email and sharedapplications, yielding an unnatural interaction.

2.2. Distance vs. On-Campus Education

As distance education has been growing in terms ofinstitutional participation and student enrollment,education science research has been targeted towardsassessing current systems and towards draftingguiding principles for putting the effectiveness ofdistance education at par with that of conventionalon-campus education.

Frequently, in distance education remote students donot have the opportunity to interact with otherparticipants directly, or interaction occurs throughdelayed, asynchronous communication, whichprecludes in-depth, continuous discussions. Suchexperiences have been shown to lead to a feeling ofalienation and isolation due to the lack of face-to faceinteraction and support from peers [Galusha 1997].Numerous research studies that experiment withdifferent strategies for augmenting communication,collaboration, participation, and peer feedback indistance education systems have shown that the levelof interactivity is a major factor for increasinglearning in distance education environments [Lavooyand Newlin 2003].

Distance education systems are deficient regardingboth student-to-student and student-to-instructorinteraction [Foshay and Bergeron 2000]. Student-to­instructor interaction is regarded as essential by manyeducators and students. Research studies on

4

udent Video 1

Remote Student Video n Classroom Video Remote Student Video 1

' Classroom Video Remote Student Video n

I Network I

Figure 3. System architecture.

interactivity reveal that students have a real need to make genuine connections with their instructor. Interaction between students and instructors helps ensure the students and instructor develop a feeling of community and connectedness to the course [Lavooy and Newlin 20031.

Distance education systems currently fail to fully engage students because they do not provide a sense of cognitive, social, and physical presence. Cognitive presence refers to having one's cognitive processes focused toward a context or task(s) with one's attention on related stimuli. Social presence, in one aspect, "implies that being with someone virtually feels like being with them physically" [Heeter 20031. "Physical presence implies being present in (or present to) the virtual or real environment: being there" [Heeter 20031. A major dimension of presence is interactivity; for example, increased levels of interaction provide more stimuli, which in turn promote a sense of presence.

and the instructor feel that the remote students are actually present in the classroom, which should be achieved through high-level of interactivity and visual realism of the virtual environment hosting the remote students.

3. System Architecture In this section we provide an overview of our system for remote lecture attendance. The system has two main components: a Classroom System that is deployed at the on-campus classroom and a Remote Student System that is deployed for each remotely located student. Figure 3 presents a deployment with n remote students.

The Classroom System extends the classroom to accommodate remote students in a way transparent to the instructor and local students. The image of a virtual 3D room is projected onto the back wall of the classroom to provide additional seats for the remote students. The instructor interacts with the remote

We conclude that distance education systems can be students naturally, much the same way shehe more effective (increase learning and more likely to interacts with students physically present in the motivate students) if they make the remote students classroom. The Classroom System communicates

Groups:• Audio All

Groups: Groups:... Classroom Video• Remote Student Video 1 • Audio All • Audio All

... Classroom Video ... Classroom Video• Remote Student Video n • Remote Student Video 1 • Remote Student Video n

~ ~ ~Classroom Remote Student Remote Student

SYstem System 1 ••• System n

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~A V A V A V

~,

~, .,, ,, ,,

~,

I GCS Server I GCS Server I GCS Server

~ ~ ~ ~ A~, .,, 9

Network I

Figure 3. System architecture.

interactivity reveal that students have a real need tomake genuine connections with their instructor.Interaction between students and instructors helpsensure the students and instructor develop a feelingof community and connectedness to the course[Lavooy and Newlin 2003].

Distance education systems currently fail to fullyengage students because they do not provide a senseof cognitive, social, and physical presence. Cognitivepresence refers to having one's cognitive processesfocused toward a context or task(s) with one'sattention on related stimuli. Social presence, in oneaspect, "implies that being with someone virtuallyfeels like being with them physically" [Heeter 2003]."Physical presence implies being present in (orpresent to) the virtual or real environment: beingthere" [Heeter 2003]. A major dimension of presenceis interactivity; for example, increased levels ofinteraction provide more stimuli, which in tumpromote a sense ofpresence.

We conclude that distance education systems can bemore effective (increase learning and more likely tomotivate students) if they make the remote students

and the instructor feel that the remote students areactually present in the classroom, which should beachieved through high-level of interactivity andvisual realism of the virtual environment hosting theremote students.

3. System ArchitectureIn this section we provide an overview of our systemfor remote lecture attendance. The system has twomain components: a Classroom System that isdeployed at the on-campus classroom and a RemoteStudent System that is deployed for each remotelylocated student. Figure 3 presents a deployment withn remote students.

The Classroom System extends the classroom toaccommodate remote students in a way transparent tothe instructor and local students. The image of avirtual 3D room is projected onto the back wall of theclassroom to provide additional seats for the remotestudents. The instructor interacts with the remotestudents naturally, much the same way she/heinteracts with students physically present in theclassroom. The Classroom System communicates

5

with each of the sites where remote students are located to send audio and video from the classroom and to receive audio and video from each remote site.

The Remote Student System simulates the classroom environment for a remote student by displaying video of the classroom, and by rendering audio from the classroom and the other remote students in real time. The Remote Student System captures the student with a video sprite which it sends to the classroom in real time. The Remote Student System also captures the student audio which it sends to each of the other remote students and to the classroom in real-time. The audio communication between remote students and the instructor is always on.

The Classroom System and the Remote Student System use a common communication infrastructure for the audio and video transmission. The communication infrastructure is based on a group communication system (GCS). A GCS is a distributed messaging system that enables efficient communication between a set of processes logically organized in groups and communicating via multicast in an asynchronous environment where failures can occur. Services provided by a GCS include group membership as well as reliable and ordered message delivery (e.g. FIFO, causal, or total ordering). The membership service informs all members of a group about the list of currently connected and alive group members, and notifies group members about every group change. A group can change for several reasons. In an idealized fault-free setting, a change can only be caused by members voluntarily joining or leaving the group. In a more realistic environment faults such as processes becoming disconnected or network partitions can prevent members from communicating. When faults are healed, group members can communicate again. These events can also trigger corresponding changes in group membership.

A typical GCS follows an architecture where the major functionality is provided by a set of GCS servers, while applications interact with a server through a GCS client library. An application can be a member of many groups and can act both as a sender and a receiver. Many GCS implement customized group communication protocols relying on UDP unicast and multicast services.

The GCS abstraction provides a modular design and an organization of the communication based on topics of interest to the participants in the system. Groups are just pre-defined names and not reserved IP addresses as in IP multicast. In our case, a single group, Audio All, is used for all audio communication since all participants need to receive all audio. All participants subscribe to this group. A participant sends the audio it captures and receives the audio captured by all other participants. Video communication is handled using n+l groups, where n is the number of remote students. We assign one group, Classroom Video, for the video captured from the classroom. All Remote Student Systems and the Classroom System are members of this group. Finally, one group is assigned for the video captured from each remote student site. The members of group Remote Student Video i are the Remote Student System i and the Classroom System, which send and receive the remote student i video, respectively.

4. Classroom System The primary task of the Classroom System is to provide support for hosting remote students within a campus classroom, in a manner that interferes minimally with normal lecture activities. We target small to medium classrooms that seat 40 or fewer on- campus students. Such classrooms allow interactive lectures, which most benefit students. Large classrooms preclude all interaction between instructor and students, so augmenting such a classroom with an interactive distance education system is futile. When the lecture is a mere monologue, asynchronous distance education systems do not place remote students at much of a disadvantage compared to on- campus students who can physically attend lecture.

The Classroom System implements the interface between the instructor and remote students. Providing the remote students with instructor audio and video does not pose unusual challenges. However, simulating the presence of the remote students for the instructor is complicated by the fact that each remote student can potentially be located at a different site. Whereas audio contact is straightforwardly enabled by mixing and rendering real-time a u d i ~ feeds, enabling visual contact is more challenging. A na'ive approach would be to display the n video feeds of the n remote students in n separate windows. Such visualization is ineffectual. The instructor cannot

with each of the sites where remote students arelocated to send audio and video from the classroomand to receive audio and video from each remote site.

The Remote Student System simulates the classroomenvironment for a remote student by displaying videoof the classroom, and by rendering audio from theclassroom and the other remote students in real time.The Remote Student System captures the studentwith a video sprite which it sends to the classroom inreal time. The Remote Student System also capturesthe student audio which it sends to each of the otherremote students and to the classroom in real-time.The audio communication between remote studentsand the instructor is always on.

The Classroom System and the Remote StudentSystem use a common communication infrastructurefor the audio and video transmission. Thecommunication infrastructure is based on a groupcommunication system (GCS). A GCS is adistributed messaging system that enables efficientcommunication between a set of processes logicallyorganized in groups and communicating via multicastin an asynchronous environment where failures canoccur. Services provided by a GCS include groupmembership as well as reliable and ordered messagedelivery (e.g. FIFO, causal, or total ordering). Themembership service informs all members of a groupabout the list of currently connected and alive groupmembers, and notifies group members about everygroup change. A group can change for severalreasons. In an idealized fault-free setting, a changecan only be caused by members voluntarily joining orleaving the group. In a more realistic environmentfaults such as processes becoming disconnected ornetwork partitions can prevent members fromcommunicating. When faults are healed, groupmembers can communicate again. These events canalso trigger corresponding changes in groupmembership.

A typical GCS follows an architecture where themajor functionality is provided by a set of GCSservers, while applications interact with a serverthrough a GCS client library. An application can be amember of many groups and can act both as a senderand a receiver. Many GCS implement customizedgroup communication protocols relying on UDPunicast and multicast services.

The GCS abstraction provides a modular design andan organization of the communication based ontopics of interest to the participants in the system.Groups are just pre-defined names and not reservedIP addresses as in IP multicast. In our case, a singlegroup, Audio All, is used for all audio communicationsince all participants need to receive all audio. Allparticipants subscribe to this group. A participantsends the audio it captures and receives the audiocaptured by all other participants. Videocommunication is handled using n+ I groups, where nis the number of remote students. We assign onegroup, Classroom Video, for the video captured fromthe classroom. All Remote Student Systems and theClassroom System are members of this group.Finally, one group is assigned for the video capturedfrom each remote student site. The members of groupRemote Student Video i are the Remote StudentSystem i and the Classroom System, which send andreceive the remote student i video, respectively.

4. Classroom SystemThe primary task of the Classroom System is toprovide support for hosting remote students within acampus classroom, in a manner that interferesminimally with normal lecture activities. We targetsmall to medium classrooms that seat 40 or fewer on­campus students. Such classrooms allow interactivelectures, which most benefit students. Largeclassrooms preclude all interaction between instructorand students, so augmenting such a classroom withan interactive distance education system is futile.When the lecture is a mere monologue, asynchronousdistance education systems do not place remotestudents at much of a disadvantage compared to on­campus students who can physically attend lecture.

The Classroom System implements the interfacebetween the instructor and remote students. Providingthe remote students with instructor audio and videodoes not pose unusual challenges. However,simulating the presence of the remote students for theinstructor is complicated by the fact that each remotestudent can potentially be located at a different site.Whereas audio contact is straightforwardly enabledby mixing and rendering real-time audio feeds,enabling visual contact is more challenging. A naiveapproach would be to display the n video feeds of then remote students in n separate windows. Suchvisualization is ineffectual. The instructor cannot

6

I I GCS Client Librarv I 1 1

I GCS Server I

Figure 4. Classroom system.

maintain visual contact simultaneously with all remote students and is forced to scan each window sequentially, spending considerable effort to adapt cognitively to each one of the multitude of contexts.

We have developed a method that allows the instructor to maintain visual contact with the remote students in parallel. The students are extracted from their individual video feeds and inserted into a unified virtual environment, which is displayed at a natural location within the field of view of the instructor (Figure 1). The virtual environment extends the classroom at the back to provide additional seats, which are used by the remote students. The 3D model of the virtual extension is integrated with the images of the remote students and is rendered from a viewpoint that matches the default instructor position. The resulting image is projected onto the back wall of the classroom. The instructor easily monitors the local students and the back wall simultaneously, as if the classroom were larger. The

body language, facial expression, and events such as a remote student raising herlhis arm are immediately apparent to the instructor, which enables effective interaction. The penalty of hosting the remote students is essentially limited to the inherent disadvantage of a slightly larger class.

Such an approach has only recently become practical, enabled by advances in video technology (e.g. high- resolution high-frame rate webcams with standard high-bandwidth interfaces, high-resolution high- brightness projectors), in general purpose computing (e.g. PCs that can execute non-trivial per-pixel operations on high-resolution images at interactive rates), and in graphics computing (e.g. add-in graphics cards that can render 3D scenes described with millions of triangles at interactive rates). The classroom system consists of four main modules (Figure 4): Virtual Extension of Classroom, Classroom Video Capture, Instructor Audio Capture, and Audio Rendering, each described in detail below.

RemoteStudentAudio 1 n

3DModel

Compr.RemoteStudentSprite 1 n

ClassroomVideo

Capture

InstructorAudio

Capture

InstructorAudio

AudioRendering

GCS Client Library

GCS Server

Figure 4. Classroom system.

maintain visual contact simultaneously with allremote students and is forced to scan each windowsequentially, spending considerable effort to adaptcognitively to each one of the multitude of contexts.

We have developed a method that allows theinstructor to maintain visual contact with the remotestudents in parallel. The students are extracted fromtheir individual video feeds and inserted into aunified virtual environment, which is displayed at anatural location within the field of view of theinstructor (Figure I). The virtual environmentextends the classroom at the back to provideadditional seats, which are used by the remotestudents. The 3D model of the virtual extension isintegrated with the images of the remote students andis rendered from a viewpoint that matches the defaultinstructor position. The resulting image is projectedonto the back wall of the classroom. The instructoreasily monitors the local students and the back wallsimultaneously, as if the classroom were larger. The

body language, facial expression, and events such asa remote student raising her/his arm are immediatelyapparent to the instructor, which enables effectiveinteraction. The penalty of hosting the remotestudents is essentially limited to the inherentdisadvantage of a slightly larger class.

Such an approach has only recently become practical,enabled by advances in video technology (e.g. high­resolution high-frame rate webcams with standardhigh-bandwidth interfaces, high-resolution high­brightness projectors), in general purpose computing(e.g. PCs that can execute non-trivial per-pixeloperations on high-resolution images at interactiverates), and in graphics computing (e.g. add-ingraphics cards that can render 3D scenes describedwith millions of triangles at interactive rates). Theclassroom system consists of four main modules(Figure 4): Virtual Extension of Classroom,Classroom Video Capture, Instructor Audio Capture,and Audio Rendering, each described in detail below.

7

Figure 5. Visualizations of classroom extension (left and middle) and back-wall projector image.

4.1. Virtual Extension of Classroom In order to provide a believable virtual environment for hosting remote students, we created a realistic 3D digital model of a classroom that matches the size of the real-world classroom. The side walls, the ceiling, and the imaginary front wall of the virtual extension match the side walls, ceiling, and back wall of the classroom, respectively, as shown in Figure 5. The seating capacity and configuration of the virtual extension is tailored to the number of remote students that it hosts. This not only saves the instructor the disappointment of numerous empty seats, but also has the tangible benefit of a good use of the projected image by spacing the remote students apart horizontally and vertically to prevent that they occlude each other. The virtual extension shown in Figure 1 has two rows of 4 and 3 seats respectively, which is sufficient to host the 5 remote students involved in our experiments.

Realistic large-scale 3D modeling is an open research problem at the confluence of computer vision, computer graphics, and optical engineering. Several approaches are possible. Manual modeling using CAD or animation software produces good results for man-made scenes, but is time consuming and requires artistic talent. Automated modeling based on directly measuring color and depth using acquisition devices such as cameras and laser rangefinders excels at capturing real-world scenes realistically, but suffers from high equipment, time, and operator expertise costs. We experimented with several approaches for producing the 3D model for the virtual extension, including scanning with a time-of- flight laser rangefinder. The manual modeling

approach using state-of-the-art animation software yielded the best results in our case, since the modeler was able to take advantage of object repetition specific to classrooms (desks, chairs). ApproximateIy 100,000 triangles are sufficient to express the geometry of the virtual extension of the classroom, a geometry load that does not pose problems even for low-end graphics cards.

The remote student sprites were placed at predetermined locations in the geometric model to match the seats. The size of the sprites was chosen to approximately match the size of a student actually seated at that respective location. The remote students are updated every time a complete remote student image is received. The virtual extension is rendered 30 times per second and projected onto the back wall of the classroom. In order to achieve this performance yet to provide realism through high- quality lighting, a global illumination solution was pre-computed and burned into the texture maps. This precludes changing the lighting in the virtual extension interactively during the lecture. If such a feature is desired, one could pre-compute high- quality lighting at several intensity levels and interpolate at run time.

4.2. Classroom Video Capture The Classroom Video Capture module acquires the view of the classroom that includes the instructor and sends it to the remote student sites. A camera with a fisheye lens is used because its large field of view allows keeping the instructor in the frame without having to track the instructor or to move the camera. Although cameras that automatically track the

Figure 5. Visualizations of classroom extension (left and middle) and back-wall projector image.

4.1. Virtual Extension of ClassroomIn order to provide a believable virtual environmentfor hosting remote students, we created a realistic 3Ddigital model of a classroom that matches the size ofthe real-world classroom. The side walls, the ceiling,and the imaginary front wall of the virtual extensionmatch the side walls, ceiling, and back wall of theclassroom, respectively, as shown in Figure 5. Theseating capacity and configuration of the virtualextension is tailored to the number of remote studentsthat it hosts. This not only saves the instructor thedisappointment of numerous empty seats, but alsohas the tangible benefit of a good use of the projectedimage by spacing the remote students aparthorizontally and vertically to prevent that theyocclude each other. The virtual extension shown inFigure 1 has two rows of 4 and 3 seats respectively,which is sufficient to host the 5 remote studentsinvolved in our experiments.

Realistic large-scale 3D modeling is an open researchproblem at the confluence of computer vision,computer graphics, and optical engineering. Severalapproaches are possible. Manual modeling usingCAD or animation software produces good results forman-made scenes, but is time consuming andrequires artistic talent. Automated modeling based ondirectly measuring color and depth using acquisitiondevices such as cameras and laser rangefinders excelsat capturing real-world scenes realistically, butsuffers from high equipment, time, and operatorexpertise costs. We experimented with severalapproaches for producing the 3D model for thevirtual extension, including scanning with a time-of­flight laser rangefinder. The manual modeling

approach using state-of-the-art animation softwareyielded the best results in our case, since the modelerwas able to take advantage of object repetitionspecific to classrooms (desks, chairs). Approximately100,000 triangles are sufficient to express thegeometry of the virtual extension of the classroom, ageometry load that does not pose problems even forlow-end graphics cards.

The remote student sprites were placed atpredetermined locations in the geometric model tomatch the seats. The size of the sprites was chosen toapproximately match the size of a student actuallyseated at that respective location. The remotestudents are updated every time a complete remotestudent image is received. The virtual extension isrendered 30 times per second and projected onto theback wall of the classroom. In order to achieve thisperformance yet to provide realism through high­quality lighting, a global illumination solution waspre-computed and burned into the texture maps. Thisprecludes changing the lighting in the virtualextension interactively during the lecture. If such afeature is desired, one could pre-compute high­quality lighting at several intensity levels andinterpolate at run time.

4.2. Classroom Video CaptureThe Classroom Video Capture module acquires theview of the classroom that includes the instructor andsends it to the remote student sites. A camera with afisheye lens is used because its large field of viewallows keeping the instructor in the frame withouthaving to track the instructor or to move the camera.Although cameras that automatically track the

8

Figure 6. Classroom image delivered to remote students.

instructor are available commercially we have decided against using them because of lack of robustness. Such cameras periodically lose the instructor, which we found to be more disturbing than the lower resolution on the instructor provided by the current solution. The classroom image is down- sampled, compressed, and sent to the remote student sites. Figure 6 shows a typical classroom image as seen by a remote student.

4.3. Audio Capture and Rendering The task of the Audio Capture module is to acquire audio at the classroom and remote sites. Instructor audio is acquired with a wireless Lavaliere microphone and sent to the remote student sites. The Audio Rendering module receives and mixes audio from each remote student. The mixed sound is rendered on the speakers of the classroom. The same software module is used to render audio for the classroom and the remote students. The software at a location L discards audio packets received from the group that originated from location L.

5. Remote Student System A Remote Student System i has two tasks. The first task is to simulate the classroom environment for remote student i by receiving and rendering video of the classroom, and by receiving and rendering audio of the classroom and the other remote students. The

second task is to capture audio and video of remote student i and to send them to the classroom for integration into the virtual extension. The Remote Student System consists of several modules (Figure 7). The Remote Student Capture module acquires remote student i and builds an effective yet compact visual representation. The Classroom Display module provides remote student i with a view of the classroom through the panoramic images received from the classroom. The Audio Capture module acquires remote student i audio through a headset microphone and sends it to the classroom and the other remote students. Finally, Audio Rendering combines the audio received from the other remote students and from the instructor to provide audio feedback to remote student i via a headset.

5.1. Remote Student Capture The Remote Student Capture module is a critical component of the Remote Student System providing the instructor with a quality visual depiction of the remote student, in real time. One option is to acquire a 3D model of the student. Compared to modeling the virtual extension of the classroom, modeling the student is complicated by the fact that the scene is dynamic. However, a good approximation of the appearance of the remote student that works well within the virtual extension of the classroom can be obtained if the student is modeled with a real-time video sprite. A video sprite is a video where the background pixels are transparent at all times. The video sprite is a good approximation in our context because the remote student is displayed far away from the instructor (beyond the back walI of the classroom), so the flatness of the sprite is not a concern.

5.2. Background Subtraction The challenge is to find the foreground pixels in each frame, in real-time. We took the approach of background subtraction. First the remote student moves outside of the field of view of the webcam and a background frame is acquired. Then the background frame is used to determine which pixels changed, which are labeled as foreground. Since the camera adjusts exposure in real time, background pixels will not match exactly the pre-recorded background frame.

Figure 6. Classroom image delivered to remotestudents.

instructor are available commercially we havedecided against using them because of lack ofrobustness. Such cameras periodically lose theinstructor, which we found to be more disturbing thanthe lower resolution on the instructor provided by thecurrent solution. The classroom image is down­sampled, compressed, and sent to the remote studentsites. Figure 6 shows a typical classroom image asseen by a remote student.

4.3. Audio Capture and Rendering

The task of the Audio Capture module is to acquireaudio at the classroom and remote sites. Instructoraudio is acquired with a wireless Lavalieremicrophone and sent to the remote student sites. TheAudio Rendering module receives and mixes audiofrom each remote student. The mixed sound isrendered on the speakers of the classroom. The samesoftware module is used to render audio for theclassroom and the remote students. The software at alocation L discards audio packets received from thegroup that originated from location 1.

5. Remote Student SystemA Remote Student System i has two tasks. The firsttask is to simulate the classroom environment forremote student i by receiving and rendering video ofthe classroom, and by receiving and rendering audioof the classroom and the other remote students. The

second task is to capture audio and video of remotestudent i and to send them to the classroom forintegration into the virtual extension. The RemoteStudent System consists of several modules (Figure7). The Remote Student Capture module acquiresremote student i and builds an effective yet compactvisual representation. The Classroom Display moduleprovides remote student i with a view of theclassroom through the panoramic images receivedfrom the classroom. The Audio Capture moduleacquires remote student i audio through a headsetmicrophone and sends it to the classroom and theother remote students. Finally, Audio Renderingcombines the audio received from the other remotestudents and from the instructor to provide audiofeedback to remote student i via a headset.

5.1. Remote Student CaptureThe Remote Student Capture module is a criticalcomponent of the Remote Student System providingthe instructor with a quality visual depiction of theremote student, in real time. One option is to acquirea 3D model ofthe student. Compared to modeling thevirtual extension of the classroom, modeling thestudent is complicated by the fact that the scene isdynamic. However, a good approximation of theappearance of the remote student that works wellwithin the virtual extension of the classroom can beobtained if the student is modeled with a real-timevideo sprite. A video sprite is a video where thebackground pixels are transparent at all times. Thevideo sprite is a good approximation in our contextbecause the remote student is displayed far awayfrom the instructor (beyond the back wall of theclassroom), so the flatness of the sprite is not aconcern.

5.2. Background Subtraction

The challenge is to find the foreground pixels in eachframe, in real-time. We took the approach ofbackground subtraction. First the remote studentmoves outside of the field of view of the webcam anda background frame is acquired. Then thebackground frame is used to determine which pixelschanged, which are labeled as foreground. Since thecamera adjusts exposure in real time, backgroundpixels will not match exactly the pre-recordedbackground frame.

9

I I GCS Client Library I 1 1

I GCS Server I

Figure 7. Remote student system 1 .

Background subtraction has been studied extensively in image and video processing. A good overview of background subtraction techniques was compiled by Piccardi [2004]. Early work in background subtraction algorithms assumed a relatively static scene and used a simple adaptive filter to update the background pixel colors. More recent work utilizes elaborate models of the background to accommodate for changes.

In our context, performance is critical in order to achieve the desired frame rate of 5fps. We developed a simple and efficient background subtraction algorithm that assumes that the background is static. The assumption can be easily satisfied by a student that joins the lecture from a private room. Joining from a large active environment requires the student to choose a location with a static background, as it is the case, for example, when the background is provided by a wall.

Our algorithm performs a series of steps on each new frame: blurring with a flat kernel, correcting the current frame for consistency with the pre-acquired background frame, and computing the foreground pixels as pixels whose color changed significantly. Blurring reduces noise and thus increases robustness. Before comparison with the background frame, the current frame is corrected to take into account the adjustments performed by the camera. Note that although it is possible to disable the dynamic exposure adjustment performed by the camera, such an approach is not desirable since it lowers the quality of the image.

The frame is corrected efficiently using a safety region at the periphery of the frame which is known to be background. In Figure 8, the middle image shows the safety region in magenta. The currently acquired sprite with the magenta safety region is displayed to provide feedback to the student. The student 1s asked to keep out of the magenta region, and to reacquire the background frame when changes

Webcam LCD Monitor Headset MicHeadsetspeakers

A~ ~ ~ ~ ~,, ,,Remote

Classroom AudioStudent

Display CaptureAudio Rendering

Capture

Remote A~ ~ ~ ~~ ~StudentRemote ~

sprite,Classroom Student

~Image Audio 1

Remote ...ICompression I Decompression I Instructor Student

Audio Audio 2 n

Compr. ~ ~RemoteI

Student I Compressed

Sprite Classroom,, Image ,,I GCS Client Library

••I GCS Server IFigure 7. Remote student system 1.

Background subtraction has been studied extensivelyin image and video processing. A good overview ofbackground subtraction techniques was compiled byPiccardi [2004]. Early work in backgroundsubtraction algorithms assumed a relatively staticscene and used a simple adaptive filter to update thebackground pixel colors. More recent work utilizeselaborate models of the background to accommodatefor changes.

In our context, performance is critical in order toachieve the desired frame rate of 5fps. We developeda simple and efficient background subtractionalgorithm that assumes that the background is static.The assumption can be easily satisfied by a studentthat joins the lecture from a private room. Joiningfrom a large active environment requires the studentto choose a location with a static background, as it isthe case, for example, when the background isprovided by a wall.

Our algorithm performs a series of steps on each newframe: blurring with a flat kernel, correcting thecurrent frame for consistency with the pre-acquiredbackground frame, and computing the foregroundpixels as pixels whose color changed significantly.Blurring reduces noise and thus increases robustness.Before comparison with the background frame, thecurrent frame is corrected to take into account theadjustments performed by the camera. Note thatalthough it is possible to disable the dynamicexposure adjustment performed by the camera, suchan approach is not desirable since it lowers thequality of the image.

The frame is corrected efficiently using a safetyregion at the periphery of the frame which is knownto be background. In Figure 8, the middle imageshows the safety region in magenta. The currentlyacquired sprite with the magenta safety region isdisplayed to provide feedback to the student. Thestudent is asked to keep out of the magenta region,and to reacquire the background frame when changes

10

Figure 8. Frame, background, and video sprite compute by background subtraction.

in the background create persistent mismatches visible as misclassified foreground pixels. The sprite is compressed and sent to the classroom.

6. Implementation We implemented and deployed the system to a first classroom using 5 remote students (Figure 1) placed at different physical locations. The system was implemented using the following hardware (Section 6.1) and software (Section 6.2) components.

6.1. Hardware The hardware equipment used at the classroom (see Figure 2) consists of

a PC (3.2GHz Intel Xeon, 3GB RAM, 512MB) .with a graphics card (PCIe x16 nVidia Quadro FX 4400), a panoramic camera to capture the classroom and the instructor (PointGreyResearch Flea camera with a fisheye lens), a Lavaliere microphone to capture instructor audio (Azden 3 1 LT), a rear-facing projector for projecting the virtual extension of the classroom (Hitachi CP-XI250 XGA Multimedia Projector), and a sound playback system (Audio Receiver Panasonic SA-HE200K, subwoofer Panasonic SB-WA100, and four speakers Panasonic SB- AFC10).

The classroom was already outfitted with the sound system and a PC for driving the usual front-facing projector; we did not use that PC in order not to

interfere with the lectures held in the classroom during the development phase of our system.

At each remote student site the hardware used was a desktop computer, a webcam to capture the student (Logitech QuickCam Pro 4000 webcam), and a headset (Logitech Premium USB Headset 350).

6.2. Software Our system is developed under the Microsoft Windows platform, which has the advantage of directly supporting any camera, headset, projector, graphics card, microphone, and sound console on the market now or in the foreseeable future, a requirement for the large-scale adoption of our system.

The virtual extension of the classroom is modeled using 3dsmax, a leading commercial animation package. The geometry of the 3dsmax model is saved in the object file format, which is imported into the Classroom System. The virtual classroom extension is then rendered with hardware support using OpenGL.

Video processing in the Remote Student System and audio processing in both the Classroom System and the Remote Student System are implemented with Directshow filters using threads for concurrency. Concurrency provides a natural separation of independent control flows-aiding development- and also achieves higher performance.

The video processing in the Classroom System is a standalone Windows application and is organized in three concurrent threads: two of them are primarily used to handle the communication, one to send and one to receive, the third is used to handle the graphics

Figure 8. Frame, background, and video sprite compute by background subtraction.

in the background create persistent mismatchesvisible as misclassified foreground pixels. The spriteis compressed and sent to the classroom.

6. ImplementationWe implemented and deployed the system to a firstclassroom using 5 remote students (Figure I) placedat different physical locations. The system wasimplemented using the following hardware (Section6.1) and software (Section 6.2) components.

6.1. Hardware

The hardware equipment used at the classroom (seeFigure 2) consists of

• a PC (3.2GHz Intel Xeon, 3GB RAM, 512MB)with a graphics card (PCIe xl6 nVidia QuadroFX4400),

• a panoramic camera to capture the classroom andthe instructor (PointGreyResearch Flea camerawith a fisheye lens),

• a Lavaliere microphone to capture instructoraudio (Azden 31 LT),

• a rear-facing projector for projecting the virtualextension of the classroom (Hitachi CP-X1250XGA Multimedia Projector), and

• a sound playback system (Audio ReceiverPanasonic SA-HE200K, subwoofer PanasonicSB-WAlOO, and four speakers Panasonic SB­AFClO).

The classroom was already outfitted with the soundsystem and a PC for driving the usual front-facingprojector; we did not use that PC in order not to

interfere with the lectures held in the classroomduring the development phase ofour system.

At each remote student site the hardware used was adesktop computer, a webcam to capture the student(Logitech QuickCam Pro 4000 webcam), and aheadset (Logitech Premium USB Headset 350).

6.2. SoftwareOur system is developed under the MicrosoftWindows platform, which has the advantage ofdirectly supporting any camera, headset, projector,graphics card, microphone, and sound console on themarket now or in the foreseeable future, arequirement for the large-scale adoption of oursystem.

The virtual extension of the classroom is modeledusing 3dsmax, a leading commercial animationpackage. The geometry of the 3dsmax model is savedin the object file format, which is imported into theClassroom System. The virtual classroom extensionis then rendered with hardware support usingOpenGL.

Video processing in the Remote Student System andaudio processing in both the Classroom System andthe Remote Student System are implemented withDirectShow filters using threads for concurrency.Concurrency provides a natural separation ofindependent control flows-aiding development­and also achieves higher performance.

The video processing in the Classroom System is astandalone Windows application and is organized inthree concurrent threads: two of them are primarilyused to handle the communication, one to send andone to receive, the third is used to handle the graphics

II

component. Specifically, the functionality of each thread is: first thread captures and sends the panoramic view of the classroom, second thread receives the sprite messages sent from the remote students, and third thread renders the virtual classroom into which the remote student sprites are integrated. The video filter in the Remote Student System also uses three concurrent threads, organized in similar fashion: first thread acquires and sends the remote student's video to the network, second thread receives the video feed from the classroom, and third thread displays the classroom. For both systems, the priority of the receiving thread is set to be the highest to guarantee a timely update of the visual feedback to the instructor and to each remote student.

The audio mixer is implemented by a filter that runs concurrently with the video processing as a separate software application. Three concurrent threads are deployed to capture and send, receive, and play back the audio feeds between the remote students and the classroom. Audio is mixed and played back using the Directsound API. This requires that the header information of the audio sample be sent to the receiving end before any sound can be played back. In order to support students joining the session late, the header is sent periodically. The overhead incurred by repeatedly sending the header information amounts to only 0.02%.

We used the Spread group communication system as our communication infrastructure [Amir and Stanton 19981. We selected Spread because it provides an easy way to deploy the system while meeting performance requirements, and because it is publicly available. Spread is a general-purpose GCS for wide- and local-area networks. It provides reliable and ordered delivery of messages (e.g. FIFO, causal, or total ordering), as well as a membership service. The system consists of a server and a client library linked with the application. A client can obtain access to the group services by connecting to a server.

Any process4lient or server-can fail. If a server fails, all clients connected to that server also fail. When a network partition takes place, Spread servers detect it and continue to provide operation within each connected component. The client and server memberships follow the model of light-weight and heavy-weight groups. This architecture amortizes the cost of expensive distributed protocols, since such protocols are executed only by a relatively small

number of servers and not by all clients. This way, a simple join or leave of a client process translates into a single message instead of a costly full-fledged membership change, which is only triggered by network partitions.

In Spread any group member can be both a sender and a receiver. A client can be a member of many groups. Spread supports a large number of small- to medium-size groups. Each packet in Spread carries its own service type, allowing for fine granularity for the communication. For example, packets sent by applications that require agreement between participants can be sent using a causal or total ordering service depending on the nature of the application while single-source broadcasting applications can use the FIFO service that is more efficient and sufficient for such applications. Spread uses customized protocols relying on UDP unicast and multicast primitives.

7. Assessment This section gives a quantitative and qualitative assessment of our distance learning system.

7.1. Performance The main computational tasks of the remote student system are the decompression of the 400x300 classroom frame and the acquisition, construction, and compression of the 160x120 remote student sprite. Each of these tasks is performed at a sustained frame rate of 5 fps, leveraging efficient MPEG-4 compression and decompression implementations provided by the XviD library, and the assumption of a static background for sprite construction.

The classroom system compresses the classroom image, decompresses the sprites of the remote students and updates their respective textures in the 3D virtual extension of the classroom. Five remote students are comfortably handled by the single classroom workstation at 5fps. In fact, simulations show that decompression performance and bandwidth to the graphics card do not become a factor up to 50 remote students (modeled with 5fps 160xl20pixel sprites).

The average bit rate for sending and receiving a 160x120 remote student video sprite at 5fps is approximately 35 kbps; the average bit rate of the classroom video is approximately 355 kbps for

component. Specifically, the functionality of eachthread is: first thread captures and sends thepanoramic view of the classroom, second threadreceives the sprite messages sent from the remotestudents, and third thread renders the virtualclassroom into which the remote student sprites areintegrated. The video filter in the Remote StudentSystem also uses three concurrent threads, organizedin similar fashion: first thread acquires and sends theremote student's video to the network,. second threadreceives the video feed from the classroom, and thirdthread displays the classroom. For both systems, thepriority of the receiving thread is set to be the highestto guarantee a timely update of the visual feedback tothe instructor and to each remote student.

The audio mixer is implemented by a filter that runsconcurrently with the video processing as a separatesoftware application. Three concurrent threads aredeployed to capture and send, receive, and play backthe audio feeds between the remote students and theclassroom. Audio is mixed and played back usingthe DirectSound API. This requires that the headerinformation of the audio sample be sent to thereceiving end before any sound can be played back.In order to support students joining the session late,the header is sent periodically. The overhead incurredby repeatedly sending the header informationamounts to only 0.02%.

We used the Spread group communication system asour communication infrastructure [Amir and Stanton1998]. We selected Spread because it provides aneasy way to deploy the system while meetingperformance requirements, and because it is publiclyavailable. Spread is a general-purpose GCS for wide­and local-area networks. It provides reliable andordered delivery of messages (e.g. FIFO, causal, ortotal ordering), as well as a membership service. Thesystem consists of a server and a client library linkedwith the application. A client can obtain access to thegroup services by connecting to a server.

Any process---client or server-can fail. If a serverfails, all clients connected to that server also fail.When a network partition takes place, Spread serversdetect it and continue to provide operation withineach connected component. The client and servermemberships follow the model of light-weight andheavy-weight groups. This architecture amortizes thecost of expensive distributed protocols, since suchprotocols are executed only by a relatively small

number of servers and not by all clients. This way, asimple join or leave of a client process translates intoa single message instead of a costly full-fledgedmembership change, which is only triggered bynetwork partitions.

In Spread any group member can be both a senderand a receiver. A client can be a member of manygroups. Spread supports a large number of small- tomedium-size groups. Each packet in Spread carriesits own service type, allowing for fine granularity forthe communication. For example, packets sent byapplications that require agreement betweenparticipants can be sent using a causal or totalordering service depending on the nature of theapplication while single-source broadcastingapplications can use the FIFO service that is moreefficient and sufficient for such applications. Spreaduses customized protocols relying on UDP unicastand multicast primitives.

7. AssessmentThis section gives a quantitative and qualitativeassessment of our distance learning system.

7.1. Performance

The main computational tasks of the remote studentsystem are the decompression of the 400x300classroom frame and the acquisition, construction,and compression of the l60xl20 remote studentsprite. Each of these tasks is performed at a sustainedframe rate of 5 fps, leveraging efficient MPEG-4compression and decompression implementationsprovided by the XviD library, and the assumption ofa static background for sprite construction.

The classroom system compresses the classroomimage, decompresses the sprites of the remotestudents and updates their respective textures in the3D virtual extension of the classroom. Five remotestudents are comfortably handled by the singleclassroom workstation at 5fps. In fact, simulationsshow that decompression performance and bandwidthto the graphics card do not become a factor up to 50remote students (modeled with 5fps l60xl20pixelsprites).

The average bit rate for sending and receiving al60x120 remote student video sprite at 5fps isapproximately 35 kbps; the average bit rate of theclassroom video is approximately 355 kbps for

12

400x300 images at 5 fps. The audio bandwidth requirement is negligible by comparison. This implies an upload/download bandwidth requirement of 351355 kbps at each remote student site, which is within the limits of commodity broadband networking (e.g. DSL or cable modem). For the classroom system the required upload/download requirement is 3551175 kbps, which is within the limits of the connectivity available at most educational institutions.

7.2. Informal assessment During a demonstration of the system, the authors, including education science experts, had the opportunity to informally assess the system. It was found that the system provides a quality audio and video link between the instructor and the remote students.

The quality of the sound is excellent, allowing for vocal intonations and nuances to be detected. This facilitates a sense of rapport among the remote students and the instructor. It also partially supports a sense of rapport between local students and remote students; the remote students are somewhat disadvantaged in that they can hear the instructor, but not the local students.

The instructor interacts naturally with both local and remote students as shehe can hear and see both sets of learners. The remote students can see the instructor clearly. The instructor is able to move freely while lecturing, which is an important advantage over systems for distance lecture delivery that employ a fixed camera and have the instructor place bound.

The sense of the remote students being present seemed greater for the instructor than for the remote students themselves, because while the local participants could see the remote students, the remote students could not see themselves integrated into the classroom. The quality of the remote students' image is sufficiently high for the instructor to discern whether the remote students are confused or disengaged.

The remote student can raise herhis hand to ask or answer a question. The remote student has no control over what they can see, they are restricted to the panoramic image of the classroom received. The instructor and local students have control in that they can choose whom to look at: local students, the

instructor, or the remote students. Presence and interaction could be increased by providing the remote students with a view of the entire classroom that includes the remote students. However, this system shows promise for enabling interaction and a sense of presence not possible with current systems.

7.3. Pilot study In order to further assess the effectiveness of the design of the distance learning system we conducted a pilot study involving 15 local and 5 remote students. None of the students had been involved in a distance learning class prior to the pilot study.

The remote students were distributed throughout the building of the local classroom and the remote student workstations were connected to the local classroom through a lOOMbps local area network. This eliminated confounding factors such as occasional networking bottlenecks, and also enabled our team to observe the remote students during the pilot.

The students attended 4 lecture sessions, each lasting 1 hour. The lecture topics were introduction to digital video, camera operations, video formats, and video delivery. Before the first session the remote students were briefed on how to use the system (e.g. how to turn audio communication on and off, and how to acquire the background to be used for the construction of their sprite).

The students were tested before and after the 4 lectures. The tests show no significant difference in pretest scores between the local and remote students, which indicates that the local and remote groups were roughly equivalent. Although the student group sizes were small, there were no outliers, which strengthens our confidence in this conclusion. The posttest scores were higher for each group than their pretest scores, which indicates that both groups learned. The posttest scores also show no significant difference between the two groups, which suggests that both groups learned the same amount.

After the 4 sessions, in addition to the posttest, the students also completed a comprehensive survey comprising 93 questions and participated in focus group sessions. A detailed description and analysis of the survey and focus groups is beyond the scope of this paper. The conclusions we have drawn are:

400x300 images at 5 fps. The audio bandwidthrequirement is negligible by comparison. Thisimplies an upload/download bandwidth requirementof 35/355 kbps at each remote student site, which iswithin the limits of commodity broadbandnetworking (e.g. DSL or cable modem). For theclassroom system the required upload/downloadrequirement is 355/175 kbps, which is within thelimits of the connectivity available at mosteducational institutions.

7.2. Informal assessment

During a demonstration of the system, the authors,including education science experts, had theopportunity to informally assess the system. It wasfound that the system provides a quality audio andvideo link between the instructor and the remotestudents.

The quality of the sound is excellent, allowing forvocal intonations and nuances to be detected. Thisfacilitates a sense of rapport among the remotestudents and the instructor. It also partially supports asense of rapport between local students and remotestudents; the remote students are somewhatdisadvantaged in that they can hear the instructor, butnot the local students.

The instructor interacts naturally with both local andremote students as shelhe can hear and see both setsoflearners. The remote students can see the instructorclearly. The instructor is able to move freely whilelecturing, which is an important advantage oversystems for distance lecture delivery that employ afixed camera and have the instructor place bound.

The sense of the remote students being presentseemed greater for the instructor than for the remotestudents themselves, because while the localparticipants could see the remote students, the remotestudents could not see themselves integrated into theclassroom. The quality of the remote students' imageis sufficiently high for the instructor to discernwhether the remote students are confused ordisengaged.

The remote student can raise herlhis hand to ask oranswer a question. The remote student has no controlover what they can see, they are restricted to thepanoramic image of the classroom received. Theinstructor and local students have control in that theycan choose whom to look at: local students, the

instructor, or the remote students. Presence andinteraction could be increased by providing theremote students with a view of the entire classroomthat includes the remote students. However, thissystem shows promise for enabling interaction and asense ofpresence not possible with current systems.

7.3. Pilot study

In order to further assess the effectiveness of thedesign of the distance learning system we conducteda pilot study involving 15 local and 5 remotestudents. None of the students had been involved in adistance learning class prior to the pilot study.

The remote students were distributed throughout thebuilding of the local classroom and the remotestudent workstations were connected to the localclassroom through a IOOMbps local area network.This eliminated confounding factors such asoccasional networking bottlenecks, and also enabledour team to observe the remote students during thepilot.

The students attended 4 lecture sessions, each lasting1 hour. The lecture topics were introduction to digitalvideo, camera operations, video formats, and videodelivery. Before the first session the remote studentswere briefed on how to use the system (e.g. how totum audio communication on and off, and how toacquire the background to be used for theconstruction of their sprite).

The students were tested before and after the 4lectures. The tests show no significant difference inpretest scores between the local and remote students,which indicates that the local and remote groups wereroughly equivalent. Although the student group sizeswere small, there were no outliers, which strengthensour confidence in this conclusion. The posttest scoreswere higher for each group than their pretest scores,which indicates that both groups learned. The posttestscores also show no significant difference betweenthe two groups, which suggests that both groupslearned the same amount.

After the 4 sessions, in addition to the posttest, thestudents also completed a comprehensive surveycomprising 93 questions and participated in focusgroup sessions. A detailed description and analysis ofthe survey and focus groups is beyond the scope ofthis paper. The conclusions we have drawn are:

13

- According to the remote students, a strength of the system was the ability to directly talk to the instructor and to other remote students.

- According to the remote students, a weakness of the system was the inability to see, hear, or talk to local students.

- The local students were little affected by the involvement of the remote students, and, in general, by the fact that the lecture was also delivered remotely.

The instructor that delivered the lectures during the pilot has extensive experience with prior synchronous and asynchronous distance learning systems. During an interview after the pilot he indicated that the distance learning system provides a closer connection to the remote students than possible with prior systems. He noted that he was able to talk to the remote students frequently and casually, and that, overall, the lectures "did not feel like distance learning".

8. Conclusions and Future Work We have designed, implemented, and deployed a distance education system based on integrating the remote students in a unified, highly-interactive virtual environment. Most classrooms already have a front- facing projector driven by a PC-the main piece of additional equipment is a projector facing towards the back of the classroom. The projector paints a virtual hole into the back wall which reveals a virtual extension of the classroom that hosts remotely located students. Preliminary tests indicate that the configuration is effective. The remote students feel and are perceived as being present in the classroom.

We are currently augmenting the system with capabilities such as instructor tracking, morphing of classroom extension to smoothly adjust to the number of students, exaggerated visualization of remote students for improved interaction efficiency, electronic whiteboard support, and distributed class materials. The deployment of the first prototype is an important milestone that enables future work in several major directions.

One direction is the use of the system in the context of actual courses to complete a rigorous evaluation of the educational impact. A second direction is to deploy the system over the Internet, which brings

new challenges such as addressing security and privacy issues.

Most of the current effort concentrated on improving the instructor-side of the interface between instructor and remote students, since it is presently the most deficient thus the most critical. A third direction of future work is to research improving the interface on the remote student side, by possibly letting the student see herhimself integrated in the classroom, together with the other remote students.

Finally, we will develop support for other on-campus interactive educational activities such as attending office hours and study groups. In addition to researching the best way of arranging the remote student avatars such as to enable round-table-like discussions and to add the capability to freely exchange hand-written notes, supporting study groups effectively will require creating a comfortable virtual setting conducive to productive collaboration. One important sub-problem is producing highly- realistic digital 3D models of coffee shops, student union lounges and other preferred campus venues.

9. Acknowledgments The authors would like to thank Carlos Morales, Mihai Mudure, Chunhui Mei, Gleb Bahmutov, Paul Rosen, Yi Zhang, and Laura Arns for their help. This was work was supported by the National Science Foundation through grant SCI-0417458.

References Amir, Y. and Stanton, J. 1998. The Spread wide area group communication system. In Technical Report CNDS-98-4, Johns Hopkins University, Department of Computer Science.

Deshpande, S. G., Hwang, J.-N. 2001. A real- time interactive virtual classroom multimedia distance learning system. In IEEE Transaction on Multimedia, vol. 3, no. 4.

Foshay, R. and Bergeron, C. 2000. Web-based education: A reality check. In TechTrends, 44, 16-9.

Galusha, J. M. 1997. Barriers to learning in distance education: University of Southern

According to the remote students, a strength ofthe system was the ability to directly talk to theinstructor and to other remote students.

According to the remote students, a weakness ofthe system was the inability to see, hear, or talkto local students.

The local students were little affected by theinvolvement of the remote students, and, ingeneral, by the fact that the lecture was alsodelivered remotely.

The instructor that delivered the lectures during thepilot has extensive experience with prior synchronousand asynchronous distance learning systems. Duringan interview after the pilot he indicated that thedistance learning system provides a closer connectionto the remote students than possible with priorsystems. He noted that he was able to talk to theremote students frequently and casually, and that,overall, the lectures "did not feel like distancelearning".

8. Conclusions and Future WorkWe have designed, implemented, and deployed adistance education system based on integrating theremote students in a unified, highly-interactive virtualenvironment. Most classrooms already have a front­facing projector driven by a PC-the main piece ofadditional equipment is a projector facing towards theback of the classroom. The projector paints a virtualhole into the back wall which reveals a virtualextension of the classroom that hosts remotelylocated students. Preliminary tests indicate that theconfiguration is effective. The remote students feeland are perceived as being present in the classroom.

We are currently augmenting the system withcapabilities such as instructor tracking, morphing ofclassroom extension to smoothly adjust to the numberof students, exaggerated visualization of remotestudents for improved interaction efficiency,electronic whiteboard support, and distributed classmaterials. The deployment of the first prototype is animportant milestone that enables future work inseveral major directions.

One direction is the use of the system in the contextof actual courses to complete a rigorous evaluation ofthe educational impact. A second direction is todeploy the system over the Internet, which brings

new challenges such as addressing security andprivacy issues.

Most of the current effort concentrated on improvingthe instructor-side of the interface between instructorand remote students, since it is presently the mostdeficient thus the most critical. A third direction offuture work is to research improving the interface onthe remote student side, by possibly letting thestudent see herlhimself integrated in the classroom,together with the other remote students.

Finally, we will develop support for other on-campusinteractive educational activities such as attendingoffice hours and study groups. In addition toresearching the best way of arranging the remotestudent avatars such as to enable round-table-likediscussions and to add the capability to freelyexchange hand-written notes, supporting studygroups effectively will require creating a comfortablevirtual setting conducive to productive collaboration.One important sub-problem is producing highly­realistic digital 3D models of coffee shops, studentunion lounges and other preferred campus venues.

9. AcknowledgmentsThe authors would like to thank Carlos MoralesMihai Mudure, Chunhui Mei, Gleb Bahmutov, PaulRosen, Yi Zhang, and Laura Arns for their help. Thiswas work was supported by the National ScienceFoundation through grant SCI-0417458.

ReferencesAmir, Y. and Stanton, J. 1998. The Spread widearea group communication system. In TechnicalReport CNDS-98-4, Johns Hopkins University,Department of Computer Science.

Deshpande, S. G., Hwang, J.-N. 2001. A real­time interactive virtual classroom multimediadistance learning system. In IEEE Transactionon Multimedia, vol. 3, nO. 4.

Foshay, R. and Bergeron, C. 2000. Web-basededucation: A reality check. In TechTrends, 44,16-9.

Galusha, J. M. 1997. Barriers to learning indistance education: University of Southern

14

Mississippi". Retrieved May 10, 2006 from http://~~~.infra~tr~~ction.com/barriers.htm

Heeter, C. 2003. Reflections on real presence by a virtual person. In Presence: Teleoperators and Virtual Environments, 12(4), pp. 335-45.

Lavooy, M. J. and Newlin, M. H. 2003. Computer mediated communication: Online instruction and interactivity. In Journal of Interactive Learning Research, 14(2), pp. 1 57- 65.

Maly, R. et al. 2001. IRI-H, a Java-based distance education system: its architecture and performance. In Journal on Education Resources in Computing (JERIC), vol. 1, issue les.

NCES (2002) Post-secondary education quick information system. In Suwey on Distance Education at Higher Education Institutions: 2000-2001.

Piccardi, M. 2004. Background subtraction techniques: A review. In Systems, Man and Cybernetics, 2004 IEEE International Conference on, vol. 4, pp. 3099-3 104.

Sagias, Yiannis N. 2002. Distributed and online distance lecturing environment (the Virtual Blackboard Project). In Educational Technology & Society, 5 (1).

Shi, Y., Xie, W., Xu, G. 2002. Smart remote classroom: Creating a revolutionary real-time interactive distance learning system. In International Conference on Web-Based Learning.

Mississippi". Retrieved May 10, 2006 fromhttp://www.infrastruction.comlbaniers.htm

Heeter, C. 2003. Reflections on real presence bya virtual person. In Presence: Teleoperators andVirtual Environments, 12(4), pp. 335-45.

Lavooy, M. J. and Newlin, M. H. 2003.Computer mediated communication: Onlineinstruction and interactivity. In Journal ofInteractive Learning Research, 14(2), pp. 157­65.

Maly, R. et al. 2001. IRI-H, a Java-baseddistance education system: its architecture andperfonnance. In Journal on EducationResources in Computing (JERlC) , vol. 1, issueles.

NCES (2002) Post-secondary education quickinfonnation system. In Survey on DistanceEducation at Higher Education Institutions:2000-2001.

Piccardi, M. 2004. Background subtractiontechniques: A review. In Systems, Man andCybernetics, 2004 IEEE InternationalConference on, vol. 4, pp. 3099-3104.

Sagias, Yiannis N. 2002. Distributed and onlinedistance lecturing environment (the VirtualBlackboard Project). In Educational Technology& Society, 5 (l).

Shi, Y., Xie, W., Xu, G. 2002. Smart remoteclassroom: Creating a revolutionary real-timeinteractive distance learning system. InInternational Conference on Web-BasedLearning.

]5