is it live or is it internet? experimental estimates of the effects of online instruction on student...

NORC at the University of Chicago

The University of Chicago

Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction onStudent LearningAuthor(s): David Figlio, Mark Rush, and Lu YinSource: Journal of Labor Economics, Vol. 31, No. 4 (October 2013), pp. 763-784Published by: The University of Chicago Press on behalf of the Society of Labor Economists and theNORC at the University of ChicagoStable URL: http://www.jstor.org/stable/10.1086/669930 .

Accessed: 21/09/2013 09:38

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

The University of Chicago Press, Society of Labor Economists, NORC at the University of Chicago, TheUniversity of Chicago are collaborating with JSTOR to digitize, preserve and extend access to Journal ofLabor Economics.

http://www.jstor.org

This content downloaded from 137.207.120.173 on Sat, 21 Sep 2013 09:38:43 AMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=ucpress

http://www.jstor.org/action/showPublisher?publisherCode=sle

http://www.jstor.org/action/showPublisher?publisherCode=norc

http://www.jstor.org/stable/10.1086/669930?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


Is It Live or Is It Internet?Experimental Estimates of the Effects

of Online Instruction onStudent Learning

David Figlio, Northwestern University, NBER, and CESifo

Mark Rush, University of Florida

Lu Yin, University of Florida

This article presents the first experimental evidence on the effects oflive versus Internet media of instruction. Students in a large intro-ductorymicroeconomics course at amajor research university wererandomly assigned to live lectures versus watching these same lec-tures in an Internet setting where all other factors ðe.g., instruction,supplemental materialsÞ were the same. We find modest evidencethat live-only instruction dominates Internet instruction. These re-sults are particularly strong for Hispanic students, male students,and lower-achieving students. We also provide suggestions for fu-ture experimentation in other settings.

Throughout the United States, public 4-year colleges and universitiesare facing fiscal constraints not seen in 4 decades. State and local appro-priations for higher education, measured as a share of personal income,

We are grateful to the anonymous university and microeconomics instructor forproviding registrar data and for permitting the experiment to take place and facili-tating the experiment. We benefited greatly from comments from seminar partici-pants at Northwestern University, the University of Florida, and the University

[ Journal of Labor Economics, 2013, vol. 31, no. 4]© 2013 by The University of Chicago. All rights reserved.0734-306X/2013/3104-0001$10.00

763



have fallen virtually monotonically since the late 1970s, and today theyare at a level not seen since the mid-1960s ðMortenson 2005Þ. Kane andOrszag ð2003Þ document the precipitous decline in per student spendingand the stature of public 4-year colleges and universities during the 1980sand 1990s, and according to the Organization of State Higher EducationExecutive Officers, higher education institutions in all but five states expe-rienced real declines in per student revenues from state and local sourcesduring the first half of the current decade, a time of flush state and localcoffers. In 2006, public 4-year colleges and universities relied on tuition forover 37% of their total revenues for the first time in modern history, andthe recent financial crisis has surely further increased the fiscal constraintsfaced by public and private universities alike.The dramatically increased fiscal constraints facing public colleges and

universities, coupled with rapid improvements in technology, have pavedthe way for higher education institutions to introduce technology-basedplatforms for mass instruction. The use of Internet classes has explodedover the past decade, especially in the past few years. Over 2.6 millionstudents took at least one online course in Fall 2005, up from 1.6 million3 years earlier ðAllen and Seaman 2006Þ. Although the majority of thesestudents are in community colleges and junior colleges, more than 80% ofdoctoral/research institutions in the United States offer online classes. All10 of the largest 4-year colleges and universities in the United States offeronline classes, some with over 400 sections and others with more than10,000 students per term enrolled in at least one online class. Today virtu-ally every institution with more than 15,000 students offers online classes.If Internet-based classes are at least reasonable substitutes for live-

lecture classes, then the use of Internet-based classes could be a cost-effective method of combating increased fiscal constraint. In theory,Internet-based classes may even dominate live-lecture classes, as they of-fer students more flexibility in the timing of attendance as well as the op-portunity to review lectures to clear up confusing points. They also providethe opportunity for improved access to higher education for residents ofremote communities. However, Internet-based lectures provide weaker in-centives for students to regularly attend and keep up with classes, and as hasbeen documented at one major 4-year institution, last-minute cramming inInternet-based courses is rampant ðDonovan, Figlio, and Rush 2006Þ. Butsince increasing live-lecture class sizes is associated with deleterious con-sequences for students ðBettinger and Long 2007Þ, offering classes throughan electronic medium may be an appealing alternative mechanism for costsavings in higher education.

of Michigan, and audience members at the annual meeting of the American Eco-nomic Association. We are grateful to Susan Dynarski for very helpful comments.All remaining errors are our own. Contact the corresponding author, David Figlio,at [email protected].

764 Figlio et al.



A major report released by the US Department of Education on June 26,2009, provides additional support for the expansion of online education.This study, a meta-analysis of the available research on live versus onlinedelivery of education ðprimarily higher educationÞ, suggests that online de-livery of material leads to improvements in student outcomes relative to livedelivery, with hybrid live-plus-Internet delivery having the largest benefitsof all. While the Department of Education’s press release on the report con-centrated on the potential benefits of integrating electronic content intoregular classrooms, the ensuing news coverage ðand the report itselfÞ alsoemphasized the relative benefits of online-only education.That said, the studies that provided the basis for this meta-analysis may

not be sufficient to draw conclusions about the relative benefits of liveversus online education. Only 16 of the studies considered in this meta-analysis used a simple randomization method to assign students into eithera treatment or a control group, with an average study size of 84 partici-pants, and only two of these studies had the same instructor teaching boththe treatment group and the control group. In these two studies ðZhang2005; Zhang et al. 2006Þ, the researchers compared a 45-minute single-session live lecture in a classroom setting to a 45-minute single-sessione-learning experience in a research laboratory. Therefore, the existing exper-imental evidence on live versus online delivery of large lectures is effectivelynonexistent.1

This article aims to fill this important gap by reporting on an experimentin which students were randomly assigned to either an online or a live sec-tion of a course taught by one instructor and for which the ancillaries forthe class, such as the web page, problem sets, and TA support, as well as theexams, were identical between the sections. The only difference betweenthese sections is the method of delivery of the lectures: some studentsviewed the lectures live, as would be the case in traditional classes, while

1 The US Department of Education meta-analysis did not include a small num-ber of papers that have been published in economics journals that we believe meettheir criteria for inclusion in the federal study. Navarro and Shoemaker ð2000Þ pre-sent evidence that students taking an introductory macroeconomics class online hadsignificantly better test scores than students taking the same course in a live lectureformat, while Brown and Liedholm ð2002Þ report just the opposite result for stu-dents taking an introductory microeconomics class. However, Navarro and Shoe-maker compared a lecture format, with no class web page, to an online format thatincluded a web page, with a bulletin board for posting questions, weekly online chatdiscussions with the instructor, and quizzes, which the students were required totake weekly, as well as giving the students a CD with the audio part of the lecturesalong with PowerPoint slides and review questions. Brown and Liedholm’s onlineversus live comparison contrasted a live class with ðapparentlyÞ no web page to anonline class with streaming videos of one semester’s lectures and a variety of ad-ditional material, such as numeric problems and repeatable quizzes. These arehardly “apples-to-apples” comparisons and so conclusions drawn from them aboutthe performance of online versus live lectures are not robust.

Effects of Online Instruction on Student Learning 765



other students viewed the lectures on the Internet. Thus we are able to de-termine how online delivery of traditional lectures compares with live de-livery. It is important to note that all students had access to a rich array ofInternet-based resources regardless of the location of where they viewed thelectures. In many ways, therefore, this is precisely the trade-off that univer-sities are increasingly facing as they decide the appropriate medium for lec-ture delivery in their large classes. The results of this experiment, therefore,have significant potential implications for public and university policies.

I. The Class and the Experiment

We utilize data from an experiment conducted in a large principles ofmicroeconomics class taught at a large selective doctorate-granting univer-sity. This class is taught to between 1,600 and 2,600 students a semester bya single instructor. Typically, the students can register for a “live” section inwhich they can watch the lecture in a room with approximately 190 seatsor they can register for an “online” section in which they watch the samelecture online. The lecture is videotaped as it is presented and then madeavailable via the class web page to all students. Once the lecture is taped, it isretained on the Internet for the entire semester. Given the room-size con-straint, most students register for an online section. In a typical semester, ap-proximately 50–60 students actually come to any given live lecture. Becausethe room has vacant seats, normally no effort is made to keep the studentswho registered for an Internet section from attending the live section. Infact, because the live section is limited to 190, most of the students attend-ing the live lecture have registered for an online section simply becausethe live section was filled when it came time for them to register for the class.The majority of the students who register for the live section ultimatelychoose to watch the majority of the lectures online.Students who register for the live section and students who register for

the online section have access to the exact same class web page. The classweb page has a link to watch the lectures as well as a substantial variety ofclass supplements: a set of online quizzes, past exams, and so forth.As such,both live-access and online-access students have a rich web-based learningenvironment to supplement the class lectures. The exams are given in theevening. Both sets of students take the exact same exams given at the exactsame time. All students, regardless of the section for which they registered,have the same access to the instructor during office hours and the sameaccess to graduate student TA help. There are no discussion sessions. Soin a typical semester the only difference between the students is the sectionin which they have registered, which, because anyone can attend the livelecture or watch the lecture online, is a meaningless distinction. Gradingin the class is based only on exams. There are three exams: two midtermsand a final exam. The exams are all multiple choice, and they are machine

766 Figlio et al.



graded. The instructor creates the exams, which are primarily based on thelectures.Because of the obvious selection problems, one cannot simply look at

the difference in the performance of students who attended the live lectureversus students who watched the lectures online. So, during the Spring2007 semester, with the support of the instructor and of the university, weconducted an experiment with this class. Before the class started, the in-structor e-mailed all the students who had enrolled and offered them thechance to participate in an experiment. Of the nearly 1,600 students in theclass, 327 students volunteered to be part of the experiment. The instruc-tor promised to boost the volunteers’ grade by half of a letter grade at theend of the term—the only incentive permitted by the university—in ex-change for allowing us the opportunity to randomly assign them to watch-ing the lecture live or watching the lecture online. Students who were as-signed to watch the lecture live had their class websites altered to removeaccess to the lecture online; otherwise no further change was made to theirwebsites. Students who were assigned to the online section were not al-lowed in the classroom to watch the live lecture. Indeed, for that semesteronly, the only students allowed in the classroom during the live lecture werestudents we had assigned to the live lecture or students who had registeredfor the live lecture and who opted to not participate in the experiment.2

Among the 327 volunteers, 112 students were assigned to the live groupand 215 were assigned to the online group. In order to start the experimentfrom the first day of class, the students were contacted before the add/dropdeadline, which occurs a week after classes start. After their registrationwas completed, 15 of the 112 students assigned to the live group requestedreassignment to an online session due to schedule conflicts. We made thisreassignment but dropped them from the analysis, leaving a total of 97 stu-dents in the live-only section.3

The specific nature of participant recruitment in this experiment leadsto potential statistical power and external validity issues. Restrictions im-posed by the Institutional ReviewBoard at the university in questionmaderecruitment of a larger fraction of the student population into the experi-ment more difficult. The instructor was limited in the degree to which hecould contact the students to recruit them into the experiment, and wewere limited as to the incentives that could be offered. The ideal situationfrom an external validity standpoint would have been to randomly assignall students to either a live or an online section of the class, but this was notpossible given the culture of the university, wheremixed live-online classes

2 We stationed graduate students at the door to enforce these regulations and tocompile a record of which students watched each live lecture.

3 Later in the article we present evidence suggesting that our results are quali-tatively invariant to our decisions as to how we treat crossover cases.




are typically characterized by complete student autonomy. Statisticalpower is still not a major concern here: even with a smaller-than-desiredsample, we can still detect effects on the order of two points on a 100-pointscale—or one-fifth the size of the incentive to participate in the study.External validity issues, on the other hand, are a much bigger potentialconcern, as our study sample may not be representative of a broader pop-ulation of potential students. We discuss the limitations to external valid-ity in Section III below.

II. The Data and the Results

Four groups of students took the course in question:

1. Students who volunteered for the experiment and were randomlyassigned to watching the lectures online. These students were re-quired to watch the lectures online. Two hundred and fifteenð215Þ students fell within this group.

2. Students who volunteered for the experiment and were randomlyassigned to watching the lectures live. These students were requiredto watch the lectures live. Ninety-seven ð97Þ students fell within thisgroup.

3. Students who did not volunteer for the experiment and were ini-tially registered in an online section. These students were requiredto watch the lectures online. One thousand, two hundred, and threestudents ð1,203Þ fell within this group.

4. Students who did not volunteer for the experiment and were ini-tially registered in the live section. These students were allowed tochoose whether to watch a lecture live or online or a hybrid thereof.Seventy-seven ð77Þ students fell within this group.

Groups 1 and 2, the participants in our experiment, had exactly the samecourse with one crucial difference: they were randomly assigned to differ-ent delivery mechanisms for the lectures. Hence, comparing their perfor-mance potentially offers us an apples-to-apples comparison of an onlineclass to a traditional live lecture without worrying about the possibility ofselection issues or how to correct for the selection.First, however, we examine whether the students who volunteered for

the experiment were different in observable ways from the nonvolunteers.Table 1 compares the students who volunteered for the experiment withthose who did not for two groups of students—those who initiated theclass and those who completed the class. The data pertaining to the stu-dents’ maternal educational attainment were obtained directly from thestudents; the remaining data were obtained from the university’s records.As can be seen in the table, experiment volunteers differ from nonvol-unteers along a number of dimensions, but the differences are not uni-

768 Figlio et al.



directional. For example, experiment volunteers are more likely to havehigher grades at the university than are nonvolunteers, but volunteers tendto have lower SAT scores than do nonvolunteers. In addition to thesethoroughly mixed differences, the differences tend to be modest in mag-nitude. The SAT scoredifference, for example,was only 18points—less than9% of the university’s interquartile range of 220 SAT points. We thereforeobserve little evidence that the volunteers are markedly different than theirnonvolunteer classmates.Table 2 compares the attributes of volunteers assigned to the live sec-

tion versus those assigned to the online treatment. As can be seen in thetable, the random assignment of volunteers successfully led to balancingof the volunteer population into the live-only section and the online-onlysection. Those assigned to watch the class online had slightly higher prioruniversity GPAs.4 The live section also had fewer mothers who attendedcollege but did not attain a degree, but overall average education levels arecomparable across the two groups. In sum, it appears that the randomi-zation in the experiment was largely successful in balancing the live andonline delivery groups.Table 3 presents the mean test scores for the two groups of students on

each of the three examinations in the course, as well as the average of thethree scores. We prefer to use the average score because it has the smallestproblem with measurement error, and indeed, the standard errors arelowest with regard to the average score. Exams are scored on the standard0–100 point scale, and the mean of the average score on the exams is justbelow 80 points. As can be seen from the table, the preponderance of theevidence indicates that students perform better in the live setting than inthe online setting, though the raw differences are uneven and statisticallyinsignificant. Students in the live section tended to do better on the firstexam, the final exam, and overall, while students in the online section per-formed trivially better on the second exam. These ðbasically zeroÞ results arerelatively precisely estimated; the two-point difference in average examscores that would be statistically detectable with the observed standard er-rors is small in comparison to the five-point incentive, considered modestby the university’s Institutional Review Board, that was offered students toparticipate in the experiment. Therefore, we are confident that the statisticalpower issues associated with not recruiting a larger fraction of the class arenot responsible for the null findings reported in table 3.Controlling for covariates in an experimental setting can lead to im-

proved precision as well as help to reveal nonrandomness in small samples.

4 The difference is smaller for course completers than at the beginning of thecourse. However, there is little evidence of differential attrition; under no circum-stance are the attributes of live-section attriters different from those of online-sectionattriters at even the 30% statistical significance level.




Tab

le1

BaselineSu

mmaryStatistics:V

olun

teersversus

Non

volunteers

Students

BeginningtheSemester

Students

EndingtheSemester

Variable

Volunteer

Nonvo

lunteer

Difference

Volunteer

Nonvo

lunteer

Difference

Number

ofobservations

312

1286

296

1186

University

GPA

3.262

3.156

.106**

3.282

3.198

.084*

ð.036Þ

ð.022Þ

ð.048Þ

ð.036Þ

ð.022Þ

ð.048Þ

SATscore

1,224.589

1,242.884

218.295*

1,228.864

1,251.541

220.108*

ð8.774Þ

ð4.77Þ

ð10.623Þ

ð8.826Þ

ð4.823Þ

ð10.31Þ

ACTscore

25.776

26.482

2.706

25.921

26.648

2.727

ð.353Þ

ð.233Þ

ð.487Þ

ð.364Þ

ð.241Þ

ð.5Þ

HighschoolGPA

3.743

3.649

.094

3.762

3.688

.074

ð.053Þ

ð.03Þ

ð.067Þ

ð.054Þ

ð.031Þ

ð.067Þ

Fem

ale

.546

.496

.05

.541

.487

.053

ð.028Þ

ð.014Þ

ð.032Þ

ð.029Þ

ð.015Þ

ð.032Þ

Black

.115

.087

.028

.108

.074

.034*

ð.018Þ

ð.008Þ

ð.018Þ

ð.018Þ

ð.008Þ

ð.018Þ

White

.610

.638

2.028

.622

.648

2.026

ð.028Þ

ð.013Þ

ð.03Þ

ð.028Þ

ð.01 4

Þð.0

31Þ

770



Asian

.109

.098

.011

.108

.102

.006

ð.018Þ

ð.008Þ

ð.019Þ

ð.018Þ

ð.009Þ

ð.02Þ

Hispanic

.115

.135

2.02

.111

.136

2.024

ð.018Þ

ð.01Þ

ð.021Þ

ð.018Þ

ð.01Þ

ð.022Þ

Mother

attended

highschoolonly

.193

.181

.011

.193

.181

.011

ð.023Þ

ð.011Þ

ð.025Þ

ð.023Þ

ð.011Þ

ð.025Þ

Mother

attended

somecollege

.176

.179

2.004

.176

.179

2.004

ð.022Þ

ð.011Þ

ð.025Þ

ð.022Þ

ð.011Þ

ð.025Þ

Mother

graduated

from

college

.301

.401

2.100***

.301

.401

2.100***

ð.027Þ

ð.015Þ

ð.032Þ

ð.027Þ

ð.015Þ

ð.032Þ

Mother

earned

graduatedegree

.223

.195

.028

.223

.195

.028

ð.024Þ

ð.012Þ

ð.026Þ

ð.024Þ

ð.012Þ

ð.026Þ

Mother’seducationunknown

.054

.042

.012

.054

.042

.012

ð.057Þ

ð.029Þ

ð.01 1

Þð.0

57Þ

ð.029Þ

ð.011Þ

NOTE.—

Sixteenvo

lunteersand100nonvo

lunteersdropped

thecourseduringthesemester.Questionsregardingmaternaleducationwereasked

duringthefinalexam

ination,so

weonly

havethesevariablesforstudentswhocompletedthecourse;therefore,thefirstandsecondsetsofcolumnsareidenticalforthesevariables.Standarderrorsarepresentedin

parentheses

beneath

means.

*Statisticallysign

ificantat

the10%

level.

**Statisticallysign

ificantat

the5%

level.

***Statisticallysign

ificantat

the1%

level.

771



Tab

le2

BaselineSu

mmaryStatistics:V

olun

teersAssigne

dto

LiveSectionversus

Volun

teersAssigne

dto

OnlineSection

Students

BeginningtheSemester

Students

EndingtheSemester

Variable

Live

Online

Difference

Live

Online

Difference

Number

ofobservations

97215

91205

University

GPA

3.138

3.321

2.183

3.193

3.321

2.128

ð.073Þ

ð.04Þ

ð.077Þ**

ð.071Þ

ð.042Þ

ð.079Þ

SATscore

1,214.133

1,228.581

214.448

1,216.447

1,228.581

212.134

ð15.944Þ

ð10.519Þ

ð18.751Þ

ð15.902Þ

ð10.519Þ

ð18.697Þ

ACTscore

25.75

25.784

2.034

2625.898

.102

ð.602Þ

ð.426Þ

ð.833Þ

ð.629Þ

ð.435Þ

ð.882Þ

HighschoolGPA

3.794

3.718

.076

3.847

3.724

.123

ð.08Þ

ð.068Þ

ð.115Þ

ð.073Þ

ð.071Þ

ð.118Þ

Fem

ale

.495

.567

2.072

.473

.571

2.098

ð.051Þ

ð.034Þ

ð.061Þ

ð.053Þ

ð.035Þ

ð.063Þ

Black

.124

.112

.012

.121

.102

.018

ð.034Þ

ð.022Þ

ð.039Þ

ð.034Þ

ð.021Þ

ð.039Þ

Asian

.093

.116

2.023

.099

.112

2.013

ð.03Þ

ð.022Þ

ð.038Þ

ð.031Þ

ð.022Þ

ð.039Þ

772



White

.639

.600

.039

.637

.615

.023

ð.049Þ

ð.033Þ

ð.060Þ

ð.051Þ

ð.034Þ

ð.061Þ

Hispanic

.082

.126

2.044

.088

.122

2.034

ð.028Þ

ð.023Þ

ð.039Þ

ð.03Þ

ð.023Þ

ð.04Þ

Mother

attended

highschoolonly

.196

.177

.019

.209

.185

.023

ð.041Þ

ð.026Þ

ð.047Þ

ð.027Þ

ð.027Þ

ð.050Þ

Mother

attended

somecollege

.093

.200

2.107

.099

.210

2.111

ð.03Þ

ð.027Þ

ð.045Þ**

ð.029Þ

ð.029Þ

ð.048Þ**

Mother

graduated

from

college

.309

.274

.035

.330

.288

.042

ð.047Þ

ð.031Þ

ð.055Þ

ð.032Þ

ð.032Þ

ð.058Þ

Mother

earned

graduatedegree

.227

.205

.022

.242

.215

.027

ð.043Þ

ð.028Þ

ð.050Þ

ð.029Þ

ð.029Þ

ð.053Þ

Mother’seducationunknown

.072

.042

.030

.077

.044

.033

ð.026Þ

ð.014Þ

ð.027Þ

ð.014Þ

ð.014Þ

ð.02 9

ÞN

OTE.—

Standarderrors

arepresentedin

parentheses

beneath

means.

**Statisticallysign

ificantat

the5%

level.

773



In the bottom row of table 3, we therefore report the results of comparisonsbetween live-only and Internet-only groups in which we control for thecovariates reported in table 2. Controlling for covariates leads to modestlysmaller standard errors and larger positive differences between the live-only and Internet-only groups; indeed, the positive differences are sta-tistically significant in the case of the final exam and average scores. There-fore, the unconditional mean comparisons reported in table 3 are likely tobe understatements of the positive effects of live-only instruction relativeto Internet-only instruction. For the remainder of this article, we use themore conservative unconditional mean comparisons, which we interpret aslower-bound estimates of the relative effects of live-only instruction.We noted earlier that 15 of the 112 students initially assigned to the live-

only treatment requested that they be removed from the experiment be-cause of scheduling problems, and our results thus far exclude these stu-dents. However, these students may be differentially selected. Therefore,in the fifth column of table 3 we include all students who were initially as-signed to the live-only group as a test to see whether these students’ in-clusion or exclusion is consequential. We find results in this intent-to-treat

Table 3Comparison of Average Test Scores for Live versus Online Instruction

Section Exam 1 Exam 2FinalExam

AverageScore

AverageScore

TreatingDefectorsas “Live”

Exam 1ExcludingAttrition

Number ofobservations 312 301 296 296 311 296

Live 84.536 76.692 75.939 79.940 79.948 85.925ð1.168Þ ð1.193Þ ð.837Þ ð.85Þ ð.876Þ ð1.093Þ

½97� ½93� ½91� ½91� ½106� ½91�Online 83.301 76.904 74.302 78.502 78.571 84.137

ð.957Þ ð.876Þ ð.799Þ ð.675Þ ð.711Þ ð.943Þ½215� ½208� ½205� ½205� ½205� ½205�

Difference 1.235 2.212 1.637 1.440 1.377 1.788ð1.626Þ ð1.534Þ ð1.426Þ ð1.209Þ ð1.179Þ ð1.592Þ

Differenceconditionalon covariates 1.981 2.270 3.000** 2.508** 2.230** 2.017

ð1.562Þ ð1.466Þ ð1.409Þ ð 1.130Þ ð1.112Þ ð1.542ÞUpper/lowerboundestimates ½1.94, 2.03� ½2.20, 2.39� ½2.86, 3.12� ½2.37, 2.61�NOTE.—The dependent variable is the exam score measured on a 0–100 point scale. Standard errors are

in parentheses beneath coefficient estimates. Numbers in square brackets are the number of observationsfor the section type ðlive or onlineÞ unless otherwise indicated.

** Statistically significant at the 5% level.

774 Figlio et al.



analysis that are quite similar to those found when these students are ex-cluded from the analysis altogether.As seen in table 2, 16 students ð6, or 6%, in the live-only group and 10,

or 5%, in the Internet-only groupÞ left our study due to dropping thecourse before the final exam, and there is a smaller difference between liveand Internet students in the university grade point average for course com-pleters than for course beginners. While there are not statistically significantdifferences in any comparison of the attributes of those who left the live-only group versus those who left the Internet-only group, suggesting thatattrition appears to be largely random, we report in the final column of ta-ble 3 the results of the first exam comparisonwhenwe limit the analysis onlyto course completers. Unsurprisingly, the set of course completers scoredtrivially better on the first exam than those who left the course, but there isno evidence of differential selection across the groups of students. Therefore,we find that the results when we limit our analysis to course completers arein the same ballpark as those when we do not limit our analysis to coursecompleters. As an alternative check, we conduct a bounding exercise inwhich we assign all attriters either a score of zero or a score of 100 on therelevant exam.5 The results of this bounding exercise, presented in the lastrow of table 3 for the three individual exams and the average exam score,suggest that no matter what we assume the test scores of attriters to be,there is little change in the results. Therefore, all available evidence sug-gests that attrition is not responsible for our findings.While the overall effect of live instruction relative to Internet delivery

is very modest and positive ðthough not statistically distinguishable fromzero in the unconditionalmean comparisonsÞ, thesemean effects maymasksubstantial differences in relative benefits of onemediumof instructionoveranother. For instance, students from different language backgrounds, ex-perience, or motivation levels might have different experiences in live ver-sus Internet-only settings. While we cannot directly measure these specifictypes of factors, we can stratify the estimated effects of live versus Internetinstruction along a few observable lines: by student race/ethnicity, sex, andprior achievement levels.6 For this last stratification, we define “high achiev-ers” as students whose prior college GPA was greater than or equal to themedian GPA, and we define “low achievers” as students whose prior collegeGPA was less than the median GPA. We report these results in table 4. The

5 This is an approach used by Krueger ð1999Þ in his analysis of the Project STARclass size experiment in Tennessee. Similar exercises have been conducted by An-grist, Bettinger, and Kremer ð2006Þ and others.

6 These are the most logical ways to stratify our data given the observed back-ground characteristics at our disposal. We were concerned that the subgroup re-sults may be mere statistical artifacts, so we attempted a variety of stratifications ofthe data. In nearly every stratification we attempted, we found that at least one sub-group had statistically significantly positive estimated effects of live-only instruction.




treatment effects reported in table 4 reflect average score differences forstudents enrolled in the live section versus those enrolled in the online sec-tion. We observe that for all racial/ethnic groups, for both male and femalestudents, and for both high and low achievers, the average test score is higherfor the set of students in live instruction versus those in online instruction.Importantly, in a number of cases this difference is statistically significant,and some of the estimated differences are large in magnitude. Most notably,the average test score grade for Hispanic students is dramatically higher inthe case of live instruction. In addition, the estimated live instruction advan-tage is statistically significantly different from zero for male students and forlow achievers. ðWe should point out, however, that the only subgroup dif-ference that is statistically significant, given our small sample size, is the oneregarding race and ethnicity.Þ While it is premature to definitely ascribe amechanism through which differential effects on outcomes may be operat-ing, we can propose a few. For instance, perhaps low-achieving and malestudents are tempted to defer instruction and cram for an exam in online-only classroom experiences or perhaps language-minority students have in-creased difficulty with listening to lectures in an Internet setting. While wedid not explicitly test these mechanisms, the results for the various sub-groups indicate that future experimentation that pays particularly close at-tention to potentially sensitive student subgroupsmay be highly informative.One possible threat to the validity of this experiment involves the po-

tential for contamination. While it was impossible for students not se-lected to be in the live section to attend the live lectures, it was certainlypossible for experimental students to surreptitiously view online lectureseven though they could not do so using their own accounts. Indeed, whilewe have no way of knowing how prevalent this behavior was, it is likelythat at least some of the live-only students did this; only 32% of “live-only” students attended at least 90% of the live lectures,7 and 36% at-tended fewer than 20% of the live lectures! ðFig. 1 presents a density plotof the live lecture attendance for the live-only students.Þ It is not clearwhether this noncompliance would bias our estimates upward or down-ward. On the one hand, if the true effect of live instruction is positive,especially for some subgroups, the fact that we could not fully prevent“live-only” students from watching classes on the Internet using a friend’saccount may mean that our results understate the true effects of live-classattendance.On the other hand, the potential contamination could upward bias our

results if our live-only treatment is really better thought of as a hybridlive-plus-Internet treatment. There is, however, reason to believe that thelive-only treatment is different from the traditional live-plus-Internet hy-brid that the 77 nonparticipant students registered to the live section ex-

7 Students can attend a maximum of 45 lectures.

776 Figlio et al.



perienced. Figure 1 compares the distribution of live-lecture attendancefor the live-only group versus the live-plus-Internet group of students whodid not participate in the experiment, and it shows that the live-only stu-dents attended appreciably more live lectures ðan average of 21.6 live lec-turesÞ than the live-plus-Internet students ðwho averaged 13.6 live lec-turesÞ. Unfortunately, it is impossible to know how many online lectureswere viewed by live-only students because they were officially blockedfrom downloading the lectures. Therefore, it is clearly the case that be-ing officially restricted to only view lectures live strongly influenced thelikelihood that the live-only participants would indeed receive their ma-terial delivery in the live format.8 Hence, though we cannot know for

Table 4Heterogeneous Effects of Live Instruction versus Online Instruction

SubgroupResults by Racial/Ethnic Group

Results byStudent’s Sex

Results byAchievement Level

White students 1.117ð1.436Þ

Black students 2.828ð3.239Þ½.632�

Hispanic students 11.276***ð3.587Þ½.014�

Asian students 4.319ð3.590Þ½.428�

Male students 3.480**ð1.680Þ

Female students 1.780ð1.576Þ½.462�

Low achievers 4.054***ð1.536Þ

High achievers 1.169ð1.635Þ½.205�

R2 .386 .370 .402

NOTE.—Dependent variable is the average test score measured on a 0–100 point scale. Number ofobservations: 296. Standard errors are in parentheses beneath coefficient estimates; p-values are in squarebrackets.

** Statistically significant at the 5% level.*** Statistically significant at the 1% level.

8 It is also the case that many of the students who are observed rarely coming toclass might actually not ever view the lectures at all. The university has several com-peting lecture note-taking services that are extremely popular with students. In addi-tion, we find in our present data that students who attend fewer live lectures do sub-




certain, we suspect that contamination of our experiment due to partici-pating students watching Internet lectures is not a major force driving ourfindings.It may also be the case that live-only participants benefit from having

other classmates in the live section who are better or more motivated stu-dents and who could therefore have positive peer effects. ðThis could hap-pen if students who enroll in the live section are systematically of higherability than those who enroll in the online section.Þ Since section regis-tration had historically had no bearing on whether a student could attendthe live lecture, we believe that it is unlikely that the nonparticipants inthe live section would be much different from the nonparticipants in theonline section, and indeed, this appears to be the case. In fact, if anything,the nonparticipants in the live section have lower observables than thosein the online section. For instance, when comparing students who did notparticipate in the experiment, we find that the mean SAT score for thosein the live section is 1,197 as comparedwith 1,245 in the online section. Forthe same two groups, maternal education is lower for live-section regis-trants than for online-section registrants ð24% vs. 18% had mothers with

FIG. 1.—End-of-semester attendance cumulative density functions for live-onlyðlower linesÞ and live-plus-Internet ðupper linesÞ students.

stantially worse on the examinations, suggesting that many of those who attend fewerlive lectures are not substituting surreptitiously downloaded Internet lectures for thelive lectures they eschew.

778 Figlio et al.



only a high school degreeÞ. In summary, there exists no evidence that thelive-only participants’ scores are being positively influenced by an im-proved peer group of nonparticipating students who insisted on being partof the live section.

III. Limitations to External Validity and Recommendationsfor Future Experiments

This article presents the first experimental evidence of the relative effi-cacy of live versus Internet-only instruction in a higher education setting.While our analysis has a high degree of internal validity, there are a numberof key reasons why we believe that our results should be taken as sugges-tive rather than conclusive and why we recommend that further experi-mentation in a variety of other settings take place before one can draw de-finitive conclusions about the effects of different modes of lecture delivery.One reason that the external validity of our analysis is limited is that the

volunteers we recruited may not reflect the overall population of studentsenrolled in the class. While our experimental live-only and Internet-onlygroups are balanced along a large number of dimensions, participation inthe experiment was voluntary. Moreover, the incentive used to induce par-ticipation was extra credit on the final course grade. There is no reason tobelieve that responsiveness to this incentive is exogenously determined,and in fact, one can easily tell stories about which types of students mightbe willing to participate in the experiment. Specifically, one might reason-ably expect that students who are motivated to achieve high grades but arerelatively concerned about their ability to earn high grades might be the stu-dents most responsive to a participation-for-points incentive, and this couldexplain why our volunteer participant group has slightly lower levels ofSAT scores but slightly higher pre-course university grades as well as ðin-significantlyÞ higher high school grades. If people who are especially grademotivated respond differently to live versus Internet instruction, then ourexperiment has less to say about the typical student enrolled in a very largeintroductory course.This concern yields important lessons for future experiments on this

topic. While the Institutional Review Board and the university culture atthe university in question did not permit us to randomize all students intolive versus Internet-only lecture categories, it will be important for futureexperiments to attempt to study the entire set of students who select into agiven class rather than a subsample of students. In the event that this is notpossible, future experiments could improve upon the external validity ofthe present study by seeking to obtain a higher participant rate. The uni-versity required us to take a more passive role than would be desirable inthe recruitment of students into the study; we could not, for instance, offerfinancial or in-kind incentives to increase participation, and we were




limited in the number of times that we could attempt to recruit students.Settings with fewer such encumbrances might yield higher degrees of ex-ternal validity.Other external validity issues associated with this experiment would

not be solved even had we been able to randomly assign 100% of the stu-dents in the introductory microeconomics class to live-only or Internet-only lecture groups. One involves the specific university setting: the uni-versity in question is one where very large lecture classes are the norm forvirtually all freshman- and sophomore-level courses, across all fields, andmoreover, most of the core courses for students majoring in business areoffered on this electronic platform. The results of an experiment in thistype of university setting may not generalize to other university settingswhere students have less experience with large auditorium lectures and elec-tronically delivered lectures. Ideally, future experiments of this nature willtake place in a wider variety of institutional settings, so that we can begin tounderstand the degree to which the findings generalize across settings. Thisis also the case because the university in question is one of the most selectivestate universities in the United States; the results may not generalize to open-enrollment institutions or those where students are drawn from lower inthe ability and achievement distribution. Of course, given that our findingssuggest that lower-ability students are a subgroup potentially most harmedby Internet delivery, the results might be particularly relevant for less selec-tive institutions.In addition, introductory economics courses are generally delivered in

traditional lecture settings even at small institutions with modest classsizes. In someways, onemight expect that this would be the type of subjectmatter where live instruction may be the least beneficial, as members ofthe class tend to be relatively passive consumers of material in the lecturesetting. It may be the case that live classes might be relatively more ben-eficial in other types of courses, with a greater role for interactive activitiesin the classroom. In such a case, our results might be an understatement ofthe effects of live versus Internet class delivery in other contexts. On theother hand, introductory economics has a number of topics that build uponone another, and it relies more heavily on technical prerequisites thanmany other subjects do; it could be that the disciplined pacing that comeswith live-only lectures might be relatively beneficial in this type of con-text, implying that the effects in other fields where pacing is less crucialmay be smaller. It is therefore important to conduct similar experimentsin a wider range of subjects, and for classes with different levels of stu-dent involvement and interactivity, in order to develop general conclu-sions about the relative efficacy of live versus Internet-based instruction.Therefore, while our study represents the first causal evidence of the ef-fects of live versus Internet-based instruction in a university course de-

780 Figlio et al.



livery setting, it can only be seen as a beginning step toward understandingthe generalized effects of different methods of instructional delivery.Finally, while not a threat to external validity and generalizability, our

subgroup-specific findings indicate that some student populations maybe particularly sensitive to the mechanism through which lecture materialis delivered. Language-minority students might have more difficulty fol-lowing recorded lectures, and some students may be relatively less disci-plined in keeping up with the pace of the course when procrastination ismore possible. ðThis might be an explanation for the relatively large esti-mated effects observed for male and lower-achievement students, thoughwe cannot say for certain that this is the reason.Þ Therefore, future experi-mentation that could directly test for some of these potential mechanismscould be highly valuable. For example, if one is interested in seeing whetherdelayed lecture viewing is a potential mechanism generating lower outcomesfor Internet-only students, one might design an experiment in which stu-dents were required to download ðor maybe viewÞ lectures within a certainnumber of days following lecture recording. In general, it would be highlyvaluable to look more deeply at the potential causal mechanisms throughwhich different lecture delivery mechanisms might affect student learning.Additional survey and qualitative work on questions such as ways in whichstudents engage with the course material, interact with the instructors andtheir peers, pay attention to lectures, and study for examinations would behighly valuable and could help universities and professors refine their coursesand instructional delivery to maximize student learning.

V. Conclusion

Given the clear scale economies associated with online instruction, ed-ucational institutions are actively incorporating online instruction into theirportfolios. While our results are not definitive, and numerous potential ex-ternal validity concerns exist, our experimental, apples-to-apples compari-son nonetheless indicates that a rush to online education may come at moreof a cost than educators may suspect.Itmay still be the case that the educational costs associatedwith Internet-

only classes are outweighed by the economies of scale associated with offer-ing traditional classes over the Internet. A benefit-cost analysis of Internet-based lecture deliverywouldweigh the educational costs of reduced humancapital against the financial benefit of exploiting scale economies.On the financial benefit side, universities might potentially enjoy tre-

mendous financial savings by recording lectures and replaying them formultiple years with an instructor taking a less hands-on approach. Buteven assuming courses could be devised and delivered anew each term,there may be financial advantages to offering Internet delivery of courses.The university where the experiment was conducted currently administers




some completely Internet-only distance-education courses in addition tothe hybrid live-Internet courses like the one that formed the basis for thisexperiment. At this university, the average administrative expenditure ðnotcounting instructor compensationÞ per student per class was about $270in 2009–10. These calculations include technical staffing, assessment ad-ministration, camera operation, and other expenditures. None of these dis-tance classes are offered at the ideal scale for this thought experiment, butif a fully scaled-up course of 1,000 students would cost half that at $135,000to deliver ða reasonable estimate based on the staffing and technical sce-narios that the university describesÞ, then whether this is a large or smallnumber would depend on the costs of staffing live courses versus Internetcourses. If five 200-student live lectures represent 1.5 faculty positions and a1,000-student Internet lecture course represents one-half of a faculty po-sition,9 then under these assumptions an Internet-only course would be costeffective—assuming there is no cost to reduced human capital—for facultysalaries ðincluding benefitsÞ exceeding $135,000 ðtranslating to a salary ofabout $111,600 given the university’s fringe benefit rateÞ.10 According tothe American Association of University Professors faculty salary surveydata for 4-year colleges and universities in 2010–11, in only three of 31broad fields is the average full professor salary greater than this value, andthe field with the highest average salaries for instructors ðlegal professionsand studiesÞ averaged $64,785.11 Therefore, in order for Internet-baseddelivery of traditional lectures to be financially beneficial, a college or uni-versity would typically need to provide fewer support services for Internet-based classes or use the same lectures for a number of yearswith a lower levelof faculty participation.These calculations make clear that, setting aside educational conse-

quences, Internet-based courses can only make financial sense either atuniversities with low faculty teaching loads, or for fields with relativelyhigh faculty salaries, or at other institutions/fields if a course is developedonce and replayed with much lower levels of faculty involvement. The re-sults of this experiment suggest that the break-even point at which Internet-only, lecture-based classes pass a benefit-cost test is higher than the figuresdescribed above. How much higher this threshold would be depends on

9 We assume that a university would either need to count a very large course asmore than a regular course or to compensate the facultymember for the extra burdenof administering a course with many students. The university in question does thelatter, and it compensates the faculty member an extra $50 per student in distance-education courses.

10 This is likely a lower bound of this critical value, as there would be start-upcosts associated with investment in the infrastructure at the institution.

11 The critical value is lower than the average salary of economics faculty mem-bers at research universities, though well above the average salary of economics in-structors at these universities.

782 Figlio et al.



the costs to students associated with diminished human capital. And thethreshold would be higher still if an Internet-based course with fewer stu-dent services and less professor contact would yield worse results than thehigh-contact, Internet-based course described in this study. The degree towhich this is true is not yet known.At the least, our findings indicate that much more experimentation is

necessary before one can credibly declare that online education is peer totraditional live classroom instruction, let alone superior to live instruction.While online instruction may be more economical in some circumstancesto deliver than live instruction, our results indicate that—consistent with afundamental lesson of principles of microeconomics—the lunch may beless free than many might believe.

References

Allen, I. Elaine, and Jeff Seaman. 2006. Making the grade: Online edu-cation in the United States, 2006. Boston: Sloan Consortium.

Angrist, Josh, Eric Bettinger, and Michael Kremer. 2006. Long-term educa-tional consequences of secondary school vouchers: Evidence from ad-ministrative records in Colombia. American Economic Review 96, no. 3:847–62.

Bettinger, Eric, and Bridget T. Long. 2007. Institutional responses to re-duce inequalities in college outcomes: Remedial and developmental coursesin higher education. In Economic inequality and higher education: Access,persistence, and success, ed. StacyDickert-Conlin andRossRubenstein.NewYork: Russell Sage Foundation.

Brown, Byron W., and Carl E. Liedholm. 2002. Can web courses replacethe classroom in principles of microeconomics? American EconomicReview 92, no. 2:444–48.

Donovan, Colleen, David Figlio, and Mark Rush. 2006. Cramming: Theeffects of school accountability on college-bound students. Workingpaper, National Bureau of Economic Research, Cambridge, MA.

Kane, Thomas, and Peter Orszag. 2003. Funding restrictions at publicuniversities: Effects and policy implications. Working paper, BrookingsInstitution, Washington, DC.

Krueger, Alan. 1999. Experimental estimates of education production func-tions.Quarterly Journal of Economics 114, no. 2:497–532.

Mortenson, Thomas. 2005. State tax fund appropriations for higher educa-tion: FY1961 toFY2005.Report, PostsecondaryEducationOpportunity,Washington, DC.

Navarro, Peter, and Judy Shoemaker. 2000. Policy issues in the teaching ofeconomics in cyberspace: Research design, course design, and researchresults. Contemporary Economic Policy 18, no. 3:359–66.




US Department of Education. 2009. Evaluation of evidence-based prac-tices in online learning: A meta-analysis and review of online learningstudies. Washington, DC: US Department of Education, Office of Plan-ning, Evaluation, and Policy Development.

Zhang,Dongsong. 2005. Interactivemultimedia-based e-learning: A study ofeffectiveness. American Journal of Distance Education 19, no. 3:149–62.

Zhang, Dongsong, Lina Zhou, Robert O. Briggs, and Jay F. NunamakerJr. 2006. Instructional video in e-learning: Assessing the impact of in-teractive video on learning effectiveness. Information and Management43, no. 1:15–27.

784 Figlio et al.



is it live or is it internet? experimental estimates of the effects of online instruction on student...

Documents