the uiuc scale experiments by dan stone (with help from lanny arvan & burks oakley) univ. of...

The UIUC SCALE Experiments

By Dan Stone (with help from Lanny Arvan & Burks Oakley)

Univ. of Illinois at Urbana-Champaign, 1988 – 2000Univ. of Kentucky, beginning 8/2000

Presentation Outline

Thanks to …!Motivation Introduction to SCALE & ALNSCALE Research Design / EvaluationSCALE ResultsThe Bottom Line: What

creates successful ALN?

Thanks to:Bob Jensen, Lanny Arvan (UIUC), Burks Oakley (UIUC) for their enthusiasm, risk-taking, leadership & innovation

More thanks & copyright disclosure: portions of this presentation

are taken from presentations by Lanny Arvan & Burks Oakley

portions of this presentation are taken from the SCALE & CET websites

See citations at end of presentation

Motivation

Huge investments in asynchronous learning network (ALN) programs (e.g., Penn State World Campus, Stanford-Online, Duke Global Exec MBA)

SCALE project: over 80 on-line courses

Motivation

Do the SCALE results suggest that ALNs can improve: educational effectiveness? efficiency? access to higher education?

What makes for successful

on-line courses & programs?

Motivation

My interest in SCALE & ALN: Teach accounting systems &technology classes Use pedagogy (technology) to

teach some course content (technology)

Adopted computer aided instruction module in 1991, adopted Powerpoint in 1992In the first group of adopters of First-Class at UIUC (1994)On-line grading & email software for 6 yearsOn-line, web-based course materials for 3 years

Motivation

My interest in SCALE & ALN:Lingering doubts … if I weren’t teaching “technology as content” would “technology as pedagogy” be worth the trouble? “Recent surveys of the instructional use of

information technology in higher education clearly indicate that there have been no significant gains in either productivity improvement or pedagogical enhancement” (Kenneth Green, 1997, “The Campus Computing Project”).

Clark 1991, “Media do not influence learning.” “ALN teaching is arduous.”

Introduction to SCALE & ALN

Sloan Center for Asynchronous Learning Environments (SCALE) (1995-1999) Origin: $2.1 million grant from Alfred P. Sloan

Foundation + UI matching Goal: Restructure undergraduate courses to

integrate asynchronous learning network (ALN) methods

20 major projects in 1995, now > 80 projects 9 “efficiency” projects – designed to lower instructional costs per student


SCALE replaced by UIUC funded:

Center for Educational Technologies (CET) Lanny Arvan, Director

UI-Online Led by Sylvia Manning

(Chancellor)


What is ALN? In-class learning is “synchronous.” Contrast synchronous in-class learning

with “Site independent learning” (distance

education) –courses delivered largely or entirely online

Adapted from Lanny Arvan presentation (the Scale Evaluation)

ALN Goals & Technologies

Enhanced learning opportunities Interactive course materials,

e.g., PCs & WWW Simulations, multimedia,

visualization, e.g., streaming video, Java, Virtual reality (e.g., digital video cases – “WWHP A CD Case study” (http://www.cba.uiuc.edu/lab/BrownBags/WWHP.pps)

Real-time homework, quizzes & grade information, e.g., on-line, interactive testing, on-line grades Adapted from Burks Oakley

presentation (“E-Learning”)

ALN Goals & Technologies

Improve access to people & information Subject matter experts (faculty, TA’s) Other students (peer-peer & group

interactions) E.g., Conferencing, email E.g., Course management

software (i.e., CourseInfo,

WebCT, Mallard). Adapted from Burks Oakley presentation (“E-Learning”)

ALN Example Applications

Econ 300 (Intermediate Microecon) – Lanny Arvan SCALE efficiency project A required (disliked) course for all

UIUC business majors ALN goals:

More feedback & learning opportunities (Mallard on-line testing)

Increase interactions (on-line) with faculty & peers (First Class)

Disincentives for cram-for-test-&-nothing-else mentality (homework & quizzes count)

Adapted from Burks Oakley presentation (“E-Learning”)

ALN Example – Econ 300 First Class Menu

SCALE Research Design / Evaluation

Separate University-sponsored evaluation team & unitResearch design: Case & field studiesEven when available, ALN vs. nonALN control groups are problematic due to: Student self-selection into courses & programs Instructor self-selection into using technology Differing evaluative methods & processes (e.g.,

differing rules for on-line vs. traditional exams, quizzes) (I.e., mixing “media” & “method” – Clark)


Most extensive evaluation is of 1st year classes (19 courses)

An impression evaluation team sought evidence supporting increased efficiency & effectiveness (after spending $3M – what else could UIUC do?)


Data Source Sample Size

Pre-course Student Surveys 1,438 Students in 16 Courses

Student Group Interviews 10 Groups

Teaching Assistant Group Interviews 2 Groups

End-of-Semester Student Surveys 1,118 Students in 17 Courses

Pre-Course Faculty Interviews 26

End-of-Semester Faculty Surveys 23

End-of-Semester Faculty Interviews 6

Computer Conference Monitoring 5 Courses

Adapted f from SCALE website

SCALE Evaluation Results

ALN...Increased communications Surveys: 51% of students reported an

increase in communication with instructor & 43% with other students.

40% of students reported an increase in the quality of their interaction with the instructor.

Improved access to information – "Information when you want/need it." "Personal control of information," "quick-response from peers & students."



ALN... Enhanced learning environment “A new and exciting way to learn," “More prepared

for class," “Time to learn out of class," "work at own pace."

"On-line homework was a great experience," “On-line quizzes were "a good way to study for exams."

70% would take another course using computer conferencing.

75% rated overall experience with computer conferencing good, very good, or excellent.

60% reported increase in learning due to computer conferencing.



ALN...Improved computing & technology knowledge "Increased my knowledge of and confidence

with computers." “I’m less apprehensive about computers"

70% indicated increased familiarity with computers.

Adapted from SCALE website


Success of ALN depended on...Student training 75% found conferencing software

easy or somewhat easy to use. Interviews: Many students wanted better

training. E.g., "Require a mandatory class on system use."

Student effort & commitment 60% used computer conferencing 1 time week. Females used conferencing slightly more than did

males (not stat significant) “Class member comments helped me learn."



Success of ALN depended on...Incentives, I.e., what’s in it for the student? Students: "Must give incentives to use the software." “Don’t just tack ALN on to an existing course."

Accessible computers Lack of computer access = most frequent complaint "It was hard to find an available computer." "The [modem] lines were always busy." "Get more computers or get rid of [ALN component]."



Success of ALN depended on...System availability & support "I couldn't login,

so I stopped using it."

"Too many system crashes"


SCALE Efficiency Project Results

Did ALN improve course efficiency?From whose perspective? (instructor vs. university) Instructor time omitted as a cost … despite

acknowledgement that “ALN teaching is arduous.” (instructor interview). Increased instructor-student contact time in ALN classes.

Never for small or writing-intensive classesFor each of the 3 examples that I use: No difference in exam performance for ALN vs.

traditional sectionsAdapted from SCALE website


Did ALN improve course efficiency?Introductory Statistics (100) Development cost funded by grant On-going cost of ALN initiatives = $1,000

peer tutor per semester ALN class sizes 50 to 100% larger

than traditional sections

(e.g., 154 vs. 101 students)Adapted from SCALE website


Did ALN improve course efficiency?Econ 300 Section sizes increased from 60 to 180 Development cost = ~ $56,224

Introductory Spanish (210) 1 time development cost =

~ $15,000 Class section sizes doubled with ALN (19 to 38)


An ALN success story (LEEP)


UIUC School of Library & Info. ScienceOnline version of on-campus MS degree (not a SCALE project) – Library Education Experimental

Program (LEEP) - accredited by the ALA

Began with 31 students in Fall 1996 (including students in Alaska & Vermont)

Currently 125 students enrolled (~ 1/2 in-state)

An ALN success story (LEEP)

As of 1999, 96 students completed MS degree (from 37 states, rural Illinois, Virgin Islands, France, Colombia, Japan, & Thailand)All faculty teach 1 online course each academic yearTo serve demand for LEEP, GSLS College: added 4.5 FTE faculty budget has grown > $750,000 New building in progress!



Do the SCALE results suggest that ALNs can improve access to higher education? Yes!!!! LEEP program & SCALE efficiency

projects


Thanks to …!Motivation Introduction to SCALE & ALNSCALE Research Design / EvaluationResultsThe Bottom Line: What


The Bottom Line: What makes for a successful ALN?

Support from the top for technological innovation (e.g., LEEP) Incentives for participation among

faculty & departments Pedagogically sound content Commitment to quality pedagogy & content

Enthusiastic & committed faculty & support staff Are faculty early (15%), middle (70%), or late (15%)

adopters? - Incentives differ by group Helpful: wired, computer literate students & faculty



Thoughtful consideration of copyright and ownership issues (faculty vs. University, publishers vs. University, faculty vs. students) Who owns what? Accessibility vs. ownership (see

recent Chronicle of Education articles)

Support & maintenance Development / programming Access to student computing is essential (& often problematic) Supported, mature products



Marketing to current & potential students, current & potential faculty, University administrators What’s in it for the students?

faculty? administrators?

What’s the goal? Improved class availability? Technology as

content? Richer content? Adapted from Burks Oakley presentation (“E-Learning”)

References & resources

Clark, Richard E., 1991, “When researchers swim upstream: reflections on an unpopular argument about learning from media,” Educational Technology (February), 34-40.

Scale website: http://w3.scale.uiuc.edu/scale/index.html

UI on-line: http://www.online.uillinois.edu/

Web-board tutorial: http://w3.scale.uiuc.edu/scale/docs/using_webboard/index.html

LEEP program: http://alexia.lis.uiuc.edu/gslis/degrees/leep.html

UIUC Center for Educational Technologies (CET): http://www.cet.uiuc.edu/

Webboard: http://webboard.oreilly.com/

WebCT: http://www.cet.uiuc.edu/webct/index.asp

Dan Stone

Email: [email protected]

WWW: http://www.cba.uiuc.edu/d-stone1/d-stone1.html

Phone: 257-3788

the uiuc scale experiments by dan stone (with help from lanny arvan & burks oakley) univ. of...

Documents

scale evaluation slide

scale results

scale aln sloan center

successful aln

uiuc scale experiments

aln teaching

scale cet websites

end of presentation