1

A Course on Digital Video Special Effects

Irfan A. Essa and Gabriel J. Brostow

GVU Center / College of ComputingAtlanta, GA 30332-0280, USA{irfan|brostow}@cc.gatech.edu

Abstract

We present a brief overview of our course "Digital Video Special Effects” taught in the College ofComputing at Georgia Institute of Technology. This course is offered as an elective at theundergraduate level every year and is very popular with the undergraduate students. Here wepresent the goals that lead to this course and a discussion of the material that is covered. Wealso present what the students are expected to do and how they learn to “deal” with digital videoas a medium. The merger of the technical aspects of video processing, with the entertainmentaspects of generating special effects make this a very unique class. It aims to make analysis,processing, and synthesis of video ubiquitous to the students, while teaching them the technicalaspects of what can be done with video.

1 Introduction

In this paper, we present a new course that we offer at the undergraduate level in the College ofComputing at Georgia Institute of Technology. This course, entitled “Digital Video SpecialEffects,” is designed to teach students video and audio manipulation, processing, analysis, andsynthesis. Students undertake a deep technical study of how video special effects aregenerated and use the material taught in this class to generate a short video production toshowcase their own special effect. This class is in its second year; the pilot offering was inspring 1999 as a quarter long class. In spring 2000, it is offered as a semester long (GeorgiaTech moved to the semester system in academic year 1999-2000) and is now a part of ourcollege’s undergraduate graphics specialization.

This paper describes the design ideas that lead to this class, a brief description of what topicsare covered, and what students are expected to learn and do during the course of the term. Wealso discuss some of the development environments that are being used in this class by thestudents and describe the equipment that is provided. In addition, we will describe the projectsfrom the last year’s class as that should provide ample understanding of why this class is well-liked by the undergraduates. Finally, we present a discussion of how we are planning to improvethis offering in the upcoming years.

2 Goals

The main goal of this course is to teach undergraduate students concepts associated with digitalmedia, specifically digital video. Students learn the basics of digital video/audio while studyingspecial effects production. This course offers undergraduates the opportunity to apply anddeepen their core computer science and engineering knowledge by extending that knowledge tomanipulation of digital video. Creating a short segment with special effects teaches thestudents to apply their graphics, imaging, and software engineering expertise. In generating this

2

segment, the students exercise their technical abilities to express their creative skills. Thetechnical content covered in this class includes:

• Methods for digital video and audio manipulation and processing: This includes issuesranging from how digital video (imagery and sound) are encoded/represented in acomputer and how these signals can be processed, analyzed, and resynthesized. This iswhere many of the computer vision aspects are covered.

• 2D and 3D graphics; camera placement and movement, lighting, and shadows: Thisincludes a study of how synthetic scenes and objects are generated and relates to theissues of how virtual cameras are represented, moved, and the role lighting andshadows play in the generation of scenes.

• Merging of video, audio, and graphics streams: This is a study of how real imagery canbe merged with graphics OR how audio streams are merged to generate syntheticscenes

• Narrative and story telling: This is an essential part of the class, as the students need togenerate a short segment with a story, to showcase their technical effect. For this westudy different existing literature on cinematography and storyboarding.

• Putting it all together to make a final production: Finally, we undertake a study of how tobring in all the pieces together into a final production. This includes a study of shot-selections and editing.

Students who take this course: Upper class undergraduate students at Georgia Tech takethis class as a senior year elective. Graduate students are allowed with some additional workas requirement to get graduate credit. This class is designed with the interests of students fromcomputing, engineering, and other fields in mind, serving as a very effective bridge betweendifferent areas.

This class introduces these students to technical concepts associated with digital video andaudio. Students learn about video manipulation, processing, and editing, for the purpose ofgenerating a short segment with their own special effects. Students develop "plug-ins" forcommercial packages using their own code to generate the effects. Plug-in writing requires thatstudents learn practical aspects of digital image processing, computer graphics, and productionteamwork.

To date, in addition to computing students, engineering students who have been exposed toclasses in signals and systems and have some programming experience also take this class.These students are interested in extending their technical skills in the area of digital media,while learning more about graphics and system development. Our Literature, Communicationand Culture department at has several courses dealing with Video/Film. The students fromthese programs, though lacking in technical abilities, are also allowed to take this course withspecific permission from the instructor, as we cover both the technical and non-technical issuesof digital video. The insight of these artisticallyoriented students is valuable for in-classdiscussions and these students work to incorporate artistic expression in their group work.Additionally, the students lacking in technical background pick up valuable technical expertise inthe course material of the class. The only pre-requisite for this class is programming

3

experience. Coursework in computer vision, and graphics, and signals and systems isconsidered a bonus. Special enrollments are allowed on a case-by-case basis.

Due to space and resource limitation, we have restricted the enrollment in this class to 21 inspring 1999, and 30 in the spring 2000 term. In both terms, the class was fully subscribedwithin the first day of advance registration. We have had waiting lists of up to 20 students bothyears. The students who have taken this class love it, as is evident by their participation andenthusiasm.

3 Syllabus

The syllabus of this course includes:

Description Lectures

Introduction to new paradigmsIntroduce the class to digital video and discuss what it means towork with digital video and film.

1

Representations of Digital MediaHow is digital video represented in acomputer and how can it be manipulated/processed. Discussionincludes both video and audio.

2

Computing MethodologiesHow to process, manipulate, and analyze video. Present toolkitsthat will be used. Tutorials.

2

Introduction to Signal Processing (and Computer Vision)A short introduction to the basic principles of Signal Processing.

1

Introduction to Audio, Image, and Video Processing.Introduce filtering, convolving, and smoothing, with examples forboth video and audio.

2

Compositing, Rotoscoping, and Chroma-keying 1

CamerasPerspective and Orthographic Projections, Camera parameters,motion, calibration, etc.

2

Introduction to Computer Graphics 2

Lighting, Shadows, Texture 1

Blending, Crossfading 1

Audio Effects / Mixing 1

Morphing (2D and 3D) 1

4

Merging Graphics with Video 1

Matched Moves 1

Tracking and Pose-Estimation 1

Editing/Storyboarding, Putting it together 1

Guest SpeakersIn spring 2000, from Sportvision, Rhythm & Hues, Adobe, etc.

3

CritiquesIn-class evaluation and critique of well-known effects from featurefilms.

2

Case StudiesIn spring 2000, “Pyramids”, [Burt & Adelson 1983a, 1983b] “Tourinto the Picture” [Horry et al. 1997], “View Morphing &Interpolation” [Seitz & Dyer 1996; Chen & Williams 1993].

4

Class Presentations and Discussions of Projects 4

The syllabus is designed to ensure that students pick up the basics of signal processing,computer vision, and computer graphics, as these are the important ingredients in the study ofspecial effects generation. In addition to lectures on various topics listed above, guest lecturesfrom various experts in the fields of special effects are also presented. In addition, some portionof the class is dedicated to presentations of case studies of some of the recent work that maybe appropriate. Finally, several lecture sessions are reserved for group discussion of ongoingprojects, and to help the students work on their specific project ideas one-on-one with theInstructor and the TA.

At the end of each semester, students present their productions to the general community tohighlight their technical and artistic skills. Each student creates a CD-ROM and Digital Video(DV) tape containing all the work accomplished during the term. This will be an artifact for theirportfolios, which they can share with their prospective employers and prospective graduateschools.

In this class, we use various educational technology innovations to enhance the classroomlectures and the out-of-class the classroom interactions between students. The class meets ina state-of-the-art classroom, equipped with video facilities to show videos (VHS, DVD,QuickTime, AVI, etc.). Additionally, the class lectures are captured using the Classroom 2000system [Abowd 1999]. A student can then review these lecture by seeing all the slides (shown inFigure 1) and the video of the lecture at a later date. To facilitate out-of-the-classroominteractions and to provide a simple way for students to work on collaborative documents, a Co-Web (called SWIKI) is used. This Co-Web is the primary repository of all the documentationthat is generated by the students for projects they undertake during the semester.

5

Books: A significant problem with this course is the absence of a suitable textbook. No book isavailable that covers the technical aspects of the concepts we are interested in teaching thestudents. For this reason, we just assign one text and then recommend various other booksthat are guides for using commercially available software. During the term various readings arehanded out from different computer vision and graphics books and proceedings as appropriate.Here is the list of recommended books:

• T. Wittenburg, Visual Special Effects Toolkit in C++, 1997.

• Adobe After Effects 4.0: Classroom in a Book, 1999

• Adobe Premiere 5.0: Classroom in a Book, MacMillan Computer Pub, 1998

• B. de Leeuw, Digital Cinematography, AP Professional, 1997

• R. Shamms Mortier, After Effects 4 In Depth, The Coriolis Group, 1999.

4 Resources provided to the class

4.1 Software

Students are introduced to various existing commercial tools like Adobe After Effects, AdobePremiere, MAYA, and Sound Forge. Development toolkits for most of these products areavailable allowing students to develop plug-ins. Based on their level of expertise and interest,students choose a development environment appropriate for their project. Licenses for all thesecommercial packages are purchased to support this class. In the first year of teaching thisclass, we encouraged everyone to develop plug-ins for Premiere; however, studentscomplained about the difficulty in using the toolkit SDK for complicated processing. .

Figure 1: Snapshots of the lectures from the Classroom 2000 system. On the left is an imagefrom the lecture on Perspective transformation, and on the right is an image from the lecture on

Gaussian Pyramids(http://eclass.cc.gatech.edu/zenpad/db/listLecture.php3?Course=cs4480_00_Spring).

6

For this reason, we are now making available to the students a homegrown software system inC/C++ on Windows NT that uses the Intel Image Processing Library, Intel Computer VisionLibrary, and the Microsoft Vision SDK for more advanced video processing and analysis tasks.The students use this system to do development of their effects and then import theprocessed/synthesized video into Premiere for editing. We have seen very positive results whenstudents are given at-least some skeleton code to deal with some of the low-level details of fileformats, compression, and UI on a given platform.

4.2 Hardware

This class has a very heavy equipment demand. Thisdemand is not just limited to the computing platforms andincludes cameras, lights, and audio equipment. For thisclass we have established a complete new PC cluster withhigh-end 600MhZ Pentium III PCs. Each PC has 256 MBRam and 30 Gigabytes of local diskspace. The reason forthe large diskspace is to allow students local temporaryspace for manipulating video. An additional 50 Gigabytesof diskspace is provided on the server for students to storedata after processing.

Each workstation is equipped with a DVRaptor card, whichcaptures video in Digital Video (DV) format from a digitalcamera. We have also purchased 10 Canon Elura Mini-DV cameras exclusively for this class. These cameras aregiven to each group for the duration of the term. In additionto this equipment, students are provided tripods, speciallights, rooms with green screens, microphones, and almostany other reasonable item to help them with theirproductions.

5 Projects undertaken by students

Students in the class are expected to do one initial project and then choose a final project. Theinitial project is undertaken individually, while the final project is worked on in teams of 3-4students. In addition to the project, each student group works on a critique of a segment from afeature film and leads a short class discussion. We describe some of these projects herebriefly.

5.1 Compositing from 2000

To provide with hands-on experience of dealing with digital video and all the equipment that isavailable to the class, their first assignment is to undertake simple chromakeying. We have builttwo separate locations in our research area dedicated to help students with this project (and thefinal projects as needed). Each of these locations is equipped with a either a large blue or greenscreen. In addition, we provide the students with appropriate diffuse lighting so that they canlight the scene uniformly without harsh shadows.

The reason for this assignment is to get the students to think about both the subject matter andalso how to control the environment for appropriate effect generation. In doing this assignment,

Figure 2: View of our “studio.”

7

the students learn to use Adobe After Effects and Premiere to do the compositing and finalediting. They also learn to use the cameras and the digitizing system. Figure 3 shows someexamples of the composites done by students in the spring 2000 class.

5.2 Critiques

To provide a stronger sense of the effort that the special effects industry undertakes and tounderstand the fine nuances of special effects generation, students are required to do a criticalevaluation of a segment from a movie or some other video footage. Students study an effect oftheir choice and then present to the class their understanding of how the effect was generated.This is a very useful exercise for presenters and stimulates good class discussions. Thesecritiques are available at http://minnow.cc.gatech.edu/dvfx/173.

5.3 Projects from 1999

A main aspect of this class is the production of one to five minute video segments to showcasethe special effects that students develop for this class. This production is perhaps the high-pointof theterm, as the students work countless hours shooting raw footage, implementing theireffect, and then editing a final production. During the course of the term, we spend much timeon trying to define a project that (1) is doable in the time available and (2) has enough technicalmerit. The students are then required to present an update every two weeks on their progress(bi-weeklies). An initial screening is scheduled three weeks before the finals week, which is justfor the classmates, instructor, and TA. The final screening is in the last week of classes and isopen to the public. The spring 1999 projects presentation was very well received and there wasno standing room in the presentation room. We also received several requests for additionalscreenings after the final show.

In order to show the technical and artistic talents of the students doing these projects, here webriefly discuss some of the projects from the spring 1999 class. Figure 4 shows stills from threeof these projects.

The left still is from a piece entitled “Tech Warz” by Haliburton, Johnson, and Peon. Thesestudents attempted to track colored sticks and replace them with renderings that appear likelight sabers from a famous science fiction movie. In order to generate this effect, the studentswrote code to track the brightly colored sticks. This code worked reliably in well-lit roomswithout complex backgrounds. However, it failed to track in complex scenes. The students

Figure 3: Examples of the compositing assignment from spring 2000 class. Floating Guitar,Vanishing person, and Super Dude. Projects by Chow, Furdell, and Wong respectively.

8

therefore had to develop a system to manually correct the tracking when and where it failed.This “failed” automatic vision-based tracking system should be perceived as a practical and veryusable tool for doing an otherwise painstaking artistic task.

The middle still is from a piece entitled “The Beast” by Gdalevich, Moore, and Shleyfman. Herethe students wrote an Adobe After Effects plug-in to take a green-screened silhouette of aperson and composite it onto a real scene to show a “cloaked” alien. This was a similar effect tothe one that appeared in the “Predator” (Twentieth Century Fox, 1987) movies.

The right image is from the “Flux Capacitor Pro” by Hedlund, Jones, and Laws. Here thestudents developed an Adobe Premiere plug-in to track a car and replace it with lightening. Thiswas an effect motivated by the “Back to the Future” (Universal Pictures, 1985) movies.

In addition to the above-mentioned three projects, there are four more projects from the spring1999 class which are available from the web pages for the class athttp://www.cc.gatech.edu/classes/cs4803d_99_spring/projects.html.

The spring 2000 class is at present working on their final projects that are scheduled for finalscreening on April 28, 2000. We will be happy to show these at the workshop. The web page forthis class is http://www.cc.gatech.edu/classes/AY2000/cs4480_spring/.

6 Future

This is a very exciting class, very popular with students and also quite appreciated by theadministration. However, we feel that there are still some ways we can improve. During thespring 2000 class, we found that the students are really interested in calibrating the camerasand the scene and tracking 3D objects. We intend to increase the amount of time spent on thisaspect of the course next year. In fact, we are planning to add a 2nd assignment that will be oncamera calibration. The students will use the existing code to do this and will develop additionalcode as needed.

The students of the spring 1999 class provided us with much feedback that helped improve ourspring 2000 offering. We hope to again get some positive feedback from this year’s offering toimprove on the course for spring 2001.

Figure 4: Three example projects from spring 1999 class. (left) Fighting with lightsabers,(center) a cloaked alien, and (right), a car vanishing into time.

9

7 Summary

We have designed a new and innovative class to teach students processing, analysis,synthesis, and manipulation of digital video. As a part of this class students develop a shortvideo segment that showcases a special effect that they have generated. During this class theylearn some basic principles of computer vision and graphics, and learn how to undertake asystem building exercise in teams to produce an entertaining video piece. The students lovethis experience and seem to be in demand by the special effects production houses. We feelthat we are succeeding in providing our students with a valuable learning experience.

8 References

Abowd 1999, “Classroom 2000: An Experiment with the Instrumentation of a Living EducationalEnvironment IBM Systems Journal, Special issue on Pervasive Computing, Volume 38,Number 4, pp. 508-530, October 1999. (http://www.cc.gatech.edu/fce/eclass)

Beier & Neely 1992, “Feature-based image metamorphosis” Proceedings of the 19th annualconference on Computer Graphics, 1992, Pages 35 - 42

Burt & Adelson 1983, "The Laplacian Pyramid as a Compact Image Code", IEEE Transactionson Communications (31), No. 4, April 1983, pp. 532-540.

Burt & Adelson 1983, "A Multiresolution Spline with Application to Image Mosaics", ACMTransactions on Graphics(2), No. 4, pp 217-236, October 1983.

Chen & Williams 1993, “View interpolation for image synthesis” Proceedings of the 20th annualconference on Computer graphics , 1993, Pages 279 - 288

CoWeb URL, “Collaborative Software Laboratory”, http://coweb.cc.gatech.edu/csl/9

Horry, Anjyo, & Arai.1997 "Tour into the picture: Using a spidery mesh interface to makeanimation from a single image." In Turner Whitted, editor, SIGGRAPH 97 ConferenceProceedings, Annual Conference Series, pages 225--232. ACM SIGGRAPH, AddisonWesley, August 1997.

Seitz & Dyer 1996 “View morphing”, Proceedings of the 23rd annual conference on Computergraphics , 1996, Pages 21 - 30