1

Digital Sound for the Cinema

Introduction to sound synthesisSandra Pauletto

sandra.pauletto@york.ac.uk

Sandra’s background

• Lecturer in Sound Design at the University ofYork, UK

• Department of Theatre, Film and TV• Background in Physics, Music and Music

Technology• Research in sound design, auditory displays,

sonic interaction design• Born in Portogruaro, resident in the UK since

1996

2

Sound in cinema

• Production Sound– Location sound - dialogue, ambience,

atmospheres, sound effects, wild tracks– Multitrack digital recorder, files on hard

drive, automatic synchronisation viatimecode

• Post-Production Sound– ADR, Foley, digital sound design, editing

and mixing (digital processing)

Digital technologies in cinema

• Digital technologies have revolutionisedcinema sound

• Sound recordings processing and soundsynthesis can all be done digitally

• Sound software most used in the industry:– Avid ProTools +– Mixing console

3

Sound processing and sound synthesis infilm

• Realism: e.g. reverb, filtering (telephone, radio)• Continuity: e.g. EQ changing depending on camera angle• Emphasis: e.g. compression to give impression of

loudness (e.g. explosion)• Space: e.g. panning and reverb• Abstract sound effects: e.g. non-realistic effects in sci-fi

films• Synthesised sounds: e.g. alien sounds, film music• Soundtrack and recording restoration: e.g. denoise,

clicks and pops elimination

How can a computer create sounds?

Air pressure wavesAir pressure waves

Analogue electrical waves

4

Directly programming waves in the computerdigital sound synthesis

Synthesised sound can be very different from anatural or real sound

• Early 20th century:Luigi Russolo - Intonarumori

• In the mid 60s Robert Moog (http://www.moogmusic.com/)and Don Buchla (http://www.buchla.com) created firstanalogue synthesisers

• These use continuous electrical signals to create sound• 1970 MINIMOOG portable synthesiser used by

Genesis, Pink Floyd, etc. Produced until the 80s• Moog synth used in the soundtrack ofClockwork Orange by StanleyKubrick (composer Wendy Carlos)

A brief history: analogue synthesis and digitalsynthesis

5

• It can simulate analogue synthesis and also domore

• 1957 Max Mathews creates first digital synthesis:Newmann Guttman, Pitch Variations, 1957 in theBell Labs (New Jersey)

• Max Matthews created the first language forsound synthesis Music I

• Music II, III, IV, V, Common Lisp Music, CSound,etc. derived from that

Digital synthesis

6

• Only when the processors were powerful enough digitalsynthesis became a viable option

• mid 70s, the Synclavier was one of the first commercialdigital synthesiser. Used by Gary Rydstrom (Star Wars,Toy Story), Howard Shore (Lord of the Rings), AlanSilvestri (Forrest Gump, Who Framed Roger Rabbit), usedin Akira the animation film, and many more

• 1979 Fairlight CMI• 1981 Casio VL-Tone• 1983 Yamaha DX-7• From here on synthesisers are usually digital

Digital synthesis

Hardware synthsare basically computers

Software synthssoftware programs which run in any computer

Modular synths(can be hardware or

software) can connect variousmodules to create different soundsynthesis techniques

Synthesisers

7

FreePure Data, JMaxCsound, Supercollider

Not freeReaktor, MAX/MspDoepfer, Nord Modular

ModularSynthesisers

Digital samples:..., x[n － 1], x[n], x[n + 1], ...

n = number (integer) of the samplex[n] is the sample of number n

Fundamentals of digital synthesis

x(t)

t

x

8

Fundamentals of digital synthesis

Sinusoid: x[n] = A * cos(ωn + φ)A = amplituden = number of the sampleω = angular frequency = 2*pi*fφ = initial phase = angle at x[0] i.e. at n = 0

Example: 440Hz cosine waveAngular frequency ω is 2*pi*f = 2764.6 rads/secInitial Phase φ = zeroAmplitude = 1 ([-1,1] amplitude range)

The cosine wave

9

Cosine

Now we can calculate each sample of the440Hz wave:

X[1] = 1*cos(2764.6*1+0)X[2] = 1*cos(2764.6*2+0)X[3] = 1*cos(2764.6*3+0)X[4] = 1*cos(2764.6*4+0)…X[n] = 1*cos(2764.6*n+0)

Sampling rateR = sampling rate = number of samples in 1

secondUsually in cinema 48000 samples per second

Sample number and timeR*t = n, number of samples in the time tSample n occurs at time:

t = n/R seconds

10

Sound parametersPerceptualparameters

Acousticalparameters

Scale Units

Pitch

Volume

Duration

Timbre

Frequency

Wave amplitude

Time

Frequencycomponents

Log

Log

Linear

N/A

Hertzor Hzdecibelsor dBs

ms

N/A

A very simple sounda sine wave consists of one frequency

Temporal description Spectral description

A sinewave in Pure Data

11

Fourier proved:any continuous periodic wave = an infinite sum of sinewaves

Instruments’sounds:

fundamentalfrequency andmany harmonics

Harmonics are atintegersmultiples of thefundamentalfrequency

Complex sound

Harmonics have various amplitudes and they determinethe timbre of a sound

12

sine, triangular, square, sawtooth, pulse, white noise

Sine wave

Basic unit generators

Square wave

• The square wavecontains only oddharmonics with theamplitudes

• A = amplitude• n = no. of harmonic• Brass sounds

13

Sawtooth wave

• The sawtooth wavecontains all harmonicsin the ratio

• A = amplitude• N = no. of harmonic• String sounds

Triangular wave

• The triangular wavecontains only oddharmonics with theamplitudes

• A = amplitude• n = no. of harmonic

14

Pulse wave

• The pulse wavecontains all theharmonics

• Subtractive synthesis• Speech sounds

White noise

• distributedspectrum, i.e.energy existseverywherewithin a range offrequencies

• no repetitions!

• randomnumbers

15

Let’s hear them in Reaktor!

Additive Synthesis and organs

• The concept of additive synthesis is many centuriesold

• It means ADDING tones of different frequenciestogether to create new sounds

• First applied in pipe organs by means of register-stops• Pulling a register-stop would mean to send air to a set

of pipes• The air was then released into the pipe by pressing a

key in the keyboard and would produce sound• This would result in creating sounds by MIXING

frequencies

16

Additive Synthesis and organs

• First ever electrical additive synthesizer (1906,Thaddeus Cahill)• Electric motors produced the A.C. ‘tones’• Signals connected to telephone receivers with papercones attached to ‘amplify’ the signals• Use of ‘stops’ to add in motors at different frequencies

The Telharmonium

http://www.discretesynthesizers.com/archives/miessner/em1936.htm

17

• The DB33 Instrument in ProTools is an example ofadditive synthesiser• Another example is the EVB3 Instrument in Logic

The Hammond Organ

• It emulates in the digital world the Hammond B3organ which was created by Laurens Hammond inthe 1930s• The original organ was electro-mechanical• Electric motors drove a set of metal wheels (called“tone wheels”) which, as they were spinning nearelectromagnets, generated electromagnetic waves(through the process of electromagnetic induction) atcontrollable frequencies• These electromagnetic wave signals were then sentto loudspeakers and perceived as sounds

The Hammond Organ

18

The Hammond Organ

• The Hammond Organ also had features such as vibratoand reverb, and various other effects

• Digital additive synthesiser• 64 harmonics per note for 8 voices• Detuning and LFO (Low Frequency Oscillator)

Kawai K5 (late 80s)

19

Basic additive synthesis

Basic Additive Synthesis in PD

Sound effects created in additivesynthesis

Old telephone bell ringing• 2 bells: additive synthesis• 1 hammer: very short noise• 1 telephone casing: reflections inside the

casing modelled with two delays

Example from Andy Farnell

Example of detuned additive sound in PD

Additive bell sound (Jean-Claude Risset)

20

The first sound that we hear in Apocalypse Now, over a black screen, is of ahelicopter circling us in the 5.1 channel space. But itʼs not an actual helicopterthat weʼre hearing. What weʼre hearing is an electronically generatedsimulation of helicopter blades. The sound was designed by Richard Beggs, aman who did a lot of wonderful sound effects fabrication for Apocalypse Now,and he made it with a fairly early version of a Moog synthesiser. Sometimeswhen Iʼm talking to classes about this sequence, I ask: ʻWhy the synthesisedhelicopter?ʼ Lord knows we had plenty of recordings of actual helicopters. Iknow because I recorded some of them myself. Why did Francis Coppola andWalter Murch, the guru of us all, who was in charge of the sound onApocalypse Now, decide to use this synthesised helicopter? The answer, ofcourse, is that is how Willard hears it in his fevered dream; he hears thisstylised dreamy strange ghostly helicopter circling him. And that messagegets through even to the least sophisticated member of the audience. Theyrealise on some level that this strange helicopter is in Willardʼs mind. (Thom,p. 105)

Randy Thom, Screenwriting for sound, The New Soundtrack 1.2 (2011):103–112

Apocalypse Now, 1979

Dir: F.F.CoppolaSound Designer: Walter Murch

21

Additive synthesis

In real sounds:• The amplitude and

frequency of each partialcan vary in time

In synthesisers:• Functions called

Envelopes and LowFrequency Oscillators(LFOs) can be used tovary parameters in time

22

Low Frequency Oscillators or LFOs

• Low Frequency Oscillators or LFOs:– oscillators of frequency between 0-20Hz– in this frequency range an oscillator can control or

modulate various aspects of a sound• Human hearing can perceive frequencies between

20Hz to 20kHz so at frequencies higher than 20Hz anoscillator starts to become audible and it is not anLFO anymore

• LFOs can have various waveforms: sine, square,sawtooth, triangular

flute

Vibrato using an LFO

Vibrato in PD

• Is slow modulation ofthe pitch of a sound

• Low FrequencyOscillator can be usedto simulate this

depth

rate

frequency

output

amplitude

LFO

23

Vibrato spectrogram

Frequency ofharmonicsmodulated by asine wave

From synth secrets of Wendy Carlos:

One way to make an interesting sound out of afew sine waves is:

1. have them all slightly detuned around a pitch(this is a chorused sound)

2. then add vibrato which creates dynamism intime

Choral tone

24

Modulation techniquesThe term modulation refers loosely to any technique that

systematically alters the shape of a waveformExamples are:

- Amplitude modulation and ring modulation- Frequency modulation

• With modulation techniques one can create sounds thatevolve in time in a complex way

• This evolution can be clearly heard and it can also beseen by looking at the spectrum of the resulting sound

• Complex sounds are created efficiently

Amplitude modulation• Modulation of the amplitude of

one oscillator (carrier) by anotheroscillator (modulator)

• The modulation happens ataudio frequencies

i.e. between 20Hz and 20kHz

• The output sound presents 2sidebands at the followingfrequencies:

(fc – fm) , fc , (fc + fm)

CARRIER

MODULATOR

In PD see example 0

output

Fc Am

Fm

Ac

25

Ring modulation

• Ring modulation is a simple case ofamplitude modulation

• In this case the amplitude of the carrier is 0• This is equivalent to simply multiplying the

two signals together• In the spectrum of the output sound new

frequencies will emerge different from theinitial ones

Ring modulation

Ring modulationremoves the carrierfrom the spectrum

MODULATORCARRIER

<-- Same as multiplying theoutputs of a carrieroscillator with amplitude 1and a modulator oscillatorwith amplitude Am

In PD see example 1

output

FcAm

Fm

26

Alien, 1979Dir: Ridley Scott

27

• Pioneered by John Chowning in the late60s

• FM can be thought as:the alteration or distortion of the frequencyof an oscillator in accordance with theamplitude of a modulating signal (Dodge & Jerse,Computer Music, 1985, p.115)

FM synthesis

http://www.youtube.com/watch?v=IsQFuesfaoI

Wall-eDir: Andrew Stanton, 2008

28

Simplest case• A sine wave (the

modulator)modulates thefrequency of asecond sine wave(the carrier)

• In FM, the frequencyof the modulator Fm

is in the audiblerange (20Hz to20kHz)

MODULATOR

CARRIER

Bouncing ball example from Farrell

Am

Fm

Fc

output

Ac

Output SpectrumfSB = fc ± n*fm

where

fSB = sidebandsfrequencies

fc = carrierfrequency

fm = modulatorfrequency

n = 1,2,3,4,…

FM creates sidebands

In PD see examples 2 and 3

29

Forbidden Planet, 1956Dir: Fred Wilcox

30

FM soundtrack for La Linea (Cavandoli)Student exercise

31

• Dodge, C. & Jerse, T. A. (1997) Computer Music:synthesis composition and performance, Publisher:Schirmer

• Roads, C. (1996) Computer Music Tutorial, MIT Press• Farnell, A. (2009) Designing Sound, MIT Press• Lynn, P. A., and Fuerst, W. (1998) Introductory Digital

Signal Processing with Computer Applications, JohnWiley and Sons Ltd.

• Puckette, M. The Theory and Technique of ElectronicMusic, 2007, available athttp://crca.ucsd.edu/~msp/techniques.htm

Resources

• Pure Data documentation:http://crca.ucsd.edu/~msp/Pd_documentation/

• Chafe, C. (1999) A short history of digital sound synthesisby composers in the USA,ccrma.stanford.edu/~cc/lyon/historyFinal.pdf

• Smith, J. (1991) Viewpoints of the history of digitalsynthesis, CCRMA

http://www-ccrma.stanford.edu/~jos/kna/

Resources