Tension in Music :Cognition, Emotion, Brain, Movement

Carol L. KrumhanslDepartment of Psychology

Cornell UniversityIthaca, NY 14853

USA

Tension: Linking Cognition, Emotion, Brain, and Motion

WIPPPPPWIPPPPP

Work in Past Published in Press Present PlannedWork in Past Published in Press Present Planned

"…expectation is always ahead of the music, creating a background of diffuse tension against which particular delaysarticulate the affective curve and create meaning.”

“Not only does music use no linguistic signs but, on one level at least, it operates as a closed system, that is, it employs no signs or symbols referring to the non-musical world of objects, concepts, and human desires. …This puzzling combination of abstractness with concrete emotional and aesthetic experience can, if understood correctly, perhaps yield useful insights into more general problems of meaning and communication.”

L. B. Meyer, Emotion and Meaning in Music, 1956

Duration of each beat in the music as performed

Mozart, Piano Sonata, K. 282Krumhansl, Music Perception, 1996

Tension and cognition

Judgments of sections ends

Duration of each beat in the music as performed

Duration of each beat in the music as performed

Judgments of new musical ideas

Krumhansl, Music Perception, 1996

(a) octave (root) level: 0 (0)(b) fifth level: 0 7 (0)(c) triadic level: 0 4 7 (0)(d) diatonic level: 0 2 4 5 7 9 11 (0)(e) chromatic level: 0 1 2 3 4 5 6 7 8 9 10 11 (0)

Lerdahl, Tonal Pitch Space, 2001

Diatonic basic space, set to I/C (C = 0, C# = 1, …B = 11).

Computing Distance from d minor (vi) chord in F major keyto C major (I) chord in C major key

Region DistanceF major to C major

Chord Distanced minor to C major

Basic Space DifferencesNew entries in Basic Space

Tensing Relaxing

Slide 40Slide 40

Krumhansl, Music Perception, 1996

Chopin Prelude

0

25

50

75

100

125

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

Event

Tension

Judged Predicted

Messiaen Quartet

0

25

50

75

100

125

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Event

Tension

Judged Predicted

Messiaen, Quartet for the end of time

Bach Chorale

0

25

50

75

100

125

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41

Event

Tension

Judged Predicted

Bach ChoraleCristus, der ist mein Liebe

WagnerParsifalGrailTheme

Diatonic

Chromatic

Lerdahl & Krumhansl, Music Perception.in press

Wagner Diatonic Hierarchical Right-Branching +

0

25

50

75

100

125

1 2 3 4 5 6 7 8 9

Event

Tension

Judged Predicted

Wagner ChromaticShifting Diatonic

Sequential to Hierarchical

0

25

50

75

100

125

1 2 3 4 5 6 7 8 9

Event

Tension

Judged Predicted

Fourier Balance OneFB1

FB 1

Stable for minor second, major seventhUnstable for fourth, fifth, and tritone

Stable for a diatonic scale

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Tipping the (Fourier) balances:A geometric approach to representing pitch structure in non-tonal music

FB 2

Stable for tritone, minor second, major seventh, fourth and fifthUnstable for minor third, major sixth

Stable for an octatonic scale

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Fourier Balance TwoFB2

FB 1 FB 2 FB 3

FB 4 FB 5 FB 6

-25

0

25

50

75

100

Y

1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0

Event

Predicting Judged Tension with FB model

Fourier Balance ModelR-squared (2,5) = .999

Predicting Judged Tension with TPS ModelR-squared (2,5) = .942

Compare Fourier Balance Model to Tonal Pitch Space Model

Y Judged Tension Predicted from FB model Predicted from TPS model

Predicting Judged Tension with FB ModelR-squared (18,21) = .823

Predicting Judged Tension with TPS ModelR-squared (2,37) = .758

0

10

20

30

40

50

60

70

80

90

100

Y

0 10 20 30 40

Rows

Y Judged Tension Predicted from FB model Predicted from TPS model

MessiaenQuartet for the End of Time

Summary:Fourier Balance model fits judged tension better thanTonal Pitch Space model

but:The two models are closely connected

International Congress on Music Perception and CognitionBologna, August. 2006

A Proto-Music Theoryfrom Unsupervised Learning

Carol L. KrumhanslDavid G. Rand

Background

Experiments demonstrate listeners are sensitive to statistically frequent patterns

“Statistical learning”

adults -- cross-cultural (Krumhansl, et al., 1999, 2000)

adults -- melodies on artificial tone-sets (Oram & Cuddy, 1995)

infants -- tone and syllable sequences (Saffran, et al. 1999)

Limitations

Low-order statistics for sequences

P(c1), P(c2 | c1), P(c3 | c1 c2)

Results show effects of acculturation only in higher-order statistics (Krumhansl et al., 1999, 2000)

Automatic discrimination of musical styles only with higher-order statistics (Krumhansl, 2000)

Coding of music ignores durations of tones

c1 c2 c3 … ck

Questions

Can a statistical learning model distill musically interpretable patterns from a musical corpus?

What does such a model show about the relevance of rhythm to melodic structure?

ADIOS (Automatic DIstillation Of Structure)Zach Solan, David Horn, Eytan Ruppin (Tel-Aviv University),Shimon Edelman (Cornell University)

Two classes of syntax modelsLanguage-specific theory of syntax (generative theory)General-purpose statistical or distributional learning models

ADIOS has features of bothDistillation of rule-like regularities out of the acquired knowledgeKnowledge acquired only from “raw” distributional information

Representational Data Structure (RDS)

Directed Graph Input: raw unlabeled corpus data (not tagged for part of speech, only BEGIN and END of each sentence)

Node = one constituent (word)

Directed edge is inserted if transition between constituents exists in corpus

Pattern Acquisition (PA) Algorithm

A Pattern is a similarly structured sequence of constituents that recurs in the corpus.

An Equivalence Class is a set of constituents from different paths that occur in the same position in a pattern

This is the syntax that ADIOS is distilling

Bundles are formed when two or more paths run in parallel

and dissolved when more paths leave the bundle than stay in

Criterion for judging pattern significance

For path c1 c2 c3 … ck:

S = e-(L/k)2 P(c1,c2,…,ck) log( P(k)(path) / P(2)(path)), where:

L = typical context length k = length of the candidate path

P(k) (path) = P(c1)P(c2|c1)P(c3|c1 c2)…P(ck|c1 c2 c3 … ck)“k-gram”

P(2) (path) = P(c1) P(c2|c1) P(c3|c2)… P(ck|ck-1)“random walk”

Bootstrapping

The identification of new equivalence classes is done using acquired equivalence classes - “bootstrapping”

Musical Corpus

Musical Themes from Classical Corpus in Themefinder300 themes in C major, 300 themes in A minor Monotone (no harmony)

C major scale tones: C D E F G A B

A minor natural scale tones: A B C D E F G

Kern format:• 2aa 2ee 4dd 8cc 8b 4a 4a 2ee 2b 4a 8g 8f 4e 4e #

2=half note, 4=quarter note, 8=eighth note, etcG = G below middle C, g = G above middle C, gg= G an octave and a fifth above middle C, etc.* = begin# = end

Each duration-pitch pair is treated as an independent constituent

There is NO information given to ADIOS to indicate that:

16a and 8a are the same pitch (with different durations)16a and 16b are the same duration (with different pitches)16G and 16g are the same pitch in different octaves (octave equivalent)

Comparison with Musicians Judgments43 Major, Minor, Major Shuffled, Minor Shuffled, Major Scrambled, Minor Scrambled11.1 years instruction on musical instruments, 3 or more music theory courses

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Overall Summary

ADIOS distills interpretable structure from musical corpusMusically interpretable Patterns and Equivalence ClassesTonal hierarchy is evident in Patterns and Equivalence ClassesWhen trained on Major, can correctly discriminate Major vs MinorWhen trained on Major and Minor, can discriminate Original vs Shuffled (duration-pitch pairs moved)

ADIOS develops “Proto-Music Theory” Not fully developed, but sufficient to determine:Neighbors (pitch proximity)Same pitch independent of durationSame duration independent of pitchOctave equivalent pitchesHarmonically related pitchesMajor vs. MinorOrder of duration-pitch pairs

Tension and Emotion

Krumhansl, An exploratory study of musical emotions and psychophysiologyCanadian Journal of Psychology, 1997

Experimental Design

Dynamic RatingsSadFearHappyTension

Sad Excerpts:Tomaso Albinoni, Adagio in G minor for Strings and OrchestraSamuel Barber, Adagio for Strings, Op. 11

Fear Excerpts:Gustav Holst: Mars -- the Bringer of War from The PlanetsModest Mussorgsky, Night on Bare Mountain

Happy Excerpts:Antonio Vivaldi, The Four Seasons, La Primavera (Spring), Danza pastoraleHugo Alfven, Midsommarvaka

Physiological Measures Recorded at 1-second Intervals

1) cardiac interbeat interval (IBI), measured in milliseconds, with shorter IBIs taken to indicate a higher level of cardiovascular arousal2) pulse transmission time to the finger (FPTT), measured in milliseconds, with shorter pulse transmission times indicative of greater autonomic (sympathetic) activation3) finger pulse amplitude (FPA), a measure of the amount of blood in the periphery, with reduced amplitude indicating greater vasoconstriction and associated with greater autonomic (sympathetic) activation4) pulse transmission time to the ear (EPTT), another measure of blood flow5) respiration intercycle interval (ICI), measuring the time between successive inspirations in milliseconds6) respiration depth (RD), which is the point of maximum inspiration minus the point of maximum expiration7) respiration-sinus asynchrony (RSA)8) systolic blood pressure (SBP)9) diastolic blood pressure (DBP)10) mean arterial pressure (MAP)11) skin conductance level (SCL), with increased skin conductance indicative of greater autonomic (sympathetic) activation12) temperature on the finger (TEM) measured in degrees Fahrenheit.

Correlations between Dynamic Emotion Ratings and Dynamic Physiology Ratings

Sad Ratings Fear Ratings Happy RatingsInterbeat-Interval

.14*** -.01 -.15***Finger Pulse Transmission Time

.10** .16*** -.09**Finger Pulse Amplitude

-.14*** -.31*** .24***Ear Pulse Transmission Time

.07* .15*** .03Respiration Intercycle-Interval

.05 -.12*** -.16***Respiration Depth

.00 .00 -.09*Respiration Sinus Asynchrony

-.02 -.11*** -.13***Systolic Blood Pressure

.37*** -.23*** -.14***Diastolic Blood Pressure

.41*** -.24*** -.14***Mean Arterial Pressure

.37*** -.25*** -.10**Skin Conductance Level

-.36*** .06 -.08*Finger Temperature

-.35*** -.20*** .21***

Factor analysis of correlations between physiological measures

Distinct groupings

Blood SCL Respiration Blood Flow Heart Blood FlowPressure Temp Rate Finger Rate Ear

Functional Magnetic Resonance Imaging (fMRI)

The neural mechanisms of tonal and rhythmic expectations were studied in two ways:

Stimulus- introducing either tonal or rhythmic violations (or both)

Task-judge either the tonal structure or the rhythmic structure -(or passively listen to the melodies)

Tension and Brain

Musical Sequences: Melodies composed by Diego Vega, Cornell University (6 sec), piano timbre. Sample sequences:

Original

Tonal Violations

Rhythmic Violations

Tonal and RhythmicViolations

Passive Listening: Tonal and Rhythmic ViolationsThis analysis contrastedmusical sequences containing both Tonal and Rhythmic violations with musical sequences with no violations.

Even when the subjects were in not performing a task, activations in:

superior temporal cortex (especially on the right)

Active Judgments: Tonal and Rhythmic Violations

This analysis also contrastedMusical Sequences containing both Tonal and Rhythmic violations with musical sequences with no violations (as before)

Judged whether a violation occurred

Superior temporal activation (as for Passive Listening), in addition, when making either Tonality or Rhythm Judgments:

right dorsolateral frontal right

inferior frontal (bilateral)

"The cognitive representation of an event unit involving human motion can bedescribed as some small set of relatively stable, preparatory motions, followed bythis relatively unstable, completing motion. Such a schematic structure enablesthe perceiver to anticipate temporal relations within an unfolding event, and to fitsuccessive parts of the temporal sequence into this anticipated structure.”

M. Lasher, Cognitive Psychology, 1981.

"The interaction between movement and sound is the most fundamental element ofdance. The dance does not mimic the music -- there is not a particular part of themusic for every gesture and step -- but the basic "kinetic feel" or "energy shape"of the music is expressed in the dance. The choreographer uses the music not onlyfor its rhythmic pulse, but also as a source of emotional and structural ideas. Thus,elements of the music are often observed in the dance.”

K. Teck, Movement to Music, 1990.

Tension and motion

Tension and motion in dance

Experimental Design

Music W. A. Mozart, Divertimento No. 15 Bb, Minuetto

Dance George Balanchine, School of AmericanBallet

Subjects Music Lessons 9.2 yearsDance Lessons 7.3 years

Conditions Music OnlyDance OnlyBoth Music and Dance

Tasks Section End (discrete judgment)during Tension (continuous judgment )videotape New Idea (discrete judgment)

Emotion Expressed (continuous judgment)

Task after Emotion Quality (overall judgment)videotape

Intrinsic Relationships Between Music and Dance

Rhythmic accent, meter, sounds produced by the dancers

Dynamic volume of musical and choreographic

Textural number of instruments/performers, homophony versus polyphony, counterpoint

Structuralcorresponding motives or figures, phrases, structures

Qualitative choreomusical parallels of tessitura, timbre, articulation, dissonance/consonance

Mimetic choreography imitates a particular sound in the music

P. Hodges (1992) Relationships between score and choreography in 20th century dance. London: Mellen.

"To see Balanchine's choreography … is to hear the music with ones eyes…The choreography emphasizes relationships of which I had hardly been aware…and the performance was like a tour of a building for which I had drawn the plansbut never explored the result.”

I. Stravinsky, quoted in S. Jordan, Dance Chronicle, 1993

Exploratory Study of Anticipating Human Movement in Dance

Barycenter

Center of sub-regionincluding head, trunk,legs

Stop Position

Target PositionJudged Position

Camurri, Krumhansl, Mazzarino, Volpe, 2004

Marcello Wanderley, IRCAM thesis, 2002Non-obvious performance gestures

(Not needed to produce tone)

Normal Expression, Exaggerated Expression,Immobile

Tension and motion in music performance

Experimental Design

Continuous Judgments of TensionContinuous Judgments of Phrasing

while:watching performance (no sound)hearing performance (no image)both watching and hearing performance

Stravinsky, Second of Five Pieces for Solo Clarinet

Tension judgments in experiment

Phrasing judgments in experiment

Analysis of data with

Functional Data AnalysisJ. Ramsay

Audio, Visual, Audio and Visual conditionssimilar for judgments of phrasing

But complex interactions between Visual and Audioin judgments of tension

Vines, Wanderley, Levitin, Krumhansl, Cognition, 2006

Tension in facial and body gesture

Cognition Emotion Motion

Brain

Behavior

Music

Tension

L. B. MeyerEmotion and Meaning in Music

1956

"…expectation is always ahead of the music, creating a background of diffuse tension against which particular delaysarticulate the affective curve and create meaning.”

“Not only does music use no linguistic signs but, on one level at least, it operates as a closed system, that is, it employs no signs or symbols referring to the non-musical world of objects, concepts, and human desires. …This puzzling combination of abstractness with concrete emotional and aesthetic experience can, if understood correctly, perhaps yield useful insights into more general problems of meaning and communication.”

L. B. Meyer, Emotion and Meaning in Music, 1956

Cognition Emotion Motion

Brain