visual texture, music, and emotion - palmerlab.berkeley.edu · not angry−angry active−passive...

Visual Texture, Music, and Emotion:Cross-modal associations between visual textures and musicThomas Langlois, Joshua Peterson & Stephen E. Palmer, Department of Psychology, University of California, Berkeley

METHODS

STIMULI

BACKGROUND RESULTS

EXPERIMENT 1EXPERIMENT 1 CONCLUSION

RESULTSEXPERIMENT 2EXPERIMENT 2

REFERENCES Emotion Dimensions

Not Angry−Angry

Active−Passive

Smooth JazzPiano

FolkMozart

CountryBach

Big BandIndie

SoundtrackBalkin Folk

ReggaeHip Hop

Progressive Rock80s Pop

ElectronicArabic

Hindustani SitarBlues

DixielandJazz

AlternativeFunk

GamelanSalsa

BluegrassIrish

TranceClassic Rock

Progressive HousePsychobillyStravinsky

SkaDubstep

Heavy Metal

PianoBach

CountrySmooth Jazz

MozartBalkin Folk

FolkIndie

ReggaeBig Band

Hip HopSoundtrack

80s Pop Alternative

BluesGamelan

Progressive RockArabic

IrishJazz

DixielandHindustani Sitar

FunkSalsa

BluegrassElectronic

TranceProgressive House

PsychobillyStravinsky

Classic RockSka

DubstepHeavy Metal

Harmonious−Disharmonious Orderly or Simple−Chaotic or Complex

Sharp−Smooth

Happy−Sad

Texture-Music Association (TMAD ): The weighted average of the three textures picked as being most consistent with the music (Cd,m)

minus the weighted average of the three textures picked as being most inconsistent with the music ( Id,m) along a given dimension D:

Cd,m = (3c1,d,m+ 2c2,d,m+ c3,d,m)/6, Id,m = (3i1,d,m+ 2i2,d,m+ 13,d,m)/6, TMAd,m = Cd,m- Id,m

1st 3rd Choices2nd

2AFC Experiment

CountryBalkin FolkBachGamelan

FolkBig BandSmooth JazzReggae

DubstepMozartPsychobilly80s Pop

IrishPianoElectronicSka

Progressive RockSalsaHeavy MetalSoundtrack

Calm−Agitated

PianoBachFolk

MozartCountry

Smooth JazzBalkin Folk

IndieBig Band

SoundtrackReggaeHip Hop

BluesHindustani Sitar

JazzDixielandGamelan

Progressive Rock Alternative

80s PopArabic

IrishFunk

ElectronicBluegrass

SalsaPsychobilly

Classic RockProgressive House

StravinskyTrance

SkaDubstep

Heavy Metal

Heavy MetalDubstep

SkaStravinsky

Classic RockTrance

PsychobillyProgressive House

IrishFunk

BluegrassSalsa

GamelanElectronicDixieland

ArabicJazz

Hindustani SitarBlues

Progressive Rock80s Pop

AlternativesHip HopReggae

SoundtrackIndie

Big BandBalkin Folk

FolkCountry

Smooth JazzMozart

BachPiano

Ska Heavy Metal Classic Rock

Trance Progressive

House Psychobilly

Stravinsky Gamelan

Soundtrack Funk

Hip Hop 80s Pop Reggae

Big Band Alternative

Balkin Folk Country

Piano Indie

Mozart Hindustani Sitar

Smooth Jazz

Electronic Jazz

Blues Irish

Bach

Folk Salsa

Progressive Rock

Arabic Bluegrass

Direct Music Ratings

Texture Music Associations (TMA)

MozartPiano

FolkBach

CountrySmooth Jazz

Balkin FolkIndie

Big BandReggaeHip Hop

Soundtrack Alternative

80s PopGamelan

BluesIrish

Progressive RockJazz

DixielandBluegrass

ArabicHindustani Sitar

ElectronicFunk

SalsaClassic Rock

PsychobillyProgressive House

StravinskyTrance

SkaDubstep

Heavy Metal

Heavy MetalDubstep

SkaProgressive House

StravinskyTrance

PsychobillyClassic Rock Alternative

IrishBluegrassGamelan

BluesJazz

SalsaFunk

Hindustani SitarBig Band

80s PopProgressive Rock

DixielandHip HopReggae

ElectronicBachIndie

MozartArabic

FolkSoundtrack

CountryPiano

Balkin FolkSmooth Jazz

80s PopReggae

FolkSoundtrack

DixielandArabic

Smooth JazzCountry

MozartProgressive Rock

ElectronicFunk

Balkin FolkPianoBlues

Hip HopBig Band

Bluegrass Alternative

Progressive HouseTrance

GamelanHindustani Sitar

BachIrish

Classic RockIndie

SalsaPsychobilly

JazzHeavy Metal

StravinskySka

Dubstep

High correlations indicate support for the emotion mediation hypothesis (Palmer, S. E., Schloss, K., et al (2013)

MDS of Emotional Space for thirty-four genres of music based on pairwise correlations of average emotional ratings

Dimension 1

Dim

ensi

on 2

Happy

Sad

Agitated

Calm

-1.5 -1 -0.5 0 0.5 1 1.5-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Salsa

Progressive Rock

ArabicBluegrass

Alternative

Blues

Gamelan

Bach

Balkan Folk

Country Classic Rock

Dixieland

TranceProgressive House

Ska

Electronic

Piano

IrishSoundtrack

Indie

Funk

Hip Hop

80s Pop

Psychobilly

Mozart

DubstepHeavy Metal

Hindustani Sitar

Stravinsky

Jazz

Reggae

Smooth Jazz

Big Band

Folk

Correlations between the emotional content of the musicand the corresponding emotional content of the colors chosen to go with it

p < .05

0.2

0.4

0.6

0.8

1.0

Corr

elat

ion

(r)

Calm/Agitated

Happy/Sad

Whim./Serious

Appeal./Disgust.

Pref.

Thirty-four musical genres clustered by similar texture pro�les*:

Associations among music, texture, emotion, and non-emotional audio-visual features:

SkaHeavy Metal

GamelanPsychobilly

Progressive HouseDixieland

SalsaDubstep

TranceBig Band

FunkReggae80s Pop

SoundtrackClassic Rock

BluegrassStravinskyElectronic

Smooth JazzHip Hop

ArabicJazz

Progressive RockIrish

Balkin FolkBlues

CountryHindustani Sitar

PianoFolk

BachIndie

AlternativeMozart

Granular−Fibrous

ArabicSalsa

DixielandElectronic

FunkProgressive Rock

Hindustani SitarSmooth Jazz

Soundtrack80s PopHip Hop

PianoReggae

BluegrassBalkin Folk

Big BandTrance

IndieJazzSka

BluesGamelan

StravinskyCountry

Classic RockMozart

FolkPsychobilly

Heavy MetalBach

DubstepIrish

Progressive House Alternative

Curved−Straight

Non-synesthetes make systematic associations between 34 musical genres and color.The associations are mediated by emotion (Whiteford, Schloss, Palmer, VSS-13).

Twenty-eight di�erent visual textures synthesized from spline curves:

Do cross-modal associations exist between music and visual texture?

Hindustani SitarIndieDixielandBluegrass

Thirty-four excerpts of songs, each representing a unique musical genre: StravinskyFunkTranceAlternative

1

Sound-Texture Choices

1) Pick 3 textures that are most consistent with the music.

Music-Feature Ratings

HappySad

HappySad

Texture-Feature Ratings

2) Pick 3 textures that are least consistent with the music.

1 2

3

Rate features of each stimulus:

Simple-Complex

Granular-Fibrous

Slanted-Not Slanted

Curved-Straight

Separate-Connected

Sharp-Smooth

Natural-Arti�cial

Happy-Sad

Calm-Agitated

Weak-Strong

Active-Passive

Not Angry-Angry

Harmonious-Disharmonious

r = .92**** MUSIC RATING

TMA r = .95**** MUSIC RATING

TMA


TMA


TMA r = .91**** MUSIC RATING

TMAr = .84**** MUSIC RATING

TMA

r = .55***MUSIC RATING

TMA r = .44**MUSIC RATING

TMA

JazzHip HopHouseBlues

Classic RockArabic

−150 −100 −50 0 50 100 150Dixieland

CountrySmooth Jazz

Hip HopPiano

Balkin FolkBach

SoundtrackFolk

Hindustani SitarGamelan

FunkMozart

JazzBig Band

IndieReggae80s Pop

Progressive RockSalsa

ElectronicBlues

SkaIrish

ArabicStravinsky

PsychobillyBluegrass

AlternativeHeavy Metal

Classic RockProgressive House

TranceDubstep

Strong Weak


TMA

00.

51

00.

51

Visual Dimensions An

gry-

Not A

ngry

Calm

-Agi

tate

d

Wea

k-St

rong

Natu

ral-A

rti�c

ial

Natu

ral-A

rti�c

ial

Shar

p-Sm

ooth

Shar

p-Sm

ooth

Curv

ed-S

traig

htCu

rved

-Stra

ight

Gran

ular

-Fib

rous

Gran

ular

-Fib

rous

Sim

ple-

Com

plex

Sim

ple-

Com

plex

Harm

onio

us-D

ishar

mon

ious

Harm

onio

us-D

ishar

mon

ious

Activ

e-Pa

ssiv

eAc

tive-

Pass

ive

Right BetterLeft Better

ThickThin

Dark Background

Medium Brightest

Darkest Medium

BluegrassBlues

GamelanBach

Dixieland

HouseFunk

Piano

Heavy Metal

Folk

Varia

nce

Expl

aine

d

20

40

60

80

100

0

Thickness

Contrast

Lightness

As with music-color cross-modal associations, non-synesthetes appear tomake systematic cross-modal associations from music to texture.

Music-texture cross-modal associations were mediated by emotion, but unlike music-color associations, happy-sad was not relevant to texture. Music-texture associations were also mediated by other visual features.

Variance explained by perceived lightness,contrast, and thickness of visual textures

Shared features:Angry-Not Angry

Harmonious-DisharmoniousActive-PassiveWeak-Strong

Texture-Music features:Sharp-Smooth

Curved-StraightGranular-Fibrous

Color-Music features:Happy-Sad

Listen to Music

Evaluate Emotional content and other

features of the music Texture-Music Association Task:People chose textures with match-ing emotional content, and match-ing non-emotional audio-visual content

Color-Music Association Task:People chose colors with matching emotional content

Happy-Sad does not mediate texture-music associations

r = -.03MUSIC RATING

TMA

Visual Dimensions Emotional Dimensions

*Contrast of texture corresponds to frequency of choice across all subjects






Texture Music Associations (TMA)Direct Music Ratings





Texture Music Associations (TMA)Direct Music Ratings





(TMA)

First color picks for �rst twenty participants

Bright Background

Acknowledgements: We thank Will Griscom, Jan Flatley-Feldman, Vivian Wung, Candita Wager, Melissa Gertler, Sheila Rajagopalan for their help with this project. This material is based on work supported by the National Science Foundation under Grants Nos. 1059088 and 0745820, and by a Google Gift to the senior author, Stephen E. Palmer.

-Whiteford, K., Schloss, K., Palmer, S., (VSS-2013). “Bach to the Blues” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013.

(r)

−150 −100 −50 0 50 100 150−150 −100 −50 0 50 100 150

−150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150

−150 −100 −50 0 50 100 150−150 −100 −50 0 50 100 150−150 −100 −50 0 50 100 150

−150 −100 −50 0 50 100 150

**** p < 0.00001 *** p < 0.0001 ** p < 0.001 * p < 0.01

-Palmer, S. E., Schloss, K., B., Xu, Z. & Prado-Leon, L., (2013) Music-color associations are mediated by emotion, Proceedings of the National Academy of Sciences. 110 (22), 8836-8841.

visual texture, music, and emotion - palmerlab.berkeley.edu · not angry−angry active−passive...

Documents