mixmash - cdv lab

DOI: 10.4018/IJACDT.2019070102

International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019

Copyright©2019,IGIGlobal.CopyingordistributinginprintorelectronicformswithoutwrittenpermissionofIGIGlobalisprohibited.

20

MixMash:An Assistive Tool for Music Mashup Creation from Large Music CollectionsCatarina Maçãs, CISUC - Department of Informatics Engineering University of Coimbra, Coimbra, Portugal

Ana Rodrigues, CISUC - Department of Informatics Engineering University of Coimbra, Coimbra, Portugal

Gilberto Bernardes, INESC TEC & University of Porto, Faculty of Engineering, Porto, Portugal

https://orcid.org/0000-0003-3884-2687

Penousal Machado, CISUC - Department of Informatics Engineering, University of Coimbra, Coimbra, Portugal

ABSTRACT

ThisarticlepresentsMixMash,aninteractivetoolwhichstreamlinestheprocessofmusicmashupcreationbyassistingusersintheprocessoffindingcompatiblemusicfromalargecollectionofaudiotracks.ItextendstheharmonicmixingmethodbyBernardes,DaviesandGuedeswithnoveldegreesofharmonic,rhythmic,spectral,andtimbralsimilaritymetrics.Furthermore,itrevisesandimprovessomeinterfacedesignlimitationsidentifiedintheformermodelsoftwareimplementation.Anewuserinterfacedesignbasedoncross-modalassociationsbetweenmusicalcontentanalysisandinformationvisualisationispresented.Inthisgraphicmodel,alltracksarerepresentedasnodeswheredistancesandedgeconnectionsdisplay theirharmonic compatibility as a result of a force-directedgraph.Besides,avisuallanguageisdefinedtoenhancethetool’susabilityandfostercreativeendeavourinthesearchofmeaningfulmusicmashups.

KeywoRDSCross-Modal Associations, Dissonance, Force-Directed Layout, Harmonic Mixing, Information Visualisation, Music Mashup, Perceptual Relatedness, Rhythmic Density, Spectral Region, Timbral Similarity

INTRoDUCTIoN

Mashupcreationisamusiccompositionpracticestronglylinkedtovarioussub-genresofElectronicDanceMusic(EDM)andtheroleoftheDJ(Shiga,2007).Itentailstherecombinationofexistingpre-recordedmusicalaudioasameansofcreativeendeavour(Navas,2014).Thispracticehasbeennurturedbytheexistingandgrowingmediapreservationmechanismsthatallowuserstoaccesslargecollectionsofmusicalaudioindigitalformatfortheirmixes(Vesna,2007).However,thescalabilityof thesegrowingaudiocollectionsalsoraises the issueof retrievingmusicalaudio thatmatchesparticularcriteria(Schedl,Gómez,&Urbano,2014).Inthiscontext,bothindustryandacademiahavebeendevotingefforttodeveloptoolsforcomputationalmashupcreation,whichstreamlinethetime-consumingandcomplexsearchforcompatiblemusicalaudio.

Early research on computational mashup creation, focused on rhythmic-only attributes,particularlythoserelevanttothetemporalalignmentoftwoormoremusicalaudiotracks(Griffin,Kim&Turnbull,2010).Recentresearch(Davies,Hamel,Yoshii,&Goto,2014;Gebhardt,Davies,

https://orcid.org/0000-0003-3884-2687


21

&Seebe,2016;Bernardes,Davies,&Guedes,2018)hasexpandedtherangeofmusicalattributesunderconsiderationtowardsharmonic-andspectral-drivenattributes.Theformeraimstoidentifythedegreeofharmoniccompatibilityinmusicalaudio,commonlyreferredtoasharmonicmixing.Thelatteraimstoidentifythespectralregionoccupiedbyaparticularmusicalaudiotrackacrossthefrequencyrange(e.g.,theconcentrationofenergyinlow,middle,andhighfrequencybands),whichcanthenguidethespectraldistributionofthemix.

Theinterfacedesignofearlysoftwareimplementationmodelsadoptsaone-to-manymappingstrategybetweenauser-definedtrackandarankedlistofcompatibletrackstoshowtheresultstotheuser(MixedinKey,n.d.;NativeInstruments,n.d.;Daviesetal.,2014).Recently,Bernardesetal.(2018)proposedaninterfacedesignwhichadoptsamany-to-manymappingstrategy,whichoffersaglobalviewofthecompatibilitybetweenalltracksinamusiccollectionandpromotesserendipitousnavigation(Figure1).Itrepresentseachaudiotrackinacollectionasagraphicalelementinanavigable2-dimensionalinterface.Distancesamongtheseelementsindicateharmoniccompatibilityandtheadditionalgraphicvariablesoftheseelements,suchascolourandshape,indicaterhythmicandspectralinformationrelevanttomashupcreation.Byexposinguserstothecompatibilitybetweenalltracksinacollection,thisinterfacedesignaimstopromoteanoverviewoftherelationsbetweentracks.

Inshort,advancesincomputationalmashupcreationmodels,emphasizeagradualincreaseinthenumberofextracteddata-drivenattributesfrommusicalaudioandaglobalviewoftheaudiocollectionsthroughinformationvisualization.Thistendencyacknowledgesthesubjectivenatureofthetaskandenhancesthedegreeofpersonalizationinthesearchforcompatibleaudioinmashupcreation.However,scalabilityoftheseaudiocollectionsnowraisesconcernsattheusabilitylevelinmany-to-manyinterfacedesign(Bernardes,Davies,&Guedes,2018;Maçãsetal.,2018).Figure1b)highlightsthethreemainlimitations:(i)theclutterresultingfromthesuperpositionofgraphicelements;(ii)thestaticrepresentationofthetracks,whichdoesnotpromoteafinerexplorationofparticulardenseareas;and(iii)thereducednumberofexistinggraphicattributesinthevisualtracksrepresentation todepictmusical audiocontent-driven information.From these limitations itwaspossibletodefinethethreemaingoalsforthepresentwork:(i)thepreventionofoverlappinggraphicelementsandsubsequentsimplificationof thevisualisation;(ii) thecreationofasystemcapableofdynamicallyadapt to theuser interactions;and(iii) thecompleterepresentationof the tracks’musicalcharacteristicsandtheirharmoniccompatibility.Toaddresstheidentifiedinterfacedesignlimitations,itwasadoptedinMaçãsetal.(2018)amethodologybasedontheThreeCycleViewofDesignScienceResearch(Hevner,2007).Thismethodologypromotedtheiterativeimplementation

Figure 1. Screenshot of the visualisation of the original MixMash interface representing (a) 50 musical audio tracks and (b) 200 musical audio tracks. Refer to Bernardes et al. (2018) for a detailed interpretation.


22

ofthevisualisationmodelandthedefinitionofthevisualrepresentationsoftheattributesofmusicalaudiotracks.

The current paper extends Maçãs et al. (2018) in four aspects. First, it provides a detailedoverviewofcomputationalmashupcreation,fromtheearlyrhythmic-onlyalignmentstothecurrentmultidimensional attribute spaces, which to the best of our knowledge has not been addressedelsewhere.Second, itexpands therangeofmusicalaudioattributesunderconsideration,notablyincluding timbreasa relevantdimension, following recentevidenceon its significant impactonlisteningpreferenceaslistenersareabletoreliablyevokechangesintimbretowardstheirpreferences(Dobrowohl,Milne,&Dean,2019).Third, itdetailsthesignalprocessingunderlyingallmetricsadoptedtoextractcontent-driveninformationfrommusicalaudio.Fourth,theforce-directedalgorithm,visualmappingsandinteractiveinterfacearethoroughlydescribed.

Theremainderofthepaperisorganisedasfollows.TheBackgroundsummarisestherelatedwork.MixMash:CompatibilityMethoddescribestheaudioanalysismethodsthatsupportanovelmusic visualisation system for assisting music mashup creation. The Methodology presents thedevelopmentstrategiesforthevisualisationmodel.MixMash:VisualisationModelpresentsanewapproachforthevisualisationofcompatiblemusicalaudiotracks,adescriptionoftheestablishedassociationsbetweengraphicelementsandmusicalaudioattributes,andtheinterfacedesign.TheConclusionstatestheconclusionsofthisworkanditsfuturedirections.

BACKGRoUND

Thepresentsectionisdividedintothreeparts,inwhichtheauthorspresentrelatedworkconcerningthefollowingtopics:harmonicmixingmethods,thevisualisationmethodsappliedtorepresentmusic,andthecharacterisationofforce-directedgraphsandtheirapplicationinthevisualisationofmusiccollections.

Harmonic MixingTherearefourmajorharmonicmixingmethodsintheliterature:keyaffinity,chromavectorssimilarity,sensorydissonanceminimisation, andhybrid (hierarchical)models.The initial three approachesfocusonsingleharmonicattributesonly,andthelatterapproachprovidesahierarchicalviewoverharmoniccompatibility.

Keyaffinityisoneoftheearliestcomputationalapproachestoharmonicmixing.Ithasbeenproposedbyindustry(MixedinKey,n.d.,NativeInstruments,n.d.)andiscomputedasdistancesintheCamelotWheelorcircleoffifthsrepresentation(MixedinKey,n.d.).Thisapproachenforcessomedegreeoftonalstabilityandlarge-scaleharmoniccoherenceofthemashupbyprivilegingtheuseofthesamediatonickeypitchset.

Chromavectorssimilarityinspectsthecosinedistancebetweenchromavectorrepresentationsofpitch-shiftedversionsoftwogivenaudiotracksasameasureoftheircompatibility(Daviesetal.,2014;Lee,Lin,Yao,Li,&Wu,2015).Thesimilarityistypicallycomputedatthetimescaleofbeatdurations,thusprivilegingsmall-scalealignmentsoverlarge-scaleharmonicstructurebetweenaudiosliceswithhighlysimilarpitchclasscontent.

Sensorydissonancemodelsfollowthesamelocalandsmall-scalealignmentstrategyaschromavectorssimilaritybetweenpitch-shiftedversionsofoverlappingmusicalaudio,yetadoptamorerefinedmetricofcompatibility,whichaimstominimizetheircombinedlevelofsensorydissonance(Gebhardt,Davies&Seebe,2016);amotivationwell-rootedintheWesternmusical traditionbyfavouringalessdissonantharmoniclexicon.

Recently,ahybridhierarchicalmodelforharmonicmixinghasbeenproposedbyBernardesetal.(2018).Itcombinespreviousapproachesfor(small-andlarge-scale)harmoniccompatibilityinasingleframework.Furthermore,itproposesanovelinterfacedesignapproach,whichoffersaglobal


23

viewover theharmoniccompatibilityofanentiremusiccollection (many-to-many),beyond theexistingone-to-manyrelationshipsbetweenauser-definedtrackandanaudiocollection(Bernardesetal.,2018).

Music VisualisationHistorically,numerousartistshavecreatedaudiovisualassociations,laterreferredtoasgraphicnotationandvisualmusic.PioneeringworksbyKandinsky,Pfenninger,Cage,Fischinger,andWhitneyexplorecombinationsofvisualprinciples—mainlycolourandshape—toemphasisetheaudiovisualexperience(McDonnell,2007).Inthepresentwork,thevisualtranslationofacollectionofmusicalaudiotracksisapproachedfromtwodifferentpointsofview:musicvisualisationandthevisualrepresentationofalargecollectionofmusicalaudiotracks.Regardingmusicvisualisation,someauthorshavetriedtosolvethisproblembyfocusingonthegeometryofmusicalstructure(Bergstrom,Karahalios,&Hart,2007),whileothershavefocusedonasolutionbasedonmappingsbetweenaspecificsetofmusicalcharacteristicsandsomevisualcharacteristics(Snydal&Hearst,2005;Wattenberg,2002;Sapp,2001).Forexample,Wattenberg(2002)usesarcdiagramstoconnectsequencescontainingthesamepitchcontent,revealingthestructureofmusicalcompositions.Ontheotherhand,Sapp(2001)presentedamulti-timescalevisualisationoftheharmonicstructureandthekeyrelationsinamusicalcomposition.AnotherexperimentalapproachhasbeenpresentedbyRodrigues,Cardoso,andMachado(2016),whereavisualisationmodeliscreatedtoprovideaperceptuallyrelevantexperiencefortheuser.Overall,theaforementionedvisualisationsdonotallowgreatmodularityofdata,oftenbindingthevisualclarityofeachelementandlimitingthecomparisonofothermusicalcharacteristics.

Although visualisations of large collections of musical audio collections have already beenaddressedbysomeauthors(Grill&Flexer,2012;Hamasaki&Goto,2013;Rauber,Pampalk,Merkl,2003;SchwarzandN.Schnell,2009;Gulik,Vignoli,&VandeWetering,2004),itstillisanareainneedofgreatdevelopment.GrillandFlexer(2012),similarlytotheworkofBernardesetal.(2018),developedavisualisationstrategycapableofrepresentingperceptualqualitiesfromalargecollectionofsounds.Althoughtheirmusicalfocuswasinsoundtextures,theyaimedatfindinganintuitiveandmeaningfulinterface.Forthispurpose,theybuiltanaudiovisuallanguagebasedonthecross-modalmechanismsofhumanperception.Inthisproject,subjectswereabletosuccessfullyassociatesoundswiththecorrespondinggraphicrepresentations.HamasakiandGoto(2013),Guliketal.(2004)andSchwartzandSchnell(2009)alsoproposedinteractivevisualisationsofseverallayersofinformation;however,theydonotrepresenttheperceptualrelevanceofthemusicalcharacteristicsatavisuallevel.

Force-Directed GraphsThe visualisation of graphs handles the representation of relational structures in data, aiding intheanalysis,modelling,andinterpretationofcomplexnetworksystems(Meirelles,2013).Graphvisualisation is characterised by the existence of two main elements: (i) nodes, representing anentity(e.g.,person,cell,machine);and(ii)edges,representingtherelationshipbetweennodes.Suchmodelsareoftenappliedtolargeandcomplexdatasets,whichentailsasetofproblemsrelatedtoperformanceandclutter.Asthegraphsgrowinsize,therequiredvisualspaceandcomputationalresourcesalsoincrease(Herman,Melançon,&Marshall,2000).Tosolvethis,clusteringtechniquesareappliedbygroupingsimilarnodesby their semanticsand/orpositionon thegraph.Throughclustering,itispossibletoreducevisualclutterandcomplexity,enhanceclarityandperformance,andcreateasimplifiedoverviewofthenetwork’sstructure(Hermanetal.,2000;Kimelman,Leban,Roth,&Zernik,1994).

Thepositioningofnodesinspacecanbedefinedthroughadifferentsetofgraphlayouts,suchasarcdiagrams,treemaps,circular,andforce-directedlayouts.Inthispaper,thelatterisadopted.Force-directedgraphsarebasedonaphysicalsystemthatorganisesthenetworkthroughforcesofrepulsion and attraction applied continuously to each node. This technique facilitates the visual


24

interpretationoftheinherentstructureofthedata,improvingtheanalysisandcomprehensionoftherelationswithincomplexnetworks(Jacomy,Venturini,Heymann,&Bastian,2014).

Force-directed graphs are used in a variety of areas, such as biology, medicine, literature,sociology(EnrightandC.A.Ouzounis,2001;Gohetal.,2007,Chen,2006;HeerandBoyd,2005;Gilbert,Simonetto,Zaidi,Jourdan,&Bourqui,2011;Fuetal.,2007),butalsointhevisualisationofmusiccollections(Vavrille,n.d.;Gibney,n.d.).IntheworkofMuelder,Provan,andMa(2010),aforce-directedlayoutwasappliedtovisualisemusiclibraries.Eachpieceofmusicrepresentsanode,positioneddependingonitssimilaritieswithothermusicalaudiotracks(e.g.,sameartist).Songrium(Hamasaki&Goto,2013),isamusicbrowsingservicethatvisualisesrelationsamongoriginalsongsandtheirderivativeworks.Eachnoderepresentsavideo-clipandeachedgeconnectsanoriginalworktoitsderivatives.Inaddition,songswithsimilarmoodsgetstrongerforces,thusarepositionedclosertogether.

Thecurrentworkexpandsthestate-of-the-artbyapplyingaforce-directedgraphtothemashupcreationprocess.Toenabletheusertoanalysethegraphandrelatemusicalaudiotracksfromlargecollections,theharmoniccompatibilitymetricsareappliedtotheforcesofthegraphnodes.Thisimprovesthevisualseparationofdistinctmusicalaudiotracks,whichcanhaveapositiveimpactonusercreativity(Henry,Fekete,&McGuffin,2007).

MIXMASH: CoMPATIBILITy MeTHoD

MixMashisasoftwareapplicationwhichaimstoassistusers infindingmusicalmashupsin thecontextofmashupcreation.ItbuildsonmetricsandmethodspresentedinBernardesetal.(2018)and expands the state-of-the-art of harmonic mixing by providing a greater amount of relevantinformationtotheprocessofmusicmashup.Itsmainnoveltyliesinahierarchicalharmonicmixingmethod,whichincludesmetricsforbothsmall-andlarge-scalestructurallevels,i.e.,local(e.g.,beatsorphrases)andglobal(e.g.,largesectionsoroverallmusicalmashup)harmonicalignmentsbetweenmusicalaudiotracks,respectively.Moreover,thismethodconsidersthreeadditionaldimensionsthatcanhelpusersdefiningthecompatibilityofmusicalaudiotracksandremainingcompositionalgoalsintermsofrhythmic(onsetdensity),spectral(region)andtimbralqualities.

Topromotean intuitivesearch forcompatible tracks inamusiccollection,amany-to-manymappingstrategywaspreviouslyintroducedintheinterfacedesign.Thisdesignopposestherankedtrack list toauser-definedtrack,adopted inprevioussystemsharmonicmixingsoftware(MixedinKey,n.d.;Native Instruments,n.d.),which is (i) inefficientcomputationally,as it recomputesintensiveaudiosignalanalysiseverytimeadifferentaudiotrackisselectedastarget,and(ii)limitedinpromotingcreativeendeavorandserendipity(Bernardesetal.,2018).Aflexiblemany-to-manymappingstrategywasallowedbytheadoptionofnovelsignalprocessingmethodsforsmall-andlarge-scaleharmoniccompatibilitymetricsinaconfinedspatialconfiguration.ThismethodisatthebasisoftheMixMashvisually-driveninterfacestrategy.Thesignalprocessingmethodsusedtocomputetheseharmoniccompatibilitymetrics,followedbytheadditionalrhythmic,spectral,andtimbralattributesaredescribednext.

Small- and Large-Scale Harmonic CompatibilityMixMashadoptstheperceptually-motivatedTonalIntervalSpace(Bernardes,Cocharro,Caetano,Guedes,&Davies,2016)forrepresentingtheharmoniccontentofmusicalaudiotracks.Eachtrackexistsasa12-dimensional(12-D)TonalIntervalVector(TIV),T k( ) ,whoselocationsrepresentuniqueharmonicconfigurations.AnaudiotrackTIV,T k( ) ,iscomputedastheweightedDiscreteFourierTransform(DFT)ofanL

1normalizedchromavector,c n( ) ,suchthat:


25

T k w k c n e kn

N j kn

N( ) = ( ) ( ) ∈=

− −

∑0

1 2π

, (1)

whereN = 12 isthedimensionofthechromavector,eachofwhichexpressestheenergyofthe12pitchclasses,andw k( ) = { }3 8 11 5 15 14 5 7 5, , . , , . , . areweightsderivedfromempiricalratingsofdyadsconsonanceusedtoadjustthecontributionofeachdimension k (Bernardesetal.,2018).k issetto1 6≤ ≤k forT k( ) sincetheremainingcoefficientsaresymmetric.T k( ) usesc n( )

whichisc n( ) normalizedbytheDCcomponentT c nn

N

00

1

( ) = ( )=

−

∑ toallowtherepresentationand

comparisonofmusicatdifferenthierarchicallevelsoftonalpitch(Bernardesetal.,2016).Torepresentvariable-lengthaudiotracks,thechromavectors,c n( ) ,resultingfrom16384samplewindowsanalysisata44.1kHzsamplingrate(≈372ms)with50%overlapareaccumulatedacrossthetrackduration.

FromtheaudiotracksTIVs,twometricsthatcapturetheharmoniccompatibilitybetweenTIVstobemixedarecomputed.Ofnoteisthesplitbetweensmall-andlarge-scaleharmoniccompatibility,whichroughlycorrespondtothesoundobjectandmesoormacrotimescalesofmusic,respectively.Inotherwords,thesmallscaledenotesthebasicunitsofmusicalstructure,fromnotestobeats,andthelargescaleinspectsthestructurallevelsbetweenthephraseandtheoverallmusicalpiecearchitecture(Roads,2001).Inthecontextofthecurrentwork,thefirstaimsmostlyatfindinggoodharmonicmatchesbetweenthetracksinacollection,andthesecondinguaranteeingcontrolovertheoverallharmonicstructureofamix,i.e.,thetonalchangesatthekeylevelacrossitstemporaldimension.

Equation2computesthesmall-scaleharmoniccompatibility, Si p,

,betweentwogivenaudiotracks,i andp ,representedbytheirTIVs,T k

i ( ) andT kp ( ) ,asthecombinationoftwoindicators:

perceptual relatedness, Ri p,

, (Equation 3) and dissonance, Di p,

, (Equation 4). The smaller theperceptualrelatedness,R

i p,,thegreatertheaffinitybetweentwogiventracks,asshownbyBernardes

etal.(2016).Thesmallerthedegreeofdissonance,Di p,

,thegreatertheircompatibility,asshownbytheempiricaldatainGebhardtetal.(2016).

S R Di p i p i p, , ,= ⋅ (2)

where:

R T k T ki p

ki p,

= ( )− ( )=∑1

6 2 (3)

DaT k a T k

a a w ki p

i i p p

i p

,= −

( )+ ( )+ ( )

1 (4)

aianda

paretheamplitudesofT k

i ( ) andT kp ( ) ,respectively.


26

Large-scaleharmoniccompatibility, Li p,

, isaderivationof theperceptual relatedness, Ri p,

,indicator,asitexpressestherelatednessofagiventrackTIVfromthem = 24 majorandminorkeyTIVs,andcanbeinterpretedasthedegreeofassociationofagiventracktoamusicalkey(Bernardes,Davies, & Guedes, 2017). As such, the large-scale harmonic compatibility can be computed byinterpretingT k

i ( ) andT kp ( ) inEquation3asatrackTIVandakeyTIV,respectively.Them = 24�

majorandminorkeysTIVarecomputedbyadoptingthe12shiftsoftheCmajorandCminorkeyschromavectors,c n( ) ,showninFigure2,inEquation1.

Rhythmic, Spectral and Timbral AttributesThreeadditionaldescriptionsofrhythmic,spectral,andtimbralaudiotrackattributesarecomputed.Theyaresubsidiaryoftheprimarysmall-andlarge-scaleharmoniccompatibilitymetricsandaimtorefinethesearchamongcompatibleaudiotracks.Next,adescriptionofthemetricandtheirmusicalinterpretationinthecontextofmusicmashupcreation,mostnotablyinMixMash,isprovided.ReaderscanrefertoBrent(2010)foracomprehensivedescriptionoftheircomputation.

Eachaudiotrack’srhythmiccontentisdescribedbyitsnoteonsetdensity,Oi,ofamusicaltrack

i ,andiscomputedbyathreefoldstage.First,aspectralfluxfunctionforeachtrackiscomputed,

Figure 2. Sha’ath’s (2011) key profiles for the C major and C minor keys


27

usingthetimbreIDlibrary(Brent,2010)withinPureData.Thisfunctiondescribestheamountofnoveltyfromawindowedpowerspectrumrepresentationoftheaudiosignal(windowsize≈46mswith 50% overlap). Second, the peaks from the function above a user-defined threshold, t , areextractedandinterpretedasnoteonsetlocations.Priortothepeakdetectionstage,abi-directionallow-passIIRfilter,withacut-offfrequencyof5Hz,wasappliedtoavoidspuriousdetections.Thenoteonsetdensity,O

i,isthencomputedastheratiobetweenthenumberofonsetsandtheentire

durationofthetrack(inseconds).Anindicatorofthespectralregion,B

i,ofamusicaltrack i ,isgivenbythecentroidofthe

accumulatedBarkspectrum,b, acrossthedurationofanaudiotrack(Equation5).TheBarkspectrum,b ,iscomputedbythetimbreIDlibrary(Brent,2010),whichbalancesthefrequencyresolutionacrossthehumanhearingrange,bywarpingapowerspectrumrepresentationtothe24criticalbands,h ,ofthehumanauditorysystem(i.e.,Barkbands).

Bb h

bi

h h

h h

=⋅

=

=

∑∑�

�1

24

1

24 (5)

wherebh

istheenergyoftheBarkbandh .TheBiindicatorcanrangefrom1to24Barkbands.

FollowingPachetandAucouturier(2004),thetimbralsimilarity,Ci p,

,betweentwotracks, i andp ,isgivenbythecosinesimilaritybetweentheirmel-frequencyspectrumcoefficients(MFCC),MiandM

p(Equation6).MFCCvectors,M ,include38componentsresultingfromapplyinga

100mel-scaledfilterbankspacinginthetimbreIDlibrary(Brent,2010).

CM M

M Mi p

i p

i p,=

⋅ (6)

Thetimbralsimilaritymetric,Ci p,

,rangesbetween1and-1,whichcorrespondstotrackswithequaltimbreandthemostdissimilartimbre,respectively.

MeTHoDoLoGy

ThemethodologyusedforthedevelopmentofthepresentworkisbasedonA Three Cycle View of Design Science Research(Hevner,2007).Thismethodologyfacilitatesthedevelopmentofinteractiveapplicationsandpromotesquickeriterationsbetweentheseveralphasesoftheapplicationdevelopment,suchasthevisualisationimplementation,thevisualcomponent’svalidation,andtheguidelinesforitsrefinement(Figure3).

Thepresentmethodologyisdividedintothreeparts:TheRelevanceCycle,theDesignCycle,andtheRigorCycle.Firstly,intheRelevanceCycle,theresearchcontextisdefined.Therequirementsandproblemsfromtheprevioussystemwerehighlighted,leadingtothedefinitionofthekeyobjectivesforthenewsystem(seeIntroduction).

Secondly,intheDesignCycle,thenewsystemwasdevelopedthroughaloopofresearchbetweenthe implementation of thevisualisation components and their assessment. This iterative processpromotestheanalysisandimprovementofpreviousstepsandtheexperimentationandrefinementofdifferentapproaches.Inparticular,thiscyclewasdividedintofourmainphases:(i)thedataanalysis,whichisrelatedtothecomputationoftheharmoniccompatibilitymatrix(seeMixMash:Compatibility


28

Method);(ii)thedesignofthevisualisationmodel,whichrelatestotheiterativeprocessbetweendefinitionof thegraphicalvariablestorepresent themusicalcharacteristicsandthevisualisationmodelanditsvalidationwithexpertsinInformationVisualisation;(iii)theimplementationofthemodel,wheretheForceAtlas2algorithmwasstudiedandimplemented;and(iv)theevaluationofthesystem.ThedefinitionofsuchphasesisalignedwiththemethodologiesproposedbyWilkins(2003)andFry(2004)fromtheInformationVisualisationfield.Thevalidationofthesystemwasconductedthroughan initial informalevaluation (Lametal.,2011), inwhich thevisualisationcomponentswerediscussedbetweentheauthorsandexternalexpertsfromthevisualisationfieldtoassessthevisualisationintuitivenessandusability.

Finally,theRigorCycleconnectswiththecentralDesignCyclethroughaniterativeexchangeofknowledge,bothfromscientificfoundationsasfromthevisualisationvalidation.Thisfinalcycleischaracterisedbyfine-tuningthevisualisationmodelthroughknowledgeacquiredfromtheevaluationsandrelatedwork.

MIXMASH: VISUALISATIoN MoDeL

Thevisualrepresentationoftherelationshipsbetweenaudiotrackswereguidedbythreeobjectives(seeIntroduction):(i)theimplementationofanadaptivevisualisationmodel(seesubsectionForce-directedSystem);(ii)thedistinctionbetweenthedifferentsoundcharacteristicsoftheseveralmusicalaudiotracks(seesubsectionGraphicVariablesandAudiovisualMappings);and(iii)theconceptionandimplementationofasimpleandintuitiveinterface(seesubsectionInterfaceDesignandInteraction).

Toimprovethescalability,interaction,andvisualisationoftheinterfacepreviouslypresentedinBernardesetal.(2018),aforce-directedgraphlayoutbasedontheForceAtlas2algorithm(Jacomy,Venturini,Heymann,&Bastian,2014)wasimplemented.Thisvisualisationtechniqueenabledthecreationofanemergent,organic,andappealingenvironmentfortheuser.Furthermore,itimprovesthereadabilityofthepreviousinterfaceofBernardesetal.,bypreventingoverlaps,arrangingthetracksinspacebytheirharmoniccompatibility,andbyenablingtheusertoexploreandinteractwithtracksofinteresteasily.Additionally,tocharacterise,distinguishandimprovethereadabilityofthetracksandtoaugmentthegraphicattributesthatrepresentanddistinguishthetracks,agraphicalrepresentationforthemusicalaudiotrackswasalsostudiedandapplied.

Toenabletheusertofilterthetrackcollectionaccordingtoharmonic,rhythmic,spectral,andtimbral attributes, a graphic interface was also implemented (Figure 4). With the force-directedalgorithm,theusercandetectthemostharmonicallycompatibletracksthroughtheirvisualproximityinthecanvas.Thisiscausedbytheforcesappliedtoeachtrack,whichdependdirectlyontheharmoniccompatibility.However,theusercanmanipulatetheimpactoftheforcesofattractionandrepulsion

Figure 3. The Three Cycle View of design science research


29

betweentracks,easingthecomprehensionofmoreclutteredzones.Additionally,highlycompatibletracksareclusteredtoreduceundesiredclutter.Theseclustersarevisuallydistinguishedfromthetracksandcanbeexpandedorwithdrawnthroughinteraction.Theforce-directedalgorithmandeachcomponentoftheinterfacearedescribedinmoredetailinthefollowingsubsections.

Force-Directed SystemTheForceAtlas2algorithm(Jacomy,Venturini,Heymann,&Bastian,2014)canbecharacterisedbyitsabilitytoplacethenodeswithinagraphaccordingtotheirconnectionsweight.Thealgorithmsimulatesaphysicalsystemthatspatiallyarrangesthenetwork’snodesinanautomaticform.Thenodeshaveforcesofrepulsiontopreventthemfromoverlapping,andtheedgesbetweennodesapplyforcesofattractiontobringthenodescloser.Theseedgeshavedifferentforcevaluesaccordingtothesimilaritybetweennodes(e.g.,theirharmoniccompatibility).Byapplyingcontinuously,thedifferentforces,thegraphconvergestoabalancedstatethataidsthesemanticinterpretationofthenetwork.TheForceAtlas2algorithmwasfullyimplemented,thusforadetaileddescriptionofthealgorithm,refertoJacomyetal.(2014).

Inthisproject,twotypesofnodesaredefined:theonesrepresentingeachmusicalaudiotrackinthecollection,t,andtheonesrepresentingeachoneofthem=24majorandminormusicalkeys.ThevisualdistinctionbetweenthenodesisdiscussedatlengthinsubsectionGraphic Variables and Audiovisual Mappings.Byrepresentingthemusicalkeys,m,throughnodes,andconsequentlytheirrelationtothemusicalaudiotracks,themostcompatiblekeyofeachmusicalaudioisindicated.Thisenablestheusertovisuallydetectthetracksandsetsoftracksmoreharmonicallycompatiblewiththedifferentkeysandrelatetracksaccordingtothiscompatibility.

Theweightoftheedgebetweentwonodesismappedaccordingtothecompatibilityvalueinthet + msquarematrix.Thissquarematrixiscomputedthroughagivenauser-definedcollectionoftaudiotracksandm=24majorandminorkeysandexpressesthemetricsforsmall-scale(Si,p)and large-scale (Li,p) scaleharmoniccompatibility (seeMixMash:CompatibilityMethod).As inForceAtlas2, theweightdefinestheforceofattractionbetweennodes.Forthisproject, themore

Figure 4. Screenshot of the visualisation interface. On the left, the interaction panel; at the centre, the graph.


30

compatiblethenodes,thehighertheforceofattraction,andconsequentlythecloserthenodes.Aforceofrepulsionisappliedtoallnodestoavoidthemtooverlap.Theseforcesareappliedtothenodesindependentlyoftheirtype,facilitatingtheinterpretationofthegraphandinteractionwhensearchingforharmonicallycompatibletracks,t.

Twomechanismswereimplementedtoenabletheusertorefinethegraphlayout:(i)aconnectivitythreshold;and(ii)musicalkeyrestriction.Bothmechanismscanbeexploredbytheuserthroughtheleftpaneloftheinterface(Figure4).Throughtheconnectivitythreshold,theusercandefineathresholdvaluetodeterminewhethertwonodesareconnected.Foreachnode,theconnectionstoothernodesonlyoccurwhentheirharmoniccompatibilityvalueislowerthanthepredefinedvalue.Thesecondmechanismlimitstheconnectionsbetweennodesandkeys.Throughthismechanism,theusercandefinewhetheratrackisconnectedtoallcompatiblemusicalkeysoronlytoitsmostprobablekey.Thismechanismisintendedtoreduceclutterandenhancetheassociationbetweenkeysandmusicalaudiotracks.

Asthenumberoftracksinacollectioncanvarygreatly,anagglomerativeclusteringalgorithm(Rokach&Maimon,2005)isimplementedtopreventclutteredgraphs.Thisalgorithmaggregatesthenodesbytheircompatibilityvalues.Aminimumnumberofthreenodesperclusterisrequiredtopreventsmallclusters,whichwouldn’tenhancetheclarityofthevisualisation.Eachclusterisalsoconnectedtothecompatiblenodesandclusters,thusexposingtheirharmonicrelationtotheneighbourhoodelements.Thesecompatiblenodesareretrievedfromthelistofcompatiblenodesofeachnodewithinthecluster.Ifanouternodeisconnectedtoaninnernodeofacluster,aconnectionbetweenthenodeandtheclusterisestablished.Theattractionforcebetweenaclusterandacompatiblenodeisequaltotheaverageforceofallforcesbetweeninnernodesandtheconnectedouternode(asdepictedinFigure5).

AsinForceAtlas2,allnodesgravitatearoundthecentreofthecanvas.Thiseffectresultsfromtheattractionforceappliedtoallnodestowardsthecanvascentrepoint.Thegravitationalforceissignificantlyweakerthantheothersand,asitisappliedequallytoallnodes,itdoesnotinterferewiththedistancebetweennodes,andonlypreventsthenodesfromdispersinginthecanvas.

Bydefault, themusicalkeynodesarepositionedby theforce-directed layout,dependingontheirrelationswiththeothertracks.Thiscausesthekeynodestohavedifferentpositionsateveryrun,whichcancreatesomeuserfatiguewhilesearchingforaspecificmusicalkeyanditsconnectedtracks.Tofacilitatethissearch,amechanismthatpositionsthekeynodesaccordingtotheCamelotWheelorcircleoffifthsrepresentationwasimplemented(Figure6).Withthismechanism,allmusicalkeyshaveafixedpositioninthecanvas,andonlythetracknodesareinfluencedbytheattractionandrepulsionforcesofthealgorithm.

Figure 5. Scheme of the forces representing outer and inner nodes of a cluster. The forces between clusters and nodes (b) are computed through the average force between the inner and outer nodes (a). The lines depict compatible nodes.


31

Graphic Variables and Audiovisual MappingsThevisualrepresentationofdataelementsstronglyimpactsthevisualisationoflargeamountsofdata.Inthissection,theadoptedvisualrepresentationofmusicalconceptsisdiscussedinlightofacarefullydesignedinteractivevisualisation.

The development of the visualisation model complies with the following guidelines andsubsequent challenges: (i) to each track there is a corresponding visual representation based onitsmusicalattributes,creatingaconsistentvisualfeedbackbetweensimilartracks;(ii)perceptualfoundationsareusedtoguidetheaestheticalchoicesconcerningvisualrepresentationoftracks;(iii)anaturalandintuitiveinteractionwiththetoolispromotedallowingtheusertoeasilynavigateamongtrackstocreatehis/hermashup.

Foreachtrack,thespectralregion,Bi,onsetdensity,Oi,andtimbralsimilarity,Ci,p,aremappedtoacorrespondingvisualvariable.Additionally,bothspectralregion,Bi,andonsetdensity,Oi,aresubdividedintothreelevelsofmagnitude,allowingamoreaccurateinterpretationandanalysisofmusicaldata.

Thespectralregionattribute,Bi,issplitintohigh,medium,andlowregions,anditscorrespondingvisualrepresentationisprimarilycharacterisedbyshape.Althoughthereisatendencytoassociatecolour with pitch levels in similar audiovisual mapping problems (McDonnell, 2007; Mudge,1920;Ward,Huckstep,&Tsakanikos,2006), ithasalsobeenproposed that theuseof shapeormonochromecolourismoreefficientthanhuetodistinguishseverallayersofinformationforthehumaneye(Arnheim,1974;Chatterjee,2013).Basedonsuchstudies(Arnheim,1974;Chatterjee,2013;Ramachandran2003;Spence,2011),high-frequencyregionsofthespectrumareassociatedwithsharpcontours(triangle),lowfrequencieswereassociatedwithroundedcontours(circle),andmediumfrequenciesareassociatedwithneutralcontours,achievedthroughtheuseofstraightlinesoftherectangle(Figure7a).Colourhue,isusedtoreinforcetheseassociations.Higherfrequenciesarerelatedtoacoldcolour(blueandlowfrequenciestoawarmcolour(orange;seeFigure7c).

Figure 6. Fixing the keys in the circle of fifths. By clicking on the Fix Tones button, all nodes that represent a key are fixed on the canvas according to their positioning in the circle of fifths.


32

Onsetdensity,Oi,issplitintohigh,mediumandlow-densitylevels,andthenconveyedinthevisualdomainasaparameterofshape-filling,asitrelatestothenumberofnotesinatrack.Lowdensitycorrespondstoanemptyshape(onlycontoursvisible),mediumdensitytoahalf-filledshape,andhighdensitytoacompletelyfilledshape(Figure7b).

Timbreisoftenreferredtointheliteratureasthecolourofinstruments(Mudge,1920;Wardetal.,2006).Inspiredbythisdefinition,timbralsimilarity,Ci,p,isconveyedinthevisualdomainthroughcolouredcirclesinthetopsideofthenodes,whichareonlyvisiblewhenacertainnodeisselected.Thechoiceofcoloursdoesnotrelyonanytypeofassociation,andforthisreason,arerandomlyselectedfromapredefinedsetofcolours.Theysimplyaimtoprovideacleardistinctionbetweendifferenttimbreswhenmultipletimbresareselected.Byselectingonetracknode,acolouredcircleandalineconnectingtheformertothenodearedrawn(Figure8a).Then,alltrackswithsimilartimbreswillalsogainacirclecolouredwiththesamecolourcodeastheclickedone(Figure8b).Ifatracknodeissimilartomultipletimbres,multiplecolouredcircleswillbedrawnoverthenode(Figure8c).

Thevisualdistinctionbetweennodesrepresentingmusicalaudiotracksandkeysishighlightedbythecolouredoutlineofthekeynodes.Thelatterhasaredoutlineandincludethetypographicrepresentationofthekey’stonicpitch,whichallowsthedirectreadingofthekey(Figure4).

Thevisualrepresentationofaclusterisdefinedbytherespectiveinnernodestoavoidtheuseofadditional(andpotentially,morecomplex)visualelements.Morespecifically,allnodesthatbelongtoaclusterarerepresentedandpositionedwithinacircularshapeoutlinedwithblackdots.Thesizeofthecircularshapedependsonthenumberofelementswithinthecluster(Figure9).Assuch,theusercandifferentiatetheclustersfromthenodes,and,simultaneously,getanoverviewoftheirinnernodes.

Figure 7. Audiovisual mappings: (a) Shape—Spectrum Frequency from low frequencies (circle) to higher ones (triangle); (b) Object Fill—Onset Density. Empty shapes correspond to low density, half-filled shapes to normal/mid density, and full shape corresponds to high density; (c) Colour—Spectrum Frequency from low frequencies (orange) to higher ones (blue).

Figure 8. Timbre representations. (a) One track is selected and there are no other tracks with similar timbres. (b) One track is connected timbre wise to the selected track. (c) The track can have more than one timbre similarities (d) The track is not related to any of the selected tracks.


33

Interface Design and InteractionMixMashenablestheusertoexplorethegraphbyallowinghim/herto(i)listenandselectindividualnodetracksofinterest(Figures10a,10b,10c);(ii)highlightnodeswithinclusters(Figure10d)oraccordingtotheirsoundcharacteristics(Figure11);(iii)modifyitsorganisation(e.g.,fixingthekeysaccordingtothecircleoffifths)(Figures10e,10f);(iv)changehowthetracknodesconnecttothekeynodes(Figures10g,10h);(v)changetheconnectionthresholdbetweennodes;and(vi)adjusttheforcesofattractionandrepulsion.Thisisallaccessiblethroughaninteractivepanelontheleftsideoftheinterface(Figure4)andthroughmouseinteractions.Avideoregardingtheseinteractionsandpossibilitiescanbeaccessedathttps://vimeo.com/270076175.

Oncethesystemhasbeeninitialised,theuserwillseethenodesandclustersestablishingtheirpositioninthecentreofthecanvas.Inadditiontothefunctionalitiespresentintheleftpanel,theusercaninteractwiththevisualisationthroughmouseinteractions.Theusercanzoomandpanthegraphtoviewmoredetails.Then,theusercaninteractwiththetracksindividually.Tolistentothetracks,theuserhastomovethemouseovereachnode.Toselectanode,theuserneedstoleft-click.Withthisaction,he/shewilllistenagaintothetracksound.Toviewthecompatibletimbresofacertaintrack,theuserhastoright-click.Bydoingso,theclosesttracks(intermsoftimbre)arecomplementedwithacolouredcircleasexplainedinsubsectionGraphic Variables and Audiovisual Mappings.Interactionswiththeclusterswerealsoimplemented.Toexpandacluster,theuserhastoleft-clickoverit.AllnodesinsidetheclusterwillbeaffectedbytheforcesaccordingtotheForceAtlas2algorithm,andadoughnutshapefigurewillbemadevisible.Thedoughnutshape,positionedinthecentreofallcorrespondingnodes,behaveslikeabuttonand,whenclicked,itclosesthecluster.Iftheuserdoesmouseoveronthislattershape,allthenodesbelongingtotheclusterwillbehighlightedthroughamagentastroke(Figure10d).Finally,allthetracksthathavenosimilaritytootherswillbeplacedatthebottomrightsideofthewindow.

Figure 9. Representation of two clusters with different sizes

https://vimeo.com/270076175


34

Additionalkeyboardinteractionswereimplemented.Whenlisteningtoanaudiotrack,theusercanclicktheSkeyonthekeyboard,andthemusicwillstopplaying.Bycontinuouslyselectingnodes,theuserissavingthetracks,whichcanbeheardatthesametimebyusingthespacekey.

Overall,thissetofinteractiontechniquesareimportanttoachieveanintuitiveandmeaningfulinteractivevisualisationtoolinthecontextofmusicalmashupcreation.Withthis,theauthorsaimtoenhancetheunderstandingofthetrack’sharmoniccompatibilityandfosterusercreativity,byallowingtheusertoefficientlyexplorealargemusicalaudiocollectiontowardsspecificcompositiongoals.

Figure 10. Visual representation of the interactions with the model: (a) no selection, (b) mouse over, (c) mouse click, (d) clusters’ nodes highlight, (e) force-directed layout, (f) circle of fifths layout, (g) connections to all compatible keys, (h) connection to the most compatible key (1st tone option)


35

CoNCLUSIoN

Theauthorsproposedanovelvisualisationsystemwhichreliesonforcesofattractionandrepulsiontopositionthetracksdependingontheirharmoniccompatibility.ThevisualisationdevelopmentwasguidedbyamethodologyproposedbyHevner(2007),whichconsistsofthreecyclesandleadtoclearandconsistentinteractionsbetweenthemusicmashupandtheinformationvisualisationmodel.

Regardingtheforce-directedalgorithm,controlledlevelsofattractionandrepulsionallowthereductionofclutterinthevisualizationoflargemusiccollections(ofroughly50musicalaudiotracksormore).Clutterwasalsominimizedbytheadoptionofclusteringtechniques,whichenhancethevisualizationofcombinedhierarchicallevelsofharmoniccompatibilityinthesamerepresentationandtheuser-controloverclusteringquality(i.e.,distancethreshold).

Afluidre-organizationofthevisualizationwasachievedbydraggingconnectedelementsintheinterface,therebyenhancinghighlydenseareasofparticularinteresttotheuser.Ontheotherhand,theFixTonesstrategyexploresastaticvisualizationofthemusicalaudiocollectionbyfixingthelocationofkeysintheTonalIntervalSpace.TheresultingrepresentationisoneofthemostfamiliarmapsoftonalregionsinWesternmusic.

Theauthorswereabletoexpandthenumberofcontent-basedmusicalaudioattributesunderconsiderationtocoverbothrhythmic,harmonic,spectral,andtimbralattributes.Thedevelopmentof a specificgraphic representation supported themusicvisualisationbyprovidingaperceptualassociation,andtherefore,theintuitiveassociationbetweenvisualandmusicalattributes.Ingeneral,thepresentedvisualsolutionwasabletopromoteamorefluidvisualisation.However,itstillhassomelimitationsduetothehighnumberofsamplesthatarebeingdisplayedinrealtimetotheuser.

Asfuturework,theauthorsintendtoimprovetheclusteringalgorithmbygivingtheuserthepossibility toclusternodesaccording todifferent audiocontent-basedattributes, e.g.,key,onsetdensity,spectralregionortimbralsimilarity.Toimprovereadability,differentsolutionstothenodes’sizedependingontheircompatibilitywillbestudied,e.g.,nodeswithhighercompatibility,growinsize,emphasisinghighlycompatibletracks.Asthenumberoftrackscanincreasedependingontheuser,afish-eyezoomtechniquewillbeimplementedsotheusercanhavedetailincertainareas

Figure 11. Frequency highlight by color


36

withoutlosingcontextofthesurroundingtracks.Finally,atimelinewillbedesigned,sothatuserscanarrangeselectedtracksintime,thusenablingthecompositionofmusicalmashupswithcomplextemporalstructures.

ACKNowLeDGMeNT

The work is supported by Portuguese National Funds through the FCT-Foundation for Scienceand Technology, I.P., under the grants SFRH/BD/129481/2017 and SFRH/BD/139775/2018.Experimentationinmusic inPortugueseculture:History,contexts,andpractices in the20thand21st centuries—Project co-funded by the European Union, through the Operational ProgramCompetitivenessandInternationalization,initsERDFcomponent,andbynationalfunds,throughthePortugueseFoundationforScienceandTechnology.


37

ReFeReNCeS

Arnheim,R.(1974).Art and visual perception.UniversityofCaliforniaPress.

Bergstrom,T.,Karahalios,K.,&Hart,J.(2007).Isochords:Visualizingstructureinmusic.InProceedingsofGraphicsInterface(pp.297–304).ACM.

Bernardes,G.,Cocharro,D.,Caetano,M.,Guedes,C.,&Davies,M.(2016).Amulti-leveltonalintervalspaceformodellingpitchrelatednessandmusicalconsonance.Journal of New Music Research,45(4),281–294.doi:10.1080/09298215.2016.1182192

Bernardes,G.,Davies,M.,&Guedes,C.(2017).Automaticmusicalkeyestimationwithadaptivemodebias.InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing(pp.316–320).IEEEPress.doi:10.1109/ICASSP.2017.7952169

Bernardes,G.,Davies,M.,&Guedes,C.(2018).Ahierarchicalharmonicmixingmethod.InM.Aramakietal.(Eds.),InternationalSymposiumonComputerMusicMultidisciplinaryResearch(pp.151–170).SpringerNatureSwitzerland.

Brent,W.(2010).AtimbreanalysisandclassificationtoolkitforPureData.InProceedings of the International Computer Music Conference.AcademicPress.

Chatterjee,A.(2013).The aesthetic brain: How we evolved to desire beauty and enjoy art.OxfordUniversityPress.doi:10.1093/acprof:oso/9780199811809.001.0001

Chen,C.(2006).CiteSpaceII:Detectingandvisualizingemergingtrendsandtransientpatternsinscientificliterature.Journal of the Association for Information Science and Technology,57(3),359–377.

Davies,M.,Hamel,P.,Yoshii,K.,&Goto,M.(2014).Automashupper:Automaticcreationofmulti-songmusicmashups.IEEE Transactions on Audio, Speech, and Language Processing,22(12),1726–1737.doi:10.1109/TASLP.2014.2347135

Davies,M.,Stark,A.,Gouyon,F.,&Goto,M.(2014).Improvasher:Areal-timemashupsystemforlivemusicalinput.Proceedings of the International Conference on New Interfaces for Musical Expression,London,UK(pp.541–544).AcademicPress.

Dobrowohl,F.A.,Milne,A.J.,&Dean,R.T.(2019).Timbrepreferencesinthecontextofmixingmusic.Applied Sciences,9(8),1695.doi:10.3390/app9081695

Enright,A.,&Ouzounis,C.(2001).Biolayout—Anautomaticgraphlayoutalgorithmforsimilarityvisualization.Bioinformatics (Oxford, England),17(9),853–854.doi:10.1093/bioinformatics/17.9.853

Fry,B.J.(2004).Computational information design[PhDthesis].MassachusettsInstituteofTechnology.

Fu,X.,Hong,S.-H.,Nikolov,N.,Shen,X.,Wu,Y.,&Xuk,K.(2007).Visualizationandanalysisofemailnetworks.Proceedings of the International Asia-Pacific Symposium on Visualization(pp.1–8).IEEE.

Gebhardt,R.,Davies,M.,&Seeber,B.(2016).Psychoacousticapproachesforharmonicmusicmixing,Applied Sciences, 6(5).

Gibney,M.(n.d.)Music-map,thetouristmapofmusic.Retrievedfromhttps://www.music-map.com

Gilbert,F.,Simonetto,P.,Zaidi,F.,Jourdan,F.,&Bourqui,R.(2011).Communitiesandhierarchicalstructuresindynamic socialnetworks:Analysis andvisualization.Social Network Analysis and Mining,1(2),83–95.doi:10.1007/s13278-010-0002-8

Goh,K.-I.,Cusick,M.,Valle,D.,Childs,B.,Vidal,M.,&Barabási,A.-L.(2007).Thehumandiseasenetwork.Proceedings of the National Academy of Sciences of the United States of America, 104(21), 8685–8690.doi:10.1073/pnas.0701361104

Griffin,G.,Kim,Y.,&Turnbull,D.(2010).Beat-sync-mash-coder:Awebapplicationforreal-timecreationofbeat-synchronousmusicmashups.Proceedings of the International Conference on Acoustics, Speech, and Signal Processing(pp.437–440).AcademicPress.doi:10.1109/ICASSP.2010.5495743

http://dx.doi.org/10.1080/09298215.2016.1182192

http://dx.doi.org/10.1109/ICASSP.2017.7952169

http://dx.doi.org/10.1093/acprof:oso/9780199811809.001.0001

http://dx.doi.org/10.1109/TASLP.2014.2347135

http://dx.doi.org/10.1109/TASLP.2014.2347135

http://dx.doi.org/10.3390/app9081695

http://dx.doi.org/10.1093/bioinformatics/17.9.853

https://www.music-map.com

http://dx.doi.org/10.1007/s13278-010-0002-8

http://dx.doi.org/10.1073/pnas.0701361104

http://dx.doi.org/10.1109/ICASSP.2010.5495743


38

Grill, T., & Flexer, A. (2012). Visualization of perceptual qualities in textural sounds. Proceedings of the International Computer Music Conference.AcademicPress.

Hamasaki,M.,&Goto,M.(2013).Songrium:Amusicbrowsingassistanceservicebasedonvisualizationofmassiveopencollaborationwithinmusiccontentcreationcommunity.Proceedings of the International Symposium on Open Collaboration.ACM.doi:10.1145/2491055.2491059

Heer,J.,&Boyd,D.(2005).Vizster:Visualizingonlinesocialnetworks.Proceedings of the IEEE Symposium on Information Visualization(pp.32–39).IEEE.

Henry,N.,Fekete,J.-D.,&McGuffin,M.(2007).Nodetrix:Ahybridvisualizationofsocialnetworks.IEEE Transactions on Visualization and Computer Graphics,13(6),1302–1309.doi:10.1109/TVCG.2007.70582

Herman,I.,Melançon,G.,&Marshall,M.(2000).Graphvisualizationandnavigationininformationvisualization:Asurvey.IEEE Transactions on Visualization and Computer Graphics,6(1),24–43.doi:10.1109/2945.841119

Hevner,A.R. (2007).Athreecycleviewofdesignscienceresearch.Scandinavian Journal of Information Systems,19(2),4.

Jacomy,M.,Venturini,T.,Heymann,S.,&Bastian,M.(2014).ForceAtlas2,acontinuousgraphlayoutalgorithmforhandynetworkvisualizationdesignedfortheGephisoftware.PLoS One,9(6),e98679.doi:10.1371/journal.pone.0098679

Kimelman,D.,Leban,B.,Roth,T.,&Zernik,D.(1994).Reductionofvisualcomplexityindynamicgraphs.Proceedings of theInternational Symposium on Graph Drawing(pp.218–225).Springer.

Lam,H.,Bertini,E.,Isenberg,P.,Plaisant,C.,&Carpendale,S.(2011).Seven guiding scenarios for information visualization evaluation.UniversityofCalgary.

Lee,C.,Lin,Y.,Yao,Z.,Li,F.,&Wu,J.(2015).Automaticmashupcreationbyconsideringbothverticalandhorizontalmashabilities.Proceedings of the International Society for Music Information Retrieval Conference,Malaga,Spain(pp.399–405).AcademicPress.

Maçãs,C.,Rodrigues,A.,Bernardes,G.,&Machado,P.(2018).MixMash:Avisualisationsystemformusicalmashupcreation.Proceedings of the 2018 22nd International Conference Information Visualisation (IV)(pp.471–477).IEEE.doi:10.1109/iV.2018.00088

McDonnell,M.(2007).Visualmusic.InVisual Music Marathon.BostonCyberartsFestivalProgramme.

Meirelles,I.(2013).Design for information.Rockportpublishers.

MixedinKey.(n.d.).Mashup2[software].Retrievedfromhttp://mashup.mixedinkey.com

Mudge,E.(1920).Thecommonsynaesthesiaofmusic.The Journal of Applied Psychology,4(4),342–345.doi:10.1037/h0072596

Muelder,C.,Provan,T.,&Ma,K.-L.(2010).Contentbasedgraphvisualizationofaudiodataformusiclibrarynavigation. Proceedings of the IEEE International Symposium on Multimedia (pp. 129–136). IEEE Press.doi:10.1109/ISM.2010.27

Native Instruments. (n.d.)TraktorPro2 [software].Retrieved fromhttps://www.native-instruments.com/en/products/traktor/dj-software/traktor-pro-2/

Navas,E.(2014).Remix theory: The aesthetics of sampling.Birkhäuser.

Pachet,F.,&Aucouturier,J.-J.(2004).Improvingtimbresimilarity:Howhighisthesky.Journal of Negative Results in Speech and Audio Sciences,1(1),1–13.

Ramachandran,V.S.,&Hubbard,E.M.(2003).Hearingcolors,tastingshapes.Scientific American,288(5),52–59.doi:10.1038/scientificamerican0503-52

Rauber,A.,Pampalk,E.,&Merkl,D.(2003).TheSOM-enhancedjukebox:Organizationandvisualizationofmusiccollectionsbasedonperceptualmodels.Journal of New Music Research,32(2),193–210.doi:10.1076/jnmr.32.2.193.16745

http://dx.doi.org/10.1145/2491055.2491059

http://dx.doi.org/10.1109/TVCG.2007.70582

http://dx.doi.org/10.1109/2945.841119

http://dx.doi.org/10.1371/journal.pone.0098679

http://dx.doi.org/10.1371/journal.pone.0098679

http://dx.doi.org/10.1109/iV.2018.00088

http://mashup.mixedinkey.com

http://dx.doi.org/10.1037/h0072596

http://dx.doi.org/10.1109/ISM.2010.27

https://www.native-instruments.com/en/products/traktor/dj-software/traktor-pro-2/

https://www.native-instruments.com/en/products/traktor/dj-software/traktor-pro-2/

http://dx.doi.org/10.1038/scientificamerican0503-52

http://dx.doi.org/10.1076/jnmr.32.2.193.16745

http://dx.doi.org/10.1076/jnmr.32.2.193.16745


39

Rodrigues,A.,Cardoso,A.,&Machado,P.(2016).Harmonicconstellation:Anaudiovisualenvironmentoflivingorganisms.Proceedings of theFourth International Workshop on Musical Metacreation.AcademicPress.

Rokach,L.,&Maimon,O.(2005).ClusteringMethods.InO.Maimon&L.Rokach(Eds.),Data mining and knowledge discovery handbook.Boston:Springer.doi:10.1007/0-387-25465-X_15

Sapp,C.(2001).Harmonicvisualizationsoftonalmusic.Proceedings of the International Computer Music Conference(pp.419–422).AcademicPress.

Schedl,M.,Gómez,E.,&Urbano,J.(2014).Musicinformationretrieval:Recentdevelopmentsandapplications.Foundations and Trends in Information Retrieval,8(2–3),127–261.doi:10.1561/1500000042

Schwarz,D.,&Schnell,N.(2009).Soundsearchbycontent-basednavigationinlargedatabases.Proceedings of the Sound and Music Computing Conference.AcademicPress.

Shaath,I.(2011).Estimation of key in digital music recordings[Master’sthesis].BirkbeckCollege,UniversityofLondon.

Shiga,J.(2007).Copy-and-persist:Thelogicofmash-upculture.Critical Studies in Media Communication,24(2),93–114.doi:10.1080/07393180701262685

Snydal,J.,&Hearst,M.(2005).Improviz:Visualexplorationsof jazzimprovisations.InCHI’05ExtendedAbstractsonHumanFactorsinComputingSystems(pp.1805–1808).ACM.

Spence,C.(2011).Crossmodalcorrespondences:Atutorial review.Attention, Perception & Psychophysics,73(4),971–995.doi:10.3758/s13414-010-0073-7

VanGulik,R.,Vignoli,F.,&VandeWetering,H.(2004).Mappingmusicinthepalmofyourhand,exploreanddiscoveryourcollection.InProceedings of the International Conference on Music Information Retrieval.Retrievedfromhttp://www.ee.columbia.edu/~dpwe/ismir2004/CRFILES/paper153.pdf

Vavrille,F.(n.d.).Liveplasma.Retrievedfromhttp://www.liveplasma.com/artist-Hard-Fi.html

Vesna,V.(2007).Database aesthetics: Art in the age of information overflow.UniversityofMinnesotaPress.doi:10.5749/j.cttts7q7

Ward,J.,Huckstep,B.,&Tsakanikos,E.(2006).Sound-coloursynaesthesia:Towhatextentdoesitusecrossmodalmechanismscommontousall?Cortex,42(2),264–280.doi:10.1016/S0010-9452(08)70352-6

Wattenberg,M.(2002).Arcdiagrams:Visualizingstructureinstrings.Proceedings of the IEEE Symposium on Information Visualization(pp.110–116).IEEE.

Wilkins, B. (2003). MELD: A pattern supported methodology for visualisation design [Doctoral thesis].UniversityofBirmingham.

eNDNoTe

1 Intheaudiodomain,12-elementchromavectorsreporttheenergyofthetwelvepitchclasses,i.e.,allchromaticnotesoftheequal-temperedscale,bywrappingthespectralenergycontentofanaudiosignalintoasingleoctave.

http://dx.doi.org/10.1007/0-387-25465-X_15

http://dx.doi.org/10.1561/1500000042

http://dx.doi.org/10.1080/07393180701262685

http://dx.doi.org/10.3758/s13414-010-0073-7

http://www.ee.columbia.edu/~dpwe/ismir2004/CRFILES/paper153.pdf

http://www.liveplasma.com/artist-Hard-Fi.html

http://dx.doi.org/10.5749/j.cttts7q7

http://dx.doi.org/10.1016/S0010-9452(08)70352-6


40

Catarina Maçãs has an Undergraduate and Master degree in Design and Multimedia from University of Coimbra. Currently is a computational designer and researcher at the Computational Design and Visualization Lab, is enrolled in the Doctoral Program of Information Science and Technology of the University of Coimbra, and is also a teaching assistant at the Department of Informatics Engineering of the University of Coimbra. In the past two years, started to work on Information Visualization and had already published in conferences such as, IJCAI and IVAPP.

Ana Rodrigues is a Computational Designer and Ph.D. Student. Aside from being currently enrolled in the Program of Information Science and Technology at the University of Coimbra, working on her thesis entitled “Cross-Modal Associations for Design Languages”, she is also doing research at the Computational Design and Visualization Lab. At the same institution she has previously completed a B.Sc. degree and a M.Sc. degree in Design and Multimedia. Currently, she also performs the role of invited assistant lecturer. She is particularly interested in exploring bridges that may emerge from crossing domains of visual communication, music, neurocognitive science, and computational creativity.

Gilberto Bernardes has a multifaceted activity as a saxophonist and researcher in sound and music computing. He holds a PhD in digital media from the University of Porto under the auspices of the University of Texas at Austin; a Master of Music, cum laude, from the Amsterdamse Hogeschool voor de Kunsten and a Degree in music performance from the Superior School of Music and Performing Arts of the Polytechnic Institute of Porto and from the ENM d’Issy-les-Moulineaux (France). Bearnardes pursues a research agenda focused on sampling-based synthesis techniques and tonal pitch spaces, whose findings have been shared to the scientific and artistic communities in over 30 scientific publications. His artistic activity counts with regular concerts in venues and festivals with recognized merit, such as Asia Culture Center (Korea); New York University (USA); Concertgebouw and Bimhuis (Holland); Teatro Monumental de Madrid (Spain) and Casa da Música (Portugal). He is a member of the Portuguese Symphonic Band and Oficina Musical. Bernardes is currently an Assistant Professor at the University of Porto, Faculty of Engineering and an fellow researcher at INESC TEC.

Penousal Machado is Associate Professor at the Department of Informatics of the University of Coimbra in Portugal. He is a deputy director of the Centre for Informatics and Systems of the University of Coimbra (CISUC), the coordinator of the Cognitive and Media Systems group and the scientific director of the Computational Design and Visualization Lab. of CISUC. His research interests include Evolutionary Computation, Computational Creativity, Artificial Intelligence and Information Visualization. He is the author of more than 200 refereed journal and conference papers in these areas, and his peer-reviewed publications have been nominated and awarded multiple times as best paper. As of February 28, 2018, his publications gathered over 1828 citations, an h-index of 21, and an i10-index of 40. He is also the chair of several scientific events, including, amongst the most recent, PPSN XV, EvoStar 2016, and EuroGP 2015; member of the Programme Committee and Editorial Board of some of the main conferences and journals in these fields; member of the Steering Committee of EuroGP, EvoMUSART and Evostar; and executive board member of SPECIES. He is the recipient of several scientific awards, including the prestigious EvoStar Award for outstanding Contribution to Evolutionary Computation in Europe, and the award for Excellence and Merit in Artificial Intelligence granted by the Portuguese Association for Artificial Intelligence.Penousal Machado has been invited to perform keynote speeches in a wide set of domains, from evolutionary computation to visualization and art. His work was featured in the Leonardo journal, Wired magazine and presented in venues such as the National Museum of Contemporary Art (Portugal) and the “Talk to me” exhibition of the Museum of Modern Art, NY (MoMA). He has been the Principal Investigator and Coordinator of several national and international projects, both at an academic and industry level.

mixmash - cdv lab

Documents