deep learning techniques for music generation (3)
TRANSCRIPT
![Page 1: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/1.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Jean-Pierre Briot
Laboratoire d’Informatique de Paris 6 (LIP6)Sorbonne Université – CNRS
Programa de Pós-Graduação em Informática (PPGI)UNIRIO
Deep Learning Techniques for Music Generation (3)
![Page 2: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/2.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Feedforward
![Page 3: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/3.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Deep Learning Techniques for Computer Music
3
• Motivation:– Multiple massive corpora available– Mature architectures and efficient implementations
• Ex: Features extraction– Ex: for Music Information Retrieval– Past: Manual rich representations
– Now: Automatic features extraction– (using ex: Stacked Autoencoders)
• Other models/techniques– Human-specified:
» Grammars, Rules...– Learnt:
» Automata, Markov models, Graphical models, Entropy models…
[Bretan et al., 2016]
![Page 4: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/4.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Deep Learning for Music Generation – Technical Challenges
4
1. Ex Nihilo Generation» vs Accompaniment (Need for Input)
2. Length Variability» vs Fixed Length
3. Content Variability» vs Determinism
4. Control» ex: Tonality conformance, Maximum number of repeated notes…
5. Structure6. Originality
» vs Conformance
7. Incrementality» vs Single-step or Iterative Generation
8. Interactivity» vs (Autistic) Automation
9. Explainability
![Page 5: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/5.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Direct Use – Feedforward – Ex 1
5
• Feedforward Architecture• Prediction (or Classification) Task• Ex1: Predicting a chord associated to a melody segment
– scale/mode -> tonality
– Training on a corpus/dataset <melody, chord>– Production (Prediction)
Input: Melody(Pich of) F
Output: Pitch of a Chord
1st
2nd
3rd
4th
Pitch Pitch
![Page 6: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/6.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Direct Use – Feedforward – Ex 1
6
• Feedforward Architecture
• Classification Task
• Ex1: Predicting a chord associated to a melody segment
– scale/mode -> tonality
– Training on a corpus/dataset <melody, chord>
– Production (Classification)
F
Input: Melody
Output: Chord (Pitch Class)
1st
2nd
3rd
4th
A
G#
A#
B
…
…
Pitch
Pitch Class
![Page 7: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/7.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Code
• Python (3)
• Keras
• Theano
• Music21
7
![Page 8: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/8.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
• Audio– Waveform
– Spectrogram (Fourier Transform)
– Other (ex: MFCC)
• Symbolic– Note– Rest– Note hold– Duration– Chord– Rhythm
– Piano Roll
Representation
8
![Page 9: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/9.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Encoding of Features (ex : Note Pitch)
• Value– Analogic
• One-Hot– Digital
• Embedding– Constructed
9
![Page 10: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/10.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Direct Use – Feedforward – Ex 2: MiniBach
10
• Feedforward Architecture
• Prediction Task
• Ex2: Counterpoint (Chorale) generation
• MiniBach [Hadjeres & Briot, 2017]
• Training on the set of (389) J.S. Bach Chorales (Choral Gesang)
Input: MelodyOutput: 3 Melodies
![Page 11: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/11.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
MiniBach Code & Examples
1
minibach-original-chorale
1
minibach-predicted-chorale
11
![Page 12: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/12.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Output Activation and Cost/Loss Functions
12
Ex multiclass single label: Classification among a set of possible notes for a monophonic melody, with only one single possible note choice (single label)
Ex multiclass multilabel: Classification among a set of possible notes for a single-voice polyphonic melody, therefore with several possible note choices (severallabels)
Ex multi multiclass single label: Multiple classification among a set of possible notes for a set of time slices for a monophonic melody, therefore for each time slice with only one single possible note choice
Application
MonophonyPolyphony
Multivoice
![Page 13: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/13.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Output Activation and Cost/Loss Functions
13
Interpretation
nonenoneargmax or samplingargsort and > threshold & max-notesp argmax or sampling
Other cost functions:
Mean absolute error, Kullback-Leibler (KL) divergence…
![Page 14: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/14.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Output Activation and Cost/Loss Functions
14
![Page 15: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/15.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Output Activation and Cost/Loss Functions
15
![Page 16: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/16.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Output Activation and Cost/Loss Functions
16
![Page 17: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/17.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
(Summary of) Principles of Loss Functions
• Probability theory + Information theory
• Probability : P(x)
• Self-information : I(x) = - log P(x)
• Entropy : H(x) = Ex~P I(x) = Ex~P - log P(x)
• Ex~P f(x) : Expectation : mean value of f(x) when x~P= Sx P(x) f(x)
• Cross entropy : H(P, Q) = - Ex~P log Q(x)
H(h(x), y) = -Si hi(x) log yi
• Kullback-Leibler Divergence : DKL(P || Q) = Ex~p [log (P(x) / Q(x))]
= Ex~p [log P(x) - log Q(x)]
• Maximum likelihood principle• Minimum Negative likelihood
17
C
…
softmax
B
Bb
0.12
0.34
0.07
hi(x) -log(hi(x))
1.08
2.12
2.66
max min
0
0
1
yi
![Page 18: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/18.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Classifier Evaluation Metrics
+ -
+
-
hy
True Positive(Hit)
False Negative
True NegativeFalse Positive(False Alarm)
18
Accuracy = TP + TN / % correctly classified
Precision = TP / P (TP + FP)true / classified positiveHow relevant the results
Recall = TP / P (TP + FN)true / real positiveHow many relevant resultsTable of Confusion
(Confusion Matrix)
![Page 19: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/19.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Skewed Classes
0 100
2
98
hy
True Positive(Hit)
False Negative
True NegativeFalse Positive(False Alarm)
19
Accuracy = TP + TN / = 0.98% correctly classified
Precision = TP / P (TP + FP) = 0true / classified positiveHow relevant the results
Recall = TP / P (TP + FN) = 0true / real positiveHow many relevant resultsTable of Confusion
(Confusion Matrix)Ex: Maligne Cancer Prediction: 2% RateOptimist Hypothese: Always (100%) False
0 2
0 98
![Page 20: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/20.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
MiniBach – Ex. of Result
20
![Page 21: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/21.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
MiniBach vs DeepBach
21
MiniBach
DeepBach
![Page 22: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/22.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
MiniDeepBach – Demo
22https://www.youtube.com/watch?v=QiBM7-5hA6o
![Page 23: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/23.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Reorchestration of Ode of Joyby DeepBach (and other techniques [Flow Machines])
23https://www.youtube.com/watch?v=buXqNqBFd6E
![Page 24: Deep Learning Techniques for Music Generation (3)](https://reader034.vdocuments.site/reader034/viewer/2022051909/62849b1b8e4ba13bd3002ba3/html5/thumbnails/24.jpg)
Deep Learning – Music Generation – 2018Jean-Pierre Briot
Reorchestration of God Save the Queenby DeepBach
24
https://www.youtube.com/watch?time_continue=1&v=x-W0ixD9Cpg