Dynamic Time Warping (DTW)

J.-S Roger Jang (張智星 )

jang@mirlab.org

http://mirlab.org/jang

MIR Lab, CSIE Dept

National Taiwan University

-2-

Dynamic Time Warping

Goal To align two sequences under certain constraints, such that the distance between these two sequences is as small as possible.

Method Dynamic programming

-3-

Distance between Same-length Sequences

Distance between

Alignment

1 2 1 2, , and , ,n nx x x x y y y y

1 2 3 4 5 6 7 8 9, , , , , , , ,x x x x x x x x x x

1 2 3 4 5 6 7 8 9, , , , , , , ,y y y y y y y y y y

1 1

2

11

, 1

( , ), 2

max ,

n

i ii

n pp np i i

i iii

i ii

x y p

L norm x y x yx y p

x y p

-4-

Distance between Different-length Sequences

1 2 1 2

1 1 2 2

, , 1

, , , , ,

We want to find an alignment path , , , , ,

(subject to the alignment constraints), such that the distance along the

path is minimized:

( , ) mini i

i i

m n

k k

p qp q i k

x x x x y y y y

p q p q p q

dist x y x y

2

1

1 1

,

with the bounary conditions:

Anchored beginning: , 1,1

Anchored end: , ,

k

i

k k

p q

p q m n

-5-

Alignment Constraints: Type 1

Temporal constraints

Local path constraints One-to-one mapping No consecutive skip-over

1 2 1 2,k kp p p q q q

x1 x2 x3

y1

x4 x5

y2 y3 y4 y5 y6 y7 y8

Alignment path =

1,1 , 2,3 , 3,5 , 4,6 , 5,8

-6-

Type-1 Alignment Path

Alignment path (mapping path)

x1 x2 x3

y1

x4 x5

y2 y3 y4 y5 y6 y7 y8

Alignment path =

1,1 , 2,3 , 3,5 , 4,6 , 5,8

i

1

),( jiD

2

3

4

5

6

7

8

1 2 3 4 5

j

-7-

Alignment Constraints: Type 2

Temporal constraints

Local path constraints 1-to-1, 1-to-many, or many-to-1 mapping

No skip-over

1 2 1 2,k kp p p q q q

x1 x2 x3

y1

x4 x5

y2 y3 y4 y5 y6 y7 y8

Alignment path =

1,1 , 2,2 , 2,3 , 3,4 , 3,5 , 4,6 , 5,7 , 5,8

-8-

Type-2 Alignment Path

Alignment path

x1 x2 x3

y1

x4 x5

y2 y3 y4 y5 y6 y7 y8

Alignment path =

1,1 , 2,2 , 2,3 , 3,4 , 3,5 , 4,6 , 5,7 , 5,8

i

1

2

3

4

5

6

7

8

1 2 3 4 5

j

-9-

Type-1 DTW: Table Fillup

i

j

x(i-1)

y(j) 1. Optimum-value function ( , ):

The DTW distance between (1: ) and (1: )

2. Recurrent equation for ( , ) :

( 1, 2)

( , ) ( ) ( ) min ( 1, 1) ,

( 2, 1)

with (1,1) (1) (1) .

3. Answ

D i j

x i y j

D i j

D i j

D i j x i y j D i j

D i j

D x y

er ( ( ), ( ))D len x len y

),( jiD

x, y: input vector/matrixLocal paths: 27-45-63 degrees

DTW formulation:

y(j-1)

x(i)

-10-

Type-2 DTW: Table Fillup

1. Optimum-value function ( , ):

The DTW distance between (1: ) and (1: )

2. Recurrent equation for ( , ) :

( , 1)

( , ) ( ) ( ) min ( 1, 1) ,

( 1, )

with (1,1) (1) (1) .

3. Answer

D i j

x i y j

D i j

D i j

D i j x i y j D i j

D i j

D x y

D

( ( ), ( ))len x len y

x, y: input vector/matrixLocal paths: 0-45-90 degrees

DTW formulation:

i

j

x(i-1)

y(j) ),( jiD

y(j-1)

x(i)

-11-

Local Path Constraints

Type 1: 27-45-63 local paths

Type 2: 0-45-90 local paths

jiD ,

jiD ,

( , ) ( ) ( )

( , 1)

min ( 1, 1)

( 1, )

D i j x i y j

D i j

D i j

D i j

( , ) ( ) ( )

( 1, 2)

min ( 1, 1)

( 2, 1)

D i j x i y j

D i j

D i j

D i j

2,1 jiD

1, jiD 1,1 jiD

jiD ,1

1,1 jiD 1,2 jiD

-12-

Path Penalty for Type-1 DTW

Alignment path of type-1 DTW 45-degree paths are likely to be avoided since we

are minimizing the total distance. So we can add penalty for paths deviated from 45-

degree.

( 1, 2)

( , ) ( ) ( ) min ( 1, 1)

( 2, 1)

D i j

D i j x i y j D i j

D i j

0

jiD ,

2,1 jiD

1,1 jiD 1,2 jiD

-13-

Path Penalty for Type-2 DTW

Alignment path of type-1 DTW 45-degree paths are likely to be taken since we are

minimizing the total distance. So we can add penalty for paths of 45-degree.

( , 1)

( , ) ( ) ( ) min ( 1, 1)

( 1, )

D i j

D i j x i y j D i j

D i j

jiD ,

1, jiD 1,1 jiD

jiD ,1

0

0

-14-

Other Minutes about DTW

Typical applications Speech recognition: MFCC matrices as inputs

(where x(i) is the MFCC vector of frame i) Query by singing/humming: Pitch vectors as

inputs (where x(i) is the pitch value of frame i)

Abundant variants for various applications Recurrent formulas Local path constraints

-15-

Applications

Applications of DTW DTW for speech recognition DTW for query by singing/humming