on temporal logic and signal processing

Ezio Bartocci Vienna University of Technology

On Temporal Logic and Signal Processing

in collaboration with:

Alexandre Donzé, Oded Maler,

Dejan Nickovic, Radu Grosu, Scott A. Smolka

UC Berkley, Verimag, IST Austria, TU Wien, SUNY Stony Brook

Outline

•  Motivation •  Temporal Logic for Signals •  Temporal Frequency Logic •  Case Study •  Future Works

Motivation

Signals come from everywhere !! From the Earth

Seismometer

From the Heart

ECG

From the Climate Changes

From Circuits From Music

Amplifier

From the Economy

El Niño/La Niña-Southern Oscillation Stock Market

Music Sheet

Signals from the Heart

P

Q

R

S

T

J

P-Q interval QRS interval

Q-T interval

Q

R

S

T

P

J

Normal ECG

Brugada ECG

The signal pattern can change in time

Normal Heart Rhythm Ventricular Tachycardia

ECG

Ventricular Fibrillation

Simulation

Experiment


Grosu, et al., Bartocci, Learning and detecting emergent behavior in networks of cardiac myocytes, Comnunications of the ACM, Vol. 52, (3)

The signal pattern can change in space


ECG


Simulation

Experiment



ECG


Simulation

Experiment

The signal pattern can change in frequency

Heart Rate Dynamics in Health and Disease

Disease

Disease

Healthy

Disease http://physionet.org/tour/


The signal pattern can change in time, space and frequency


ECG


Simulation

Experiment

Research Challenges (1)

1.  Can we formally specify the emergent behavior of a disease in a logical language ? What are the ingredients ? (Specification)

•  Time •  Space •  Frequency

(A priori of our knowledge for Kant, Critique of Pure Reason)

Holter Recording

Example:

“Eventually an atrial fibrillation that last for at least 10 seconds will occur.”

(The cells are oscillators, but social and physical phenomena are in general coupled periodic or quasi-periodic oscillators)

Research Challenges (2)

2.  Can we learn a logic formula that satisfies the emergent behavior of a disease and does not satisfy an healthy behavior (Robustness) ?

3.  Can we use the logic formula to explore the regions of the parameter space that will make the model satisfying (or not) the formula (Diagnosis) ?

(A) Wave forms on a 1000x1000 network of cardiac cells for varying parameter values.

(B) The associated tip movement and regions of fibrillation.

Research Challenges (3) 4.  Can we control the input of a plant model, using the logic formula

as our reference input ? (Control)

5.  Can we minimally modify the model, such that the logic formula may be satisfied ? (Model Repair)

6.  Can we monitor real-time signals using the logic formula as a signal pattern specification ? (Real-time Verification)

Temporal Logic for Signals

Families of Temporal Logics Linear Temporal Logic (LTL)

A. Pnueli, 1977

Semantics:

Syntax:

Syntactic sugar for Eventually and Globally:

Discrete-time semantics

Kripke Structure


A. Pnueli, 1977

Time Propositional Temporal Logic (TPTL)

R. Alur, T. A. Henzinger, 1994

Metric Interval Temporal Logic (MITL)

R. Alur, T. Feder, T. A. Henzinger, 1996

They extend LTL with continuous-time semantics


A. Pnueli, 1977





Syntax:

Syntactic sugar for Eventually and Globally:

Semantics: Minkowski sum and difference:

Boolean Signals:


A. Pnueli, 1977





Signal Temporal Logic (STL)

O. Maler, D. , Nickovic, 2004

They extend MITL with Real-Valued Signals

Signal Temporal Logic

Property to verify:


A. Pnueli, 1977





Signal Temporal Logic (STL)

O. Maler, D. , Nickovic, 2004

We extend the Signal Temporal Logic to deal with time-dependent frequency

amplitudes

Temporal Frequency Logic (STL) Our attempt, 2012

Temporal Frequency Logic

Frequency Analysis in a Nutshell Fourier Transform

Fourier series:

T0 =1ω 0

x[t]= a02+ ak cos 2πω kt( )

k=0

+∞

∑ + bk sin 2πω kt( ) with ω k = kω 0

Euler's formula: eix = cos(x)+ isin(x)

x[t]= cke2iπkω0t

k=−∞

+∞

∑Continuos Fourier Transform:

cω = x̂ ω( ) = x[t]e−2iπωt dt−∞

+∞

∫Inverse Continuos Fourier Transform:

x[t]= cωe2iπωt dt

−∞

+∞

∫

Combining Time and Frequency The Short-Time Fourier Transform

0 1 2 3 4 5⌥1

0

1

0 1 2 3 4 50

5

10

15

20

25

30

35

40

45

50

0 1 2 3 4 50

0.1

0.2

0 0.5 1 1.5

Continuous-time STFT:

STFT {x(t)}(τ ,ω ) ≡ X(τ ,ω ) = x t( )−∞

∞

∫ ω (t −τ )e− jωtdt

Discrete-time STFT:

STFT {x n[ ]}(m,ω ) ≡ X(τ ,ω ) = x n[ ]ω[n −m]e− jωnn=−∞

∞

∑Fixed Resoultion:

It depends on the window size

New predicate for the logic:

by one of the authors, was indeed a Blues melody. To do this, we built aformula based on the fact that standard blues is characterized by a 12-barstructure (a bar being basically four beats). In the key of E, it is as follows:E E E E | A A E E | B A E E. Note that a bar in E does not mean that weplay four beats of E notes in a row. There can be di↵erent notes, but the overallbar should sound like a melody in the key of E (music is a mix of strict and verysubjective rules that is likely to both fascinate and exasperate the most rigorouscomputer scientists). If we assume though that in a bar of E there should be atleast one E note played, and similarly for A and B, it is easy to write a formulathat directly translates the above structure. For example, if b is the duration ofone bar:

µ

E

^⌃[b,2b]µE

^ ⌃[2b,3b]µE

^ ⌃[3b,4b]µE

^⌃[4b,5b]µA

^ ⌃[5b,6b]µA

^ ⌃[6b,7b]µE

^ ⌃[7b,8b]µE

^⌃[8b,9b]µB

^ ⌃[9b,10b]µA

^ ⌃[10b,11b]µE

^ ⌃[11b,12b]µE

)

This formula is still, however, too strict in transcribing the above blues pattern,for the simple blues line that we recorded. Indeed, our melody does not havean E in the fourth bar (but has other notes that sound cool with an E line).Instead, we verified a simpler (but definitely bluesy) formula which looks for astarting E, an A in bars 5-6, and the so-called “turn-around” (the sequence BA E) in bars 9-11:

'blues = µ

E

^ ⌃[5b,6b]µA

^ ⌃[8b,9b](µB

^ ⌃[b,2b]µA

^ ⌃[2b,3b]µE

) (10)

Our results are presented in Figure 5.

Implementation We implemented function pitch in Matlab and defined STLformulae using the Breach tool [5]. All examples in this paper are also avail-able at http://www-verimag.imag.fr/

~

donze/breach_music_example.html

Breach implements the full STL syntax (Boolean and temporal operators) ontop of STL predicates of the form

µ = f(x, p) > ✓

where f is some arbitrary signal operator and p is one or more parameters. Thefunction f can be an explicit arithmetic expression such as 2 ⇤ x[t] + p or aroutine implemented separately and available in the Matlab environment, suchas the pitch

!

routine or any other implementation of spectral operators f

L,!

.This makes the implementation of FTL straightforward in the Breach framework.Moreover, Breach allow fully parameterized formulae, i.e., with parameters notonly in the predicates (such as p and ✓) but also in the timing constraints, e.g.:⌃[p0,p1]µ. This can be very useful in the context of music ,where the same piececan be played at di↵erent tempos: the same formula can be used to recognize apattern independently from the execution speed.

Combining Time and Frequency The Wavelet Transform

Fourier Transform Short Time Fourier Transform Wavelet Transform

Mexican Hat Morlet Haar

Case Study: Music

Case Study: Music Chromatic Scale:

C4 C4# D4 D4# E4 F4 F4# G4 G4# A4 A# B4 C5

Octave

261.63 Hz 523.25 Hz

ωD4 = 2112ωC 4 #ωC 4 = 2

− 112ωC 4 #

ωC5 = 2ωC 4

Whole note

Half note

Quarter note

eighth note

Note Rest Value

Tempo measured in bpm = beats for minute

Largo 40-60 bpm Adagio 66-76 bpm Moderato 108-120 bpm Allegro 120-168 bpm Presto 168-200 bpm

60 bpm means that: 4 sec

2 sec

1 sec

0.5 sec

Note Rest Value

Our logic, time-frequency logic (TFL) is obtained by adding the operators{f

L,!

} to STL. A spectral signal y = f

L,!

(x), like any other signal, can partic-ipate in any TFL formula as an argument to predicates and arithmetic expres-sions. The monitoring machinery is similar to that of STL except for the factthat the raw signals x are pre-processed to yield the spectrogram from whichspectral signals y are extracted. This can be done before the monitoring processstarts or be integrated in an online procedure as in [19] where segments of yare computed incrementally upon the arrival of segments of x. In the followingsection we illustrate the expressivity of TFL by defining properties of musicalmelodies and monitoring acoustic waveforms against these properties.

5 Music

As observed in [10], human acoustic perception is a prime example of analyzingsignals based on a combination of time and frequency features. Music, the mostmathematical acoustic expression, employs a system of notations and conven-tions expressing both aspects. In this section, we illustrate the applicability ofTFL in formalizing and recognizing melodies starting with the basic task of notedetection.

Note Detection Each note is basically a sinusoid oscillating at a specific fre-quency !, or pitch, with a finite duration in time. To specify TFL formulaerecognizing patterns in a musical signal, we first need to define a spectral oper-ator that specializes in note detection. We define it as pitch

!

, with the intendedmeaning that pitch

!

(x)[t] is the amplitude of ! in x around time t.To be useful, this operator must satisfy two conflicting constraints: to be

able to tolerate small pitch variations (for instruments which are not perfectlytuned) and, on the other hand, to discriminate a note from its two closest distinctneighboring notes. These notes have frequencies !1 = 2�

112! and !2 = 2

112!.

This makes the closest frequency to be discriminated at a distance d! = !� !1

of !. Thus in our implementation, pitch is defined as the STFT pitch!

(x) =x̂

L

(!), where the size L of the window function is automatically chosen (based onfrequency-temporal resolution constraints of STFT described in the Appendix)to achieve the required resolution. Once pitch

!

is defined we can define the pitch-detection predicates. For instance, a predicate detecting the note A with pitch!

A

= 440Hz will be written as µA

= pitch!A

(x) > ✓. The only parameter whichremains to be fixed is the threshold ✓. This will determine the robustness of thepredicate to variations in volume, pitch or duration.

Note that the Breach tool, used for monitoring all examples in this paper,implements the quantitative semantics of [7]. Breach can thus return a satis-faction signal equal to pitch

!

(x) � ✓ such that the signal’s sign and magnitudedetermine Boolean satisfaction and robustness, resp. Robustness is propagatedto STL formulas using the min-max equivalent of the Boolean and temporaloperators as defined in (2). In Fig. 3, we show the result of applying the pitchfunction to the detection of an F.

Single note detection

!"

!"

!#"

$"

%&'" %&(" %&)" %&*" %&+" %&," %&-" %&." %&/"%"

01234567!8"9"24"

01234567!#8"9"24"

01234567$8"9"24"

:1;<=>"

))%"" )*%"" )+%"" ),%"" )-%""?@":A3&"

%&%%*"

%&%%*+"

%&%%+"

%&%%++"

%&%%,"

%&%%,+"

%&%%-"

%&%%-+"

%&%%'"

%&%%("

%&%%)"

%&%%*"

%&%%+"

%&%%,"

%&%%-"

%&%+" %&'" %&'+" %&(" %&(+" %&)" %&)+" %&*" %&*+"%" :A3&"

%"

'"

!#"$" !"=8"

B8"

38"

Diatonic Scale with rest

!" #" $" %" &" '" (")*+,"

-)./0"

123420"

5" 6" 7" 8" 9" :" ;" +*<="

675">?@"

5"

5"

6"

6"

-)./0"

123420"

5" 6" 7" 8" 9" :" ;" +*<="

5"

5"

-)./0"

123420"

5" 6" 7" 8" 9" :" ;" +*<="

5"

5"

6"

6"

-)./0"

123420"

5" 6" 7" 8" 9" :" ;" +*<="

5"

5"

6"

6"

-)./0"

123420"

5" 6" 7" 8" 9" :" ;" +*<="

5"

5"

6"

6"

Recognizing a piece of blues

0 5 10 15 20 25 30⌃1

⌃0.5

0

0.5

1

0 5 10 15 20 25

0

2

4

x 10⌃3

0 5 10 15 20 25

⌃1

0

1

2

x 10⌃3


µ

E

^⌃[b,2b]µE

^ ⌃[2b,3b]µE

^ ⌃[3b,4b]µE

^⌃[4b,5b]µA

^ ⌃[5b,6b]µA

^ ⌃[6b,7b]µE

^ ⌃[7b,8b]µE

^⌃[8b,9b]µB

^ ⌃[9b,10b]µA

^ ⌃[10b,11b]µE

^ ⌃[11b,12b]µE

)


'blues = µ

E

^ ⌃[5b,6b]µA

^ ⌃[8b,9b](µB

^ ⌃[b,2b]µA

^ ⌃[2b,3b]µE

) (10)



~



µ = f(x, p) > ✓


!


L,!


Blues pattern (12 bar):


µ

E

^⌃[b,2b]µE

^ ⌃[2b,3b]µE

^ ⌃[3b,4b]µE

^⌃[4b,5b]µA

^ ⌃[5b,6b]µA

^ ⌃[6b,7b]µE

^ ⌃[7b,8b]µE

^⌃[8b,9b]µB

^ ⌃[9b,10b]µA

^ ⌃[10b,11b]µE

^ ⌃[11b,12b]µE

)


'blues = µ

E

^ ⌃[5b,6b]µA

^ ⌃[8b,9b](µB

^ ⌃[b,2b]µA

^ ⌃[2b,3b]µE

) (10)



~



µ = f(x, p) > ✓


!


L,!


Turn around

1 bar = 4 beats

•  Test our tool with ECG patterns and circuits signals •  Introduce quantitative operator for robustness •  Add a space operator for detecting frequency-

spatio-temporal patterns •  Sensitivity Analysis guided by a logic formula •  Automatic learning of logic formulae for specifying

observed anomalous signal patterns (disease) •  Synthesis of online monitors

Future Works

on temporal logic and signal processing

Technology

motivation temporal

logic formulaas

logic formulamay

realtime signals

research challenges

asignal pattern specification

plant model

model repair6