a dynamically reconfigurable data stream processing system

A Dynamically Reconfigurable Data Stream

Processing System

Jerzy M. NogiecFermilab

A Dynamically Reconfigurable Data Stream Processing SystemJ.M. Nogiec, Fermilab 2

OutlineOutline

Dynamism in Software Systems …..…

EMS Framework ……………………….

Reconfiguration in EMS ……………….

Dynamism using Scripting ………..…..

Wrap-Up ……………..………….………


Dynamism in Software Systems

Dynamism in Software Systems

Open dynamic systemChanges at runtime that were not anticipated at build time.

Closed dynamic systemConfiguration changes performed at runtime, but planned at build time.

Tailorable systemSome aspects of the system can be modified at runtime.

Configurable system Many different configurations can be set up.

Static system System behavior fully described at build time.

dynamism


Dynamic Reconfiguration

Dynamic Reconfiguration

Dynamic reconfiguration is a process of modifying/evolving a running application, in which a composition of interacting components changes at runtime. It could be used to:

Modify behavior of the system why preserving continuity of its execution (mission or safety critical systems)

Apply patches to a running system Extend base application functionality with additional nonfunctional services (end

user customization and extension) Adapt to changing requirements or environment that are difficult to anticipate

(dynamically optimizing system functionality or recovering from errors and failures).


EMS FrameworkEMS Framework

EMS is a component-based framework for building measurement, data acquisition, and data stream processing systems. The goal of the EMS project was to design, implement, and deploy a system that is extensible, flexible and dynamic. EMS can be used to develop configurable and dynamically reconfigurable systems.

*Framework – the skeleton of an application that can be customized and reused**Component – an independently released software module suitable for composition (together with other components) into multiple applications


Visual ComponentsVisual Components


Component InterfacesComponent Interfaces

• properties• state• error status

data data

control

exception debug

Components have properties and state. Typical components input, process, and output data. Their behavior depends on their state and property values. Components can be forced to perform certain actions by sending control events to them. Components also output debug and exception information.


Component ConnectivityComponent Connectivity

splitter

merger

filter

There are several categories of data processing components:

Data sinks Data sources Data filters/processors Data splitters Data mergers

source

sink


Connecting ComponentsConnecting

Components EMS provides for

independent “wiring” of components for property, data, exception, control, and debug events.

EMS supports unicast, multicast, and broadcast patterns.

Source routing (dynamic, run-time) & routing tables (static, configuration-time).

<configuration version="0.1" title="Display Test XML">

 <component id="Producer” class="ems.core.components.SimpleDataGenerator"> <property name="title" value="Data Generator Component"/> </component> <component id="Chart" class="ems.measurement.chart.ChartDataDisplay"> <property name="XPosition" value="0"/> <property name="YPosition" value="200"/> <property name="title" value="Plot Display"/> </component>

 <route type="Data" origin="Producer" destination="Chart" />

 <control signal="init" destination="!"/> <control signal="start" destination="!"/>

</configuration>

EMS XML DialectEMS XML Dialect

Producer Chart


Reconfiguration in EMS

Reconfiguration in EMS

There is several techniques that can be employed to alter the behavior of the system:

Use of component properties that are modifiable also at runtimeDefinition and modification of event pathsDeactivation/activation of components Deactivation/activation of data paths using gate components“Coloring” data in data driven processing Addition/removal of componentsUse of externally controllable source routing


Reconfiguration in EMS: Modification of Component Properties

Reconfiguration in EMS: Modification of Component Properties

Each component has a set of properties that both control its behavior and exhibit its state. These properties can be introspected and modified externally at runtime.


Reconfiguration in EMS: Modification of

Data Paths

Reconfiguration in EMS: Modification of

Data Paths Data paths between

components can be manipulated (added, modified or removed) at runtime.


Reconfiguration in EMS: Deactivation/Activation of

Components

Reconfiguration in EMS: Deactivation/Activation of

Components A component can be made

inactive, so it passes the data but does not perform any processing.


Reconfiguration in EMS: Gate Components

Reconfiguration in EMS: Gate Components

A gate component passes or stops all data traffic going through it. Therefore it effectively activates or deactivates the data path that originates at it.


Reconfiguration in EMS: Coloring Data

Reconfiguration in EMS: Coloring Data

A “coloring” component (Marker) is used to mark data. Other components act only on the data marked (“colored”) to be processed by them.


Reconfiguration in EMS: Adding/Removing

Components

Reconfiguration in EMS: Adding/Removing

Components A component can be added to

or removed from the system.


Reconfiguration in EMS: Source Routing

Reconfiguration in EMS: Source Routing

A component using source routing can send data to a recipient defined in one of its properties that can be manipulated by other

(controlling) components.


Reconfiguration Components

Reconfiguration Components

Separation of concerns between functional code and reconfiguration code can be achieved by introducing a dedicated reconfiguration component (-s).


Reconfiguration with Scripting

Reconfiguration with Scripting

Scripting allows for automation of processing and for quick construction of new tests or data processing logic.

Scripting allows for easy implementation of reconfiguration in closed dynamic systems.The script control panel controlling execution of a

Jython script.


WrapUpWrapUpEMS is a flexible component-based framework for building

dynamic systems.The system has been applied to building measurement and data

acquisition systems where continuous data streams are processed in real-time.

EMS allows for high level of reuse.The architecture supports separation of concerns between

reconfiguration and functional aspects.EMS features allow for agile development of new applications and

easy adaptation of existing applications to changing requirements.


ResourcesResources EMS web site http://sdsg.fnal.gov/emsweb

EMS on-line documentationhttp://wwwtsmtf.fnal.gov/ems_javadoc2.X

a dynamically reconfigurable data stream processing system

Documents

components ems

ems framework ems

output data

data acquisition

data stream processing

ems project

static system system

system functionality