f. lelli, advanced training workshop on instruments and sensors on the grid,trieste, april 25, 2007...

F. Lelli, Advanced Training Workshop on Instruments and Sensors on the Grid,Trieste, April 25, 2007

GridCC: Real-time Instrumentations Grids

A real-time interactive GRID to integrate instruments, computational and information

resources widely spread on a fast WAN

Francesco LelliIstituto Nazionale di Fisica Nucleare

Laboratori Nazionali di Legnaro, Legnaro Italy


Overview• Introduction the IE in the Global scenario

• Bringing Instrument into the Grid: the Instrument Element

• Instrument Instrumentation • Fast Instrument Communication Channel• Standard Grid Interaction• Current Implementation performance analysis• Hands On: a taste


StorageElementsStorage

Elements

ComputingElement

ComputingElement

InstrumentElement

Instrument Element (IE): The Basic Idea

ComputingElement

StorageElement

InstrumentElement

InstrumentElement

Existing Grid Infrastructures

Web ServiceInterface

Virtual Control Room


ExecutionService



(VCR)

All end user access is via

the VCR

Instrument elements

(IE)

The IE is a virtualization of the real physical instrument

Instrument elements

(IE)Instrument elements

(IE)

Of course there may be many IEs

Compute and Storage Elements (with advanced reservation)

StorageElement

(SE)

Compute element

(CE)

Of course Many CEs and SEs

StorageElement

(SE)

Compute element

(CE)StorageElement

(SE)

Compute element

(CE)

CollaborativeServices

(CS)


(VCR)

Users generally not working alone

Direct access to IE

SE (and CE) possible but often not desirable

Information and Monitoring

Services(IMS)

“Fast” all pervasive messaging system

Information System

(IS)Slowly updating information

Security Services

Security is essential to the success of the project

Global ProblemSolver

Watching (via the IMS) for problems anywhere in the system and acting to resolve them.

Execution Services

More complex workflows, including advanced reservation and QoS guarantees , allowed

The GridCC Architecture


IE RequirementsWeb Services

Instrument Element

Any Protocol or physical connection

Sensor Network

Instrument

Instrument

GridGrid

ComputingComputing ElementElement

StorageStorage ElementElement

ComputingComputing ElementElement InstrumentInstrument

ElementElement

W

EF

A

B

C

D

1: Provide a uniform access to the physical device

2: Allow a standard grid access to the instruments

3: Allow the cooperation between different instruments that belong to different VOs


Instrument Element: a Black Box

IEVIG

SInstrument

Instrumentation

Fast communication channel

• The term Instrument Element describes a set of services that provide the neededinterface and implementation that enables the remote control and monitoring of physical instruments.

Grid Interaction

Da

ta M

ove

r

Instruments

Quick Answers to the previous slide: 1) The VIGS provide the a uniform

instrument instrumentation way 2) The fast communication channel

disseminate the acquired information between instruments

3) The Data Mover provide a standard Grid Interface in order to be accessed by others Grids components like the SE and the CE

IE Key Developers: E. Frizziero1, M. Gulmini1,3, F. Lelli1,2 ,G. Maron1, A. Petrucci1, S. Squizzato1, S. Traldi1

1 Istituto Nazionale di Fisica Nucleare, Laboratori Nazionali di Legnaro2 Dipartimento di Informatica, Università Ca’ Foscari di Venezia

3 CERN European Organization for Nuclear Research


Instrument Instrumentation


Device Virtualization Model

Instrument

Parameters

Attributes

Control

Model

XML Based

Language

1. Parameters hold configuration information 2. Attributes hold instrument variables 3. Control Model hold actions 4. XML Based Language to allow the device to describe itself

• Parameters: Maximum Voltage, Minimum voltage• Attributes: measured Voltage• Commands: Perform a measure

Voltmeter


Instrument Instrumentation

getContextsgetInstrumentManagersgetInfo

getIstanceget/Set ParametersgetCommandsexecuteCommandgetStategetStateMachine

IE

VIG

S

lockInstrumentsunlokInstrumentsretrieveLoked

getRemoteExecutionTimegetOneWayCostgetTotalMethodExecutionTime

Instruments

We can divide the Instrumentation in 3 main parts: • The direct access to the Instruments• The advance instrument reservation (interaction with the Agreement Service (AS)) in order to achieve (hard) guarantees• The Possibility to predict the execution time of the instrumentation methods in a concurrent access (soft guarantees)

Instrumentation method Documentation http://gladgw.lnl.infn.it:2002/IEFacade

Crucial non-Functional Requirements: • Instruments could be order of 106

• Only authorized people should access to the instruments of a VO• The instrumentation is not a batch process like a job submission! Interactivity is mandatory

• A Distribute and hierarchic implementation is mandatory • the Security overhead should be negligible


Instrument Element ArchitectureVirtual Instrument Grid Service (VIGS)

ResourceService

Inf & MonService

ProblemSolver

InstrumentManager

Instrument Element

Data Mover

IMSProxy

ControlManager

DataCollector

Real Instruments

Data Flow

Control Flow

State FlowError FlowMonitor Flow

• The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments.

Acc

ess

Con

trol

Man

ager

execute()

getState()

create()

destroy()

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager


Instrument Element Implementations

ResourceService

Inf & MonService

ProblemSolver

InstrumentManager

Instrument Element

Data Mover

Acc

ess

Con

trol

Man

ager

The IE components are typically implemented into a fully equipped Machines (e.g. dual core cpus, large memory, large disks, etc). This is true for RS, IMS and PS. For IM (and DM) there are 2 possibilities, according to the application type:• IM implemented in a fully equipped machine• IM embedded into the instrument that should be controlled

IM

RSIMS

IM

IM

IM

Embedded Web Service


Instrument Manager

IM is composed by 3 main components:- Control Manager:

-Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs, errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability-FSM. A finite state machine is implemented-Resource Proxy. It handles all the outgoing connections with the resources.

-- Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen.

- IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS

IMSProxy

DataCollector

Instrument Manager

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

Instruments

Data Flow

State Flow

Error Flow

Monitor Flow

Control Flow

Customizable Plug-in modules to interface to the instruments


Poviding QoS over Web Sevices

t0 t2t1 t3

t8 t4t7 t6 t5

Serialization

Serialization

Deserialization

Deserialization

Transmission

TransmissionProcessing

Operationexecution

Client side Network Service side

Performing a remote method Invocation in a given amount of time:

• Avg =f(Cpu, Inputsize, Outputsize, Algorithm, Key-Factor, net) • SDev =F(Cpu, Inputsize, Outputsize, Algorithm, Key-Factor, net)

Cpu = machine HD + machine load (client and server side)

Algorithm = method semantic

Net = bandwidth + RTT

Key-Factor = input value that change the method semantic

Inputsize, Outputsize =effective type and dimension

Crucial Times are:

t3-t0 One Way Cost t4-t0 Remote Execution Cost t7-t0 Total Method Execution Cost


Virtualization of Real devices

linked

Web Cam Position

Video Streaming

linked

Max Value

Temperature

ResourceService

Inf & MonService

IE

Data Mover

execute()

getState()

create()

destroy()IM Sensor

Data for Model Calculations

Predictions

Each IM Represent the virtualization of a device

IM Cam

UnlinkedUnlinked

min Value


Virtualization of Real devices (I)

linked Unlinked

Web Cam Position

Video Streaming

linked

Max Value

Temperature

Unlinked

min Value

ResourceService

Inf & MonService

IM CamIE

Data Mover

execute()

getState()

create()

destroy()

IM Sensor


Predictions

Each IM Represent the virtualization of a device

IM Master Controller


Virtualization of Real devices (II)

linked Unlinked

Web Cam Position

Video Streaming

linked

Max Value

Temperature

Unlinked

min Value

R S IMS

IM Cam IE Cam

Data Mover

R S IMS

IMSensor IE Sensor

Data Mover

R S IMS


IE Master

Data Mover


Predictions

Each Instrument is virtualized and a 3° IE use this others IE in order to accomplish a complex functionality


Virtualization of Real devices (III)

linked Unlinked

Web Cam Position

Video Streaming

linked

Max Value

Temperature

Unlinked

min Value

ResourceService

Inf & MonService


IE

Data Mover

execute()

getState()

create()

destroy()Data for Model Calculations

Predictions

Cam Proxy Sensor Proxy


Fast Instrument Communication Channel


Message Oriented Middleware

Brokered Implementation

Broker-less (P2P) Implementation

• Subscribers Subscribe to a given Topic/Queue with a subscribe condition

• Publisher publish message in asynchronous in a given Topic/Queue way with a given message condition

• Publisher and subscribers can be part of the same program or in WAN distributed machines

• Web Service performance are totally inadequate

In the IE Use Case: • Each instrument can be a data

publisher or a data consumer

Haifa Research Lab


Publish/Subscribe Message Rate: one-to-many

•32 Dual Xeon 2.4GHz 1.5GB RAM machines, 1 GB Ethernet switch • At most 1 publisher, subscriber, or broker- (Sun MQ3.6) per machine • No message lost• RMM throughput: 48-80 Mbytes/sec.

Msg rate- msg size 100 Bytes

100

1000

10000

100000

1000000

1 5 10 15 20 25 30

Number of Subscribers

msg

/sec RMM

MQ3.6

Manta


100

1000

10000

100000

1 5 10 15 20 25 30


msg

/sec RMM

MQ3.6

Manta


100

1000

10000

1 5 10 15 20 25 30


msg

/sec RMM

MQ3.6

Manta

Haifa Research Lab

SubscribersTopicPublisher


Standard Grid Interaction


Data Mover

• The task of this element is to get data from the “data collector” of the IM• Data can be accessed via:

– Web service interface for generic data dump (e.g. slow storage, spy stream, etc.)– grid storage element (SE) and available CEs can access to the data via an SRM

Interface– Http server and TCP communication for high performance had-hoc data transfer

• The Data Mover exposes its methods to the IE web service and can be instrumented itself as an instrument.

Instrument Resources

DataMover

DataCollector

IM

IE Web Service Interface: get_data()

SRM interface

Http Server andTCP/IP raw socket

DataCollector

IM

DataCollector

IM


Current IE Implementation a fist taste


Instrument Manager Performances (I)

Instrument Manager Invocations

0

10

20

30

40

50

60

1

HTTP Transport Layer

Invo

cati

on

per

Sec

on

d

Average

min

Max

Variance

Asyncronous msg Rate

0

50

100

150

200

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Number of Client

Mes

sag

es p

er S

eco

nd

Average

min

Max

Variance

Virtual Instrument Grid Service (VIGS)A

cces

s C

ont

rol M

an

ager

execute()

getState()

create()

destroy()

IMSProxy

DataCollector

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

Test 1

Test 2

Test 1: Web Service invocation and status switch of FSM

Test 2: Soap Server receiving XML message format. DOM based parser

Virtual Instrument Grid Service (VIGS)A

cces

s C

ont

rol M

an

ager

Acc

ess

Co

ntro

l Ma

nag

er

execute()

getState()

create()

destroy()

IMSProxy

DataCollector

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control ManagerIMSProxy

DataCollector

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

Test 1

Test 2

Test 1: Web Service invocation and status switch of FSM

Test 2: Soap Server receiving XML message format. DOM based parser


Instrument Manager Performances (II)Command Distribution Time

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

10 50 80 120

Number of Instrument

Ave

rag

e T

ime

(sec

)

1FM

1FM M

3FM

3FM M

3FM 3PC

3FM 3PC M

1

2

3

1 + 2

3

1

3

1

Optimized environment

IM with CMS Instruments


Instrument Manager

IM is composed by 3 main components:- Control Manager:

-Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs, errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability-FSM. A finite state machine is implemented-Resource Proxy. It handles all the outgoing connections with the resources.

-- Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen.

- IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS

IMSProxy

DataCollector

Instrument Manager

InputManager

EventProcessor

FSMEngine

ResourceProxy

Control Manager

Instruments

Data Flow

State Flow

Error Flow

Monitor Flow

Control Flow

Customizable Plug-in modules to interface to the instruments


State machine

OffOnError

GoingOn

GoingOff

Resetting

TurnOn

TurnOff

ResetReset

Reset

SetOff

SetOn


Transition Table Current

State

Input

State

Off

State

GoingOn

State

On

State

GoingOff

State

Resetting

State

Error

TURNON GoingOn - - - - -

SETON - On - - - -

TURNOFF - - GoingOff - - -

SETOFF - - - Off Off -

RESET Resetting Resetting Resetting Resetting Resetting Resetting

SETERROR Error Error Error Error Error Error


ComponentsCallbacks

• Send a command go to B• Check if the transition is valid• Exiting callback from the state A• Callback transition from A to B• Entering callback in the state B• Callback for State changed

StateA

StateB

Transition A to B Action

A to B Failed Action

Entering State Action

enter state

on failure

Exit

Enter

Go to B

exit state Exiting State Action

State changed Action

on success


Instrument Manager classes

List of Function Manager classes :

• InstrumentManager (Mandatory) : Main class of the Function Manager

• StateMachineDefinition (Mandatory) : Definiton of the State Machine

• StateActions (Optional): Entered and Exiting Actions• EventHandler (Optional): Function Manager Event Handler

(CMSError, StateNofication,UserEvent etc…)• TransitionActions (Optional): Function Manager Transiton Callbacks• TransitionFailedActions (Optional): Function Manager

TransitonFailed Callbacks• States (Optional): Function Manager States• Inputs (Optional): Function Manager Inputs


public class HelloFunctionManager extends UserFunctionManager {...

public void init() throws StateMachineDefinitionException,EventHandlerException {

// define our State MachinesetStateMachineDefinition(new StateMachineDefinition());// add an handler for state and error notifications and MyEventaddEventHandler(new EventHandler());

// Add the State Machine parameters// onCounter counts the number of times the turnOnAction is executed.

parameterSet.put("onCounter", "0"); // offCounter counts the number of times the turnOffAction is executed. parameterSet.put("offCounter", "0");

}…}

InstrumentManager(1): init()


public class HelloInstrumentManager extends UserFunctionManager {...public void createAction() throws UserActionException {

// Initializes the list of resources to control.containerFMChildren = new

qualifiedResourceContainer(qualifiedGroup.seekQualifiedResourcesOfType(new FunctionManager()));

// Set calcState equals initial StatecalcState = States.OFF;try {

// Initializes the QualifiedGroup: calls the init on all the resourcesthis.qualifiedGroup.init();

} catch …// define the condition state vectors only here// since the group must have been qualified beforethis.defineConditionState();

}…}

InstrumentManager(2): createAction()


public class HelloFunctionManager extends UserFunctionManager {...public void destroyAction() throws UserActionException { if (!containerFMChildren.isEmpty()) {

// Destroy FM childrenIterator it = containerFMChildren.getQualifiedResourceList().iterator();while (it.hasNext()) {

FunctionManager fmClient = (FunctionManager) it.next();try {

fmClient.destroy();} catch ….

} }}…}

InstrumentManager(3): destroyAction()


StateMachineDefinition(1) : states

public class StateMachineDefinition extends UserStateMachineDefinition {...

// Defines the steady StatesaddState(States.OFF);addState(States.ON);addState(States.ERROR);

// Defines the transitional StatesaddState(States.GOINGON);addState(States.GOINGOFF);addState(States.RESETTING);

// Defines the Initial statesetInitialState(States.OFF);

…}



// Defines the Inputs from steady state to transitional stateaddInput(Inputs.TURNOFF);addInput(Inputs.TURNON);addInput(Inputs.RESET);

// Defines the Inputs from transitional state to steady state.// These Inputs are not allowed from the GUI.// They are instead used inside the FM callbacks.addInput(Inputs.SETERROR);addInput(Inputs.SETOFF);addInput(Inputs.SETON);Inputs.SETERROR.setVisualizable(false);Inputs.SETOFF.setVisualizable(false);Inputs.SETON.setVisualizable(false);

…}

StateMachineDefinition(2) : Inputs



// The TURNON command accepts a Parameter "TurnOffSec" // that gives the number of seconds the FM remains ON. // After it moves to OFF State. ParameterSet onParameters = new ParameterSet(); onParameters.put("TurnOffSec", ""); Inputs.TURNON.setParameters(onParameters);…}

StateMachineDefinition(3) : Parameters

• Command with Parameters



// Defines the State TransitionsaddTransition(Inputs.TURNON, States.OFF, States.GOINGON);addTransition(Inputs.SETON, States.GOINGON, States.ON);addTransition(Inputs.TURNOFF, States.ON, States.GOINGOFF);addTransition(Inputs.SETOFF, States.GOINGOFF, States.OFF);addTransition(Inputs.RESET, State.ANYSTATE, States.RESETTING);addTransition(Inputs.SETOFF, States.RESETTING, States.OFF);addTransition(Inputs.SETERROR, State.ANYSTATE, States.ERROR);

…}

StateMachineDefinition(4): Transitions



// Set the Exiting State callbacks.setExitingStateActions(new StateActions());

// Defines the callbacks executed when a given State is left.addExitingStateAction(States.GOINGOFF, "sleepAction");addExitingStateAction(States.GOINGON, "sleepAction");addExitingStateAction(States.RESETTING, "sleepAction");

// Set the Entered State callbacks.setEnteredStateActions(new StateActions());// Defines the callbacks executed when a given State is reached.addEnteredStateAction(States.RESETTING, "fireUserEventAction");addEnteredStateAction(States.GOINGOFF, "fireUserEventAction");addEnteredStateAction(States.GOINGON, "fireUserEventAction");

…}

StateMachineDefinition(5): Callbacks

• Entered and Exiting Actions


public class EventHandler extends UserStateNotificationHandler{…

public EventHandler() throws EventHandlerException {// Let's register also the UserEvent triggered by Entered ActionssubscribeForEvents(UserEvent.class);// Adds callbacks action associated to a specific Function Manager// State.addAction(States.GOINGOFF, "whileGoingOff");addAction(States.GOINGON, "whileGoingOn");addAction(States.RESETTING, "whileResetting");

}public void whileResetting(Object obj) throws UserActionException {

if (obj instanceof UserEvent) {…

}if (obj instanceof StateNotification) {

computeNewState((StateNotification) obj);return;

}}

…}

EventHandler (1): UserStateNotificationHandler


public class StateActions extends UserActions {...public void init() {

helloFunctionManager = HelloFunctionManager)getUserFunctionManager();

}public void fireUserEventAction() throws UserActionException {

System.out.println("Executing fireUserEventAction");logger.info("Executing fireUserEventAction");

helloFunctionManager.fireEvent(new UserEvent());

logger.info("fireUserEventAction executed");System.out.println("fireUserEventAction executed");

}…}

StateActions

• Entered and Exiting Actions Callbacks


public class EventHandler extends UserErrorHandler{…

public EventHandler() throws EventHandlerException {// Adds callbacks action associated to a specific Function Manager// State.addAction(States.OFF, "setError");addAction(States.ON, "setError");

}public void setError(Object obj) throws UserActionException {

if (obj instanceof CMSError) {// Catched the CMSError CMSError myError = (CMSError) obj;// Print out and log the CMSErrorthis.errorAction(myError);return;

}}

…}

EventHandler (2): UserErrorHandler


• State Transitions with transition callback:

public class StateMachineDef extends UserStateMachineDefinition { ... // OFF to ON transition addTransition(

Inputs.TURNON,States.OFF, // From StateStates.ON, // To State“onAction” // callback method

}

public class TransitionActions extends UserActions { ... public void onAction() throws UserActionException {

logger.info( "Executing onAction" );getQualifiedGroup(). …

}}

• transition callback

TransitionActions


• State Transitions:public class StateMachineDef extends UserStateMachineDefintion { ... // OFF to ON transition addTransition(

Inputs.TURNON,States.OFF, // From StateStates.ON, // To State“onAction”,“onFailed”

}

public class TransitionActions extends UserActions { ... public void onFailed() throws UserActionException {

logger.info( "Executing onFailedAction" ); }}

• Transition Failed Callback

TransitionFailedActions


Question?

• Thx for your time

Acknowledgement: The GridCC project is supported under EU FP6 contract 511382.

More information: www.gridcc.orgOn-line Demo at: http://gladgw.lnl.infn.it:2002/IEFacade


Algorithm and Key-Factor Example

• Remote method Y=F(X)

where Y,X are double

and F=

y= -1 if x<0

y=sqr(x) if x>0

The complexity (i.e. the algorithm that need to be remotely executed) depend on the key factor X

f. lelli, advanced training workshop on instruments and sensors on the grid,trieste, april 25, 2007...

Documents

term instrument element

vcr instrument elements

advanced training workshop

different instruments

standard grid access

realtime interactive

execution services

set of services