overview
DESCRIPTION
Grid enabled remote instrumentation and sensors with distributed control The GridCC Project Gaetano Maron Istituto Nazionale di Fisica Nucleare Laboratori Nazionali di Legnaro, Legnaro Italy. Overview. Bringing Instrument into the Grid: the Instrument Element The GridCC Project: Introduction - PowerPoint PPT PresentationTRANSCRIPT
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Grid enabled remote instrumentation and
sensors with distributed control
The GridCC Project
Gaetano Maron
Istituto Nazionale di Fisica Nucleare
Laboratori Nazionali di Legnaro, Legnaro Italy
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Overview
• Bringing Instrument into the Grid: the Instrument Element
• The GridCC Project: Introduction
• The GridCC Test-bed: Pilot applications
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
The Grid Technologies to extend the limit of a single computer (center)
User Interface
ComputingElement
ComputingElement
ComputingElement
StorageElement
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Extending the Grid Concepts
Satellite viewsto monitor the volcano
Control and Monitor RoomTo model calculations
and disaster predictions
Terrestrial probes to monitorThe volcano activities
Instrument Element
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
The GridCC Project
Instruments Grid
Computational Grid
Data for Model Calculations
Predictions
Control and Monitor Room
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
General on the GridCC ProjectParticipant name Country
Istituto Nazionale di Fisica Nucleare Italy
Institute Of Accelerating Systems and Applications
Greece
Brunel University UK
Consorzio Interuniversitario per Telecomunicazioni
Italy
Sincrotrone Trieste S.C.P.A Italy
IBM (Haifa Research Lab) Israel
Imperial College of Science, Technology & Medicine
UK
Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale
delle Ricerche
Italy
Universita degli Studi di Udine Italy
Greek Research and Technology Network S.A.
Greece
• It is a 3 years project. Started the 1st September 04
• Funded by EU in the Frame Program 6
• 10 Partners from 3 EU Countries + (Israel)
• About 40 people engagged
• www.gridcc.org
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
StorageElementsStorage
Elements
ComputingElement
ComputingElement
InstrumentElement
Instrument Element: global scenario
ComputingElement
StorageElement
InstrumentElement
InstrumentElement
Existing Grid Infrastructures
Web ServiceInterface
Virtual Control Room
Virtual Control Room
Exec. Service
WfMS
WMS
AgrSUser direct
ActionIndirect Action
SE
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Virtual Control Room
(VCR)
All end user access is via the
VCR
Instrument elements
(IE)
The IE is a virtualization of the real physical instrument
Instrument elements
(IE)Instrument elements
(IE)
Of course there may be many IEs
Compute and Storage Elements (with advanced reservation)
StorageElement
(SE)
Compute element
(CE)
Of course Many CEs and SEs
StorageElement
(SE)
Compute element
(CE)StorageElement
(SE)
Compute element
(CE)
CollaborativeServices
(CS)
Virtual Control Room
(VCR)
Users generally not working alone
Direct access to IE
SE (and CE) possible but often not desirable
Information and Monitoring
Services(IMS)
“Fast” all pervasive messaging system
Information System
(IS)Slowly updating information
Security Services
Security is essential to the success of the project
Global ProblemSolver
Watching (via the IMS) for problems anywhere in the system and acting to resolve them.
Execution Services
More complex workflows, including advanced reservation and QoS guarantees , allowed
The GridCC Architecture
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
IE RequirementsWeb Services
Instrument Element
Any Protocol or physical connection
Sensor Network
Instrument
Instrument
Grid
ComputingComputing ElementElement
StorageStorage ElementElement
ComputingComputing ElementElement InstrumentInstrument
ElementElement
1: Provide a uniform access to the physical device
2: Allow a standard grid access to the instruments
3: Allow the cooperation between different instruments that belong to different VOs
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Element: a Black Box
IEVIG
SCommands
Status
Fast communication channel
• The term Instrument Element describes a set of services that provide the neededinterface and implementation that enables the remote control and monitoring of physical instruments.
Grid Interaction
SR
M/S
E
Instruments
Quick Answers to the previous slide: 1) The VIGS provide a Web Service acccess to
the instrumentation2) The fast communication channel disseminate
the acquired information between instruments3) The Data Mover provide a SRM/SE interface
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Device Virtualization Model
Instrument
Parameters
Attributes
Control
Model
XML Based
Language
1. Parameters hold configuration information 2. Attributes hold instrument variables 3. Control Model hold actions 4. XML Based Language to allow the device to describe itself
• Parameters: Maximum Voltage, Minimum voltage• Attributes: measured Voltage• Commands: Perform a measure
Voltmeter
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Instrumentation
getContextsgetInstrumentManagersgetInfo
getIstanceget/Set ParametersgetCommandsexecuteCommandgetStategetStateMachine
IEV
IGS
lockInstrumentsunlokInstrumentsretrieveLoked
getRemoteExecutionTimegetOneWayCostgetTotalMethodExecutionTime
Instruments
We can divide the Instrumentation in 3 main parts: • The direct access to the Instruments• The advance instrument reservation (interaction with the Agreement Service (AS)) in order to achieve (hard) guarantees• The Possibility to predict the execution time of the instrumentation methods in a concurrent access (soft guarantees)
Instrumentation method Documentation http://sadgw.lnl.infn.it:2002/IEFacade
Crucial non-Functional Requirements: • Instruments could be order of 106
• Only authorized people should access to the instruments of a VO• The instrumentation is not a batch process like a job submission! Interactivity is mandatory
• A Distribute and hierarchic implementation is mandatory • the Security overhead should be negligible
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Element ArchitectureVirtual Instrument Grid Service (VIGS)
ResourceService
Inf & MonService
ProblemSolver
InstrumentManager
Instrument Element
Data Mover
IMSProxy
ControlManager
DataCollector
Real Instruments
Data Flow
Control Flow
State FlowError FlowMonitor Flow
• The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments.
Acc
ess
Con
trol M
anag
er
execute()
getState()
create()
destroy()
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Interacting with Instrument Elements1) Full GridCC Environment
2) Partial GridCC Environment
3) Standalone Environment
IMS
ComputingElementComputingElement
ComputingElement
StorageElements
StorageElements
StorageElementGPS
VCR
Instrument
Element
Instrument
Element
Instrument
Element
Security
Service
Exec
utio
nSe
rvice
IMS
ComputingElement
ComputingElement
ComputingElement
StorageElements
StorageElements
StorageElementGPS
Instrument
Element
Instrument
Element
Instrument
Element
VCR
Security
Service
Exec
utio
nSe
rvice
IE is web service based, any web service compliantclients can reach it. This mode of work is very usefull for small systems and for prototyping and debug large systems
Instrument
Element
Instrument
Element
Instrument
Element
VCR
Secur ity
Ser vic e
. NET
Visu
al Ba
sicVi
sual
C++
WS - I
Perl
C ++
Java
JSP
This mode of operation can be used when the application does not need to access CEs and SEs. It coud for instance exploit the workflow manager of the execution service to do unattended cycles of operations and control the system via VCR
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Element Implementations
ResourceService
Inf & MonService
ProblemSolver
InstrumentManager
Instrument Element
Data Mover
Acc
ess
Con
trol M
anag
er
The IE components are typically implemented into a fully equipped Machines (e.g. dual core cpus, large memory, large disks, etc). This is true for RS, IMS and PS. For IM (and DM) there are 2 possibilities, according to the application type:• IM implemented in a fully equipped machine• IM embedded into the instrument that should be controlled
IM
RSIMS
IM
IM
IM
Embedded Web Service
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Element on a Chip
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Manager
IM is composed by 3 main components:- Control Manager:
- Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs, errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability - FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources.
- Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen.
- IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS
IMSProxy
DataCollector
Instrument Manager
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
Instruments
Data FlowState Flow
Error Flow
Monitor Flow
Control Flow
Customizable Plug-in modules to interface to the instruments
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Resource Service Architecture
• The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). • A resource can be any hardware or software component involved in the IE (instruments, Instrument
Managers, IMS components)• RS stores the configuration data of the resources and download them to resource target when necessary• Resources can be discovered, allocated and queried. • It is the responsibility of the RS to check resource availability and contention with other active partitions
when a resource is allocated for use. • A periodic scan of the registered resources keeps the configuration database up to date.• RS is interfaced to the WMS
DiscoveryManager
SubscribeManager
Partition&LockManager
ConfigurationManager
Available Resources
PartitionDefinitions
ConfigurationDefinitions R
S D
ata
Bas
es
Partition/Configurationretrieve methods
Partition and Locksetting methods
Configurationsetting methods
Discoverymethods
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Information and Monitor System (IMS)
PUBLISHERS(Instruments nodes)SUBSCRIBERS
Errors Log infoMonitorState
• The Information and Monitor Service (IMS) collects messages and monitor data coming from GRID resources and supporting services and stores them in a database. There are several types of messages collected from the sub-systems. The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms.
• The IMS collects and organizes the incoming information in a database and publishes it to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level.
Instrument
Manager
Inst
rum
ents
Instrument
Manager
Inst
rum
ents
Instrument
Manager
Inst
rum
ents
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Problem Solver
IMSProxy
ControlManager
Instrument Manager
Pub/Sub
IMSProxy
ControlManager
Instrument Manager
IMSProxy
ControlManager
Instrument Manager
IMSProxy
ControlManager
Instrument Manager
DBData Mining Tools
Algorithms evaluations :Rule Induction, Tree, Functions, Lazy, Clusters and Associative
State FlowError FlowMonitor Flow
On Line Analisys
Problem Solver
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
iris
glas
s
brea
st c
ance
r
bupa
votin
g-re
cord
s
hous
ing
bala
nce-
scal
e
Bre
ast
Can
cer W
isco
nsin
Pim
a-In
dian
s-D
iabe
tes
tic-ta
c-to
e
Seg
men
t
Seg
men
tatio
n
Sic
k-eu
thyr
oid
Pag
e-B
lock
s
mus
hroo
m
Shu
ttle(
2)
Lette
rRec
ogni
tion
krko
pt
Shu
ttle(
1)
conn
ect-4
dataset
accu
racy
Average Rule AccuracyAverage Tree AccuracyAverage Function AccuracyAverage Instance AccuracyAverage Cluster Accuracy
Step 1 The control manager can perform an autonomous recovery action where the cost for the determination it is not so heavy .
Step 2 Persistent information can be analyzed in order to extract knowledge
Step 3 On-line information can be analyzed in order to detect possible malfunctions
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Fast Instrument Communication Channel
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Message Oriented Middleware Topic A
Topic B
• Subscribers Subscribe to a given Topic with a subscribe condition
• Publisher publish message in asynchronous way with a given message condition
• Publisher and subscribers can be part of the same program or in WAN distributed machines
• JMS Provide a standard set of API that standardize this communication system
• Many Commercial and academic implementation of this API exist in both C/C++ and Java (NaradaBrokering, Sun, IBM, SonicMQ etc etc )
In Our Case: • Each instrument can be a data
publisher or a data consumer • For more demanding application
an instrument must send/receive data in a streaming way
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
RMM-JMS• RMM-JMS is a JMS implementation on top of our high performance Reliable Multicast
Messaging (RMM) layer which provides one-to-one, one-to-many data delivery or many-to-many data exchange, in a message-oriented middleware point-to-point or publish/subscribe fashion
• The exceptional performance supports remote and distributed control and operation of scientific instruments such as sensors and probes
• Multicast transport for publish/subscribe messaging: Supporting the JMS Topic-based messaging and API, with matching done at the IP multicast level. The transport is a Nack-based reliable multicast protocol.
• Direct (broker- less) unicast for point-to-point messaging: JMS Queues are implemented over RMM queues. The transport is the TCP protocol.
• Brokered unicast transport for publish/subscribe messaging. The broker receives messages from the producer in either unicast or multicast delivery mode, and sends the messages to the subscribers in either mode
• broker serves as a bridge in a LAN-WAN-LAN configuration
Main Contribution of IBM Haifa Research Lab (Israel)
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Performance: message rate – the many-to-one
• Blade center with 12 CPUs and 1GB Ethernet switch • No message loss • Total throughput: 61MBytes/sec. and 67MBytes/sec. for (a) and (b) respectively
(b) rate - msg size 100000 bytes
0
100
200
300
400
500
600
700
800
900
1000
0 5 10 15 20Number of Publishers
msg
/sec min
MaxAvgSDev
(a) rate - msg size 1000 bytes
0
10000
2000030000
4000050000
60000
70000
80000
90000
100000
0 5 10 15 20
Number of Publishers
msg
/sec min
Max
Avg
SDev
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Performance: message rate – the one-to-many
• Blade center with 12 CPUs and 1GB Ethernet switch • No message loss • Peak result of over than 400000 msg/sec. was reached
Rate, msg size 1 Byte
0
100000
200000
300000
400000
500000
600000
0 5 10 15 20 25 30Number of Subscribers
msg
/sec min
Max
Avg
SDev
Rate, msg size 1000 bytes
0
10000
2000030000
40000
50000
6000070000
80000
90000
0 5 10 15 20 25 30
Number of Subscribers
msg
/sec min
MaxAvgSDev
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Performance: round trip time (RTT, Latency)
• Two machines with a single publisher and a single subscriber on each one • Average round trip time computed over 1000 samples
RTT
0.01
0.1
1
10
100
1 10 100 1000 10000 100000 1000000Messages Size
Tim
e (m
Sec)
AvgSdevPing
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Standard Grid Interaction
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Data Mover
• The task of this element is to get data from the “data collector” of the IM• Data can be accessed via:
– Web service interface for generic data dump (e.g. slow storage, spy stream, etc.)– grid storage element (SE) and available CEs can access to the data via an SRM
Interface– Http server and TCP communication for high performance had-hoc data transfer
• The Data Mover exposes its methods to the IE web service and can be instrumented itself as an instrument.
Instrument Resources
DataMover
DataCollector
IM
IE Web Service Interface: get_data()
SRM interface
Http Server andTCP/IP raw socketData
Collector
IM
DataCollector
IM
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Current IE Implementation a fist taste
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Manager Performances (I)Instrument Manager Invocations
0
10
20
30
40
50
60
1
HTTP Transport Layer
Invo
catio
n pe
r S
econ
d
Average
min
Max
Variance
Asyncronous msg Rate
0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Number of Client
Mes
sage
s pe
r S
econ
d
Average
min
Max
Variance
Virtual Instrument Grid Service (VIGS)A
cces
s C
ontro
l Man
ager
execute()
getState()
create()
destroy()
IMSProxy
DataCollector
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
Test 1
Test 2
Test 1: Web Service invocation and status switch of FSM
Test 2: Soap Server receiving XML message format. DOM based parser
Virtual Instrument Grid Service (VIGS)A
cces
s C
ontro
l Man
ager
Acc
ess
Con
trol M
anag
er
execute()
getState()
create()
destroy()
IMSProxy
DataCollector
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control ManagerIMSProxy
DataCollector
InputManager
EventProcessor
FSMEngine
ResourceProxy
Control Manager
Test 1
Test 2
Test 1: Web Service invocation and status switch of FSM
Test 2: Soap Server receiving XML message format. DOM based parser
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Instrument Manager Performances (II)Command Distribution Time
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
10 50 80 120
Number of Instrument
Ave
rage
Tim
e (s
ec) 1FM
1FM M
3FM
3FM M3FM 3PC
3FM 3PC M
1
2
3
1 + 2
3
1
3
1
Optimized environment
IM with CMS Instruments
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Summary Test output disabled
0
1000
2000
3000
4000
5000
6000
7000
8000
0 2 4 6 8 10 12 14 16 18 20 22
n publisher clientsm
sg/s
ec
xdaq
java
c++ (not xdaq)
java-C++contemporary
IMS
Errors/log/states messages(xml and java objs)
DB
TCP/IPPub/Sub
(JMS)
WebService
Interface
Summary Java vs Xdaq( C++) 1 socket output
0500
1000
15002000
25003000
35004000
0 2 4 6 8 10 12 14 16 18 20 22
n publisher clients
msg
/sec java
xdaq
JMS increasing the subscribers number
0
100
200
300
400
500
600
700
0 2 4 6 8 10 12
n subscriber clients
msg
/sec
w ith selectorfield
w ithoutselector field
Summary Mysql vs Oracle
0
500
1000
1500
2000
2500
0 2 4 6 8 10 12 14 16 18 20 22
n publisher clients
msg
/sec
Oracle 100Oracle 1
Mysql 100 Mysql 1
IMS Performances
IMSProxy IMS
Proxy
IMSProxy
….
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Main IE Pilot Applications: Power Grid
Instrument Manager
Instrument Element
...
Virtual Control Room
Virtual Control Room
Gas
Solar
Power Grid V.O
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Main GridCC Pilot Applications: Control and Monitor of high energy experiments
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Main GridCC Pilot Applications: Control and Monitor of high energy experiments
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
The CMS Data Acquisition
• O(104 ) distributed Objects to– control– configure– monitor
• On-line diagnostics and problem solving capability
• Highly interactive system (human reaction time - fraction of second)
• World Wide distributed monitor and control
2 107 electronics channels 40 MHz
100 Hz
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
CMS Prototype
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
CMS Prototype: IEs at work
Det 1Det 1Det 1
DAQ
TTS
FedBuilder RuBuilder
FilterFarm
Trigger
TOP
GTPe
DAQ
Detector1
8
- GridCC middleware used for CMS MTCC (Magnet Test and Cosmic Challenge)
- 11 Instrument Elements with a hierarchical topology
- Instruments are in these case Linux hosts where the cms on-line software is running
- More than 100 controlled hosts
- 25 days to the start of the data taking !
CMS Instrument Elements
DAQ IE Instrument Managers
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
1. Taking Control
Target domain
"zombies"
Pirated machinesDomain A
Pirated machinesDomain B
X
IDS Intrusion Detection System
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
A DDoS Attack Domain-wiseA DDoS Attack Domain-wise
Sensor Instrument Element
Target Domain
Sources of the attack
Sensor Instrument Element
Sensor Instrument Element
Sensor Instrument Element
Sensor Instrument Element
IDS Intrusion Detection System
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Main GridCC Pilot Applications: Remote Operation of an Accelerator
Elettra Synchrotron
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
The other GridCC pilot applications• Meteorology (Ensemble Limited Area Forecasting)
• Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories
• Geo-hazards: Remote Operation of Geophysical Monitoring Network (see first slides)
• Medical Devices need a close loop between the data acquisition and the output result
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Conclusion
• The GridCC project is integrating instrument into traditional computational/storage Grids.
• IEs need an high interaction and interactivity between itself and the users.
• The GridCC IE implementation is currently installed in heterogeneous applications
G. Maron, EuChinaGrid 1st Conference, Rome, September 18-19, 2006
Question?
• Thx for your time
Acknowledgement: The GridCC project is supported under EU FP6 contract 511382.
More information: www.gridcc.orgOn-line Demo at: http://sadgw.lnl.infn.it:2002/IEFacade