urbanflood wp5 common information space (cis) after year 1
DESCRIPTION
UrbanFlood WP5 Common Information Space (CIS) after Year 1. Marian Bubak, Bartosz Baliś and WP5 team ACC Cyfronet AGH, Kraków, Poland {bubak,balis}@agh.edu.pl. Plan. Motivation, goals, tasks, and the team Objectives for Year 1 Results of requirement analysis Common Information Space - PowerPoint PPT PresentationTRANSCRIPT
UrbanFlood WP5 Common Information Space
(CIS)after Year 1
Marian Bubak, Bartosz Baliś
and WP5 teamACC Cyfronet AGH, Kraków, Poland
{bubak,balis}@agh.edu.pl
PlanMotivation, goals, tasks, and the
teamObjectives for Year 1Results of requirement analysisCommon Information Space
◦architecture◦ implementation status
CIS-based Flood EWSSummary of achievementsPlans for Year 2
Motivation & GoalsFacilitate creation, deployment and
robust operation of Early Warning Systems
Early Warning System: any system working according to four steps:◦ Monitoring◦ Analysis◦ Judgment◦ Action
Example: environmental monitoring Also: cloud infrastructure monitoring,
EWS self-monitoring
Tasks and the Team WP5 leader: Marian Bubak T5.1: Execution framework
◦ Integration platform: Marek Kasztelnik, Bartosz Balis◦ Data access: Piotr Nowakowski, Tomasz Gubala◦ Resource allocation: Tomasz Bartynski◦ Development of CIS-based EWSs: Bartosz Balis, Marek
Kasztelnik, Jeroen Broekhuijsen, Artem Ozhigin T5.2: Event infrastructure
◦ Communication Bus: Bartosz Balis, Marek Kasztelnik, ◦ Self-monitoring: Grzegorz Dyk, Bartosz Balis
T5.3: Provenance logging (Metadata)◦ Generic metadata service: Tomasz Gubala, Piotr
Nowakowski, Adam Belloum T5.4: User & developer GUIs
◦ Jeroen Broekhuijsen, Tomasz Gubala
Milestones: M12: The CIS integrates the (distributed) software
modules of the EWS
M24: The CIS manages (online) computing resources
M33: The CIS has been demonstrated to SEIS and /or INSPIRE representatives
Timeline
Requirement analysisExisting sources of live and archive sensor
dataExisting legacy applications
◦ Heterogeneous platformsComplex scenarios emerge, involving:
◦ Real-time processing of sensor data◦ Decision-making, automatic or by human
interaction◦ Compute-intensive simulations◦ What-if scenarios
High priority (urgent) computingSharing of limited resources
System requirementsApplication integration: enable data
exchange between diverse components◦ Legacy, heterogeneous
Orchestration of components into composites◦ Scientific / business workflows◦ Integration patterns
Dynamic resource allocation managementMetadata management
◦ Domain (applications, data)◦ System information (resources, data sources,
running tasks)◦ Provenance
Overview of CIS architecture
PlatIn: the Integration PlatformStart and stop EWS
capability implemented (Every EWS part delivers management WS)
Integrated with UfoReg (receiving EWS metadata)
EWS parts can be created using BPEL or Camel integration technology
UFoReg: the UrbanFlood Registry
Stable prototype deployed at Cyfronet and populated with initial sensor/EWS/Cloud data
Basic GUI available RESTful API interfaces are in place to enable integration
of UFoReg with DyReAlla and visualization components
Currently includes three data models:◦ Sensor metadata
model◦ EWS model
(describing EWSs and their appliances)
◦ Cloud model (describing the available hardware resources and virtual system images, and recent cloud state)
UfoReg Service
Integration platform UI
Applicationmanager
Administration Tool
Rule engineRoute engineBPEL EngineUser Tool
Provenance API
log provenance and historical data
Search API
search configuration
search domain and historical metadata
search appliance
REST API
Storage
Persistence
Domain Model
Deployed schemalessdatabase populateddynamically with domainmodel objects
Flexible, extensible modelnotation to cover growingnumber of use cases andvarious types of metadatato store and publish
DyReAlla: Dynamic Resource Allocation service Integrated with
cloud infrastructure (receiving reports about resource availability and load metrics)
Integrated with UfoReg (storing cloud status data)
Not in the first milestone / release of CIS
Paradigms and technologies usedEnterprise Service Bus (Open ESB)Business Process Management
(BPEL)Enterprise Application Integration
framework (Apache Camel)Message-Oriented communication
middleware (ActiveMQ)Robust NoSQL database (MongoDB)
Early Warning System – the CIS view
Each EWS is composed of: (1) Appliances, (2) Parts, (3) External components
Appliance: application component exposed as a service and wrapped into a virtual image◦ Typically legacy EWS-specific application
EWS Part: composite integrating and orchestrating control and data flow between appliances (and possibly other parts)◦ Also can be exposed as a high-level service
External component: external producer or consumer of data, not managed by CIS
Generic operations: ◦ Start, stop◦ Change alert
levelInvokes
appliancesFulfills well-defined high-level service exposed via part-specific interface
Sends messages to and receives from a message bus
External Component
EWS Part
start stop
Invoke<<service bus>>
part-specific interface
Message Bus
send message
receive message
Appliance
alert levelEWS Part
Loose coupling
Parts are loosely coupled Communicate via the message busNo direct dependencies between partsFacilitates extendibility of the EWS
◦ Addition of new parts that further process already published data
◦ Connecting new visualization front-ends◦ Can even be done at runtime
EWS PartEWS Part Message Bus
Attention (alert) level
General state of every EWS shared by all its partsA Part may send ’alert level change’ (raise or
decrease) messageAlert Level Manager (general-purpose
specialized EWS part) receives the messages, calculates a new alert level, and sends ’alert level set’ message
Other EWS parts may adjust their operation depending on the alert level
Attention Level Manager
Other EWS Part Message Bus
Receive attention level
change messages
Send attention level
change messages
Send attention level set
messages
Receiveattention level set
messages
Flood Early Warning SystemMonitoring of dikes using wireless
sensorsAI-based detection of sensor signal
anomaliesDike failure predictionSimulation of inundation due to failureVisualization and user interactions on
Multi-touch Tables
Flood EWS implemented with CIS
AnySense
Storage (AnySense)
Multi-Touch Table
HRW DRFSM
HRW Hydrograph
FloodSimulation
Part
Simulation commands
Simulation results
Reliable
AI-based Monitoring Part
Sensordata /
Anomaly probability
AIFiltered
sensor data
Attention level
change
Sensordata
Reliable Monitoring Part
Filtered sensor data
Attention level
change
Simulation commands
Simulation results
Sensordata
Anomaly probability
CIS Message Bus
Archiver Part
Attention level set /
Dike failure prob.
Dike failure probability
Attention Level Manager Part
Attentionlevel set
Attention level
change
EWS Part
ApplianceExternal
componentCIS
Technology
UFoReg (CIS metadata registry)
Legend:
Self-monitoring & cloud monitoring as Early Warning Systems
Each EWS has: Its own instance of
PlatIn, the CIS integration platform,
Specific appliances in the cloud
EWS Parts: Composites which orchestrate the execution of appliances
Multiple EWSs: EWS1, EWS2: two instances of Dike Monitoring EWS EWS3: Monitoring and management of other EWSs
(DyReAlla and UFoReg) EWS4: Cloud monitoring and management
Deliverables, milestones & meetings in Y1Deliverables:
◦ D5.1 Common Information Spaces (description of state of the art and future developments) (M8)
◦ D5.2 Specification of the architecture and interfaces of the Common Information Space (M9)
◦ D5.3 Orchestrating the information flow in a Common Information Space (M12)
Milestones:◦ M12: The CIS integrates the (distributed) software
modules of the EWSMeetings:
◦ Kick-Off meeting in Groningen◦ Consortium meeting in Wallingford (definition of first
EWS)◦ Integration meeting in Cracow (first EWS protoptype)◦ Integration meeting in Amsterdam (first EWS
production run)◦ Teleconferences: every Wednesday (current issues
discussions, progress reports)
Publications & presentations B. Balis, T. Bartynski, M. Bubak, M. Kasztelnik, and P. Nowakowski: The
UrbanFlood Common Information Space. Poster and presentation at CGW10 – Cracow Grid Workshop 2010, Krakow, October 11-13, 2010, http://www.cyfronet.pl/cgw10
M. Bubak: Towards Collaborative Workbench for Science 2.0 Applications. Presentation at HPC 2010 – High Performance Computing, GRIDS and clouds, An International Advanced Workshop, June 21 – 25, 2010, Cetraro, Italy, http://www.hpcc.unical.it/hpc2010
B. Balis, T. Bartynski, M. Bubak, M. Kasztelnik, P. Nowakowski: Common Information Space, a framework for creating and hosting Early Warning System. Poster and presentation at Joint UrbanFlood & SSG4Env Workshop, 11-12 November 2011, Amsterdam, http://urbanflood.eu/urbanFloodWorkshop2010.aspx
B. Balis, M. Kasztelnik, M. Bubak, T. Bartynski, T. Gubala, P. Nowakowski, J. Broekhuijsen: The UrbanFlood Common Information Space for Early Warning Systems. Submitted to the ICCS 2011 Conference; to be published (if accepted) in Elsevier Procedia Computer Science, http://www.iccs-meeting.org/
V.V. Krzhizhanovskaya, G.S. Shirshov, N.B. Melnikova, R.G. Belleman, F.I. Rusadi, B.J. Broekhuijsen, B. Gouldby, J. L'Homme, B. Balis, M. Bubak, A.L. Pyayt, I.I. Mokhov, A.V. Ozhigin, B. Lang, R.J. Meijer. Flood early warning system design and implementation. Submitted to the ICCS 2011 Conference; to be published (if accepted) in Elsevier Procedia Computer Science, http://www.iccs-meeting.org/
Plans for Y2Task 5.1 Improvement of functionality of all modules
(e.g. support for self-monitoring) Implementation of the dynamic resource
allocation mechanismFurther implementation and refinement of
components of the CIS-based flood EWS. Task 5.2Development of a robust software sensor
network for self-monitoring Improvement of scalability and flexibility of
self-monitoring infrastructure Improvement of robustness of the
communication services.
Plans for Y2 (ctd)Task 5.3 Continuous optimization of storage, gathering
and publishing techniques inside UFoReg Support for the provenance model Continuous extending domain models when
requested by partners and required by the CIS/EWS design
Task 5.4 Implementation of graphical dashboards for
users and developers for monitoring status and health of CIS and EWSs
Integration of CIS services and CIS-based EWS services with graphical user interfaces of the Decision Support System
Development of auxiliary developer tools for automation of CIS-based EWS development.
Means to high-quality softwareAdoption of mature, industry-quality,
standards-based approaches and technologies ◦ Apache, MongoDB, OpenESB, BPEL, Passenger
At the same time adoption of platform and technology-independent design in order to avoid vendor lock-in◦ SOA, loose coupling between components ◦ Communication based on well-defined protocols
Self monitoring inherent part of the CIS design
Test driven development
Hardware contributed by CyfronetComputer host (Y1)
◦ 2 dualcore processors Intel Xeon CPU 5150 @2.66GHz, 4 GB RAM, 500 GB storage
◦ Deployed services CIS (ActiveMQ JMS, Glassfish server, UFoReg,
DyReAlla, Health monitoring) EWS parts (message filters, converters etc)
4 nodes (from February 2011)◦ 2 quadcore processors Intel Xeon CPU
L5420 @2.50GHz, 16 GB RAM, 120 GB local storage, access to shared storage over iSCSI
◦ Dedicated for hosting UF appliances (Flooding simulator,Reliable, Hydrograph, DRFSM, AnySense, AI)
www.urbanflood.eudice.cyfronet.pl/cis