large-scale system integration with dds for scada, c2, and finance

The Real-TimeMiddleware Experts

Large-Scale System Integration with DDS for SCADA, C2, and Finance

Rick Warren, Principal [email protected]

mailto:[email protected]

© 2009 Real-Time Innovations, Inc. 2

What do I mean by “large system”?

Systems of systems– Modular, hierarchical design– Legacy components,

subsystems– Multiple technologies– Global scale

Decoupled subsystem lifecycles– Independent development– Independent deployment and

use– Independent management– Independent revision and

retirement

Multiple communities of interest– Different data interest,

entitlements– Non-uniform levels of trust

LANLAN

DDS #1(Initech)

DDS #1(Initech)

JMS(Dunder Mifflin)

JMS(Dunder Mifflin)

DDS #2(Acme)

DDS #2(Acme)

Legacy(COBOL ‘R’

US)

Legacy(COBOL ‘R’

US)

Satellite Links


What matters to integrators of large systems?

Governance (including over security)– What information will be exchanged?– Under what conditions will it be exchanged?– Who is allowed to exchange this information?– If these SLAs are violated, can the exchange be prevented?

Can I be notified?– (In the past, what has occurred wrt these SLAs?)

Isolation– When I connect A and B, ensure they don’t break (each other)– When I disconnect A, ensure B doesn’t break– When I connect C, don’t change A or B

Scalability (like in any system,

Fault Tolerance but stakes are higher)


Part 1: Architecture

(we’ll get to technology later)


SystemSystem

Component

Component

Component

Class

A Modest Proposition

State

Behavior

Class

Fundamental design principles scale– Abstraction: Provide interface based on relevant concepts– Encapsulation: Hide internal implementation, communication– Composition: Combine existing capabilities into new ones

Subsystem 2

P P

PP P

Subsystem 1

P P

PP P

Schematic of a Composed System

Subsystems may have different network environments

Integration may have different network environment than subsystems themselves

Data may need to be transformed / cleansed as it moves among subsystems

Routing / gateway services will adapt data types / formats / protocols

LAN LANWAN

Router/Gateway

Router/Gateway

Isolation

Additional Governance

Schematic of a Composed System

Wash, rinse, repeat

Subsystem 1

P P

PP P

Subsystem 2

P P

PP P

Router/Gateway

Router/Gateway

Same data model?Same network env.?

Same lifecycle?

Behavior unaffected?Understandable?

Apples and Oranges

P P

PP P

Subsystem 1 + Subsystem 2

P P

PP P

Subsystem 2

P P

PP P

Subsystem 1

P P

PP P

Router/Gateway

Router/Gateway


Nothing New Under the Sun

Subsystem Data Space






Integration Data Space


Router/Gateway

Router/Gateway

Router/Gateway



U.S. CombatData Space

U.S. CombatData Space

Allied CombatData Space

Allied CombatData Space

U.S. C4IData Space

U.S. C4IData Space



Router/Gateway

Router/Gateway

Router/Gateway

Defense Industry



NYC TradingData Space

NYC TradingData Space

London TradingData Space

London TradingData Space

Tokyo TradingData Space

Tokyo TradingData Space



Router/Gateway

Router/Gateway

Router/Gateway

Financial Services Industry



GenerationData Space

GenerationData Space

Substation #1Data Space






Router/Gateway

Router/Gateway

Router/Gateway

Power Industry


Scalability and Fault Tolerance

Fault tolerance: Router/gateway can’t be single point of failure. 3 answers:

– Persistent data survives system failures– Redundancy allows continued service during failure– Segmented/load-balanced configuration limits scope of failures

Scalability: How big can a subsystem be?1. How big a subsystem can you understand, test, and debug?2. How well does infrastructure scale?


Part 2: DDS


How Does DDS Stack Up?

Governance– Structural SLA: data type– Behavioral SLA: delivery QoS– Security– Problem

detection/prevention/notification– System monitoring and recording

Isolation– Prevent data “leaks” and side effects– Allow dynamic (dis-|re-)connection– Support independent integration and

evolution

Fault tolerance– Data persistence– Segmented/load-balanced routing– Redundant routing

Scalability– WAN connectivity– Peer-to-peer communication– Brokered/managed communication

– Built in– Built in– Products available; future standards– Built in

– Products available

– Built in: domains, partitions

– Built in– Built in, esp. w/ DDS-XTypes

– Built in– Built in if router uses DDS– Built in if router uses DDS

– Products available; future standards– Highly scalable– Protocol support if necessary


DDS-RTPS Supports Redundant Data Routing

WriteMyData

ReadMyData'

Transform and Forward

Router 1

Router 2

Multiple readers:

no problem

Multiple writers:prevent

duplicates!

Fortunately, built into

DDS-RTPS



ReadMyData'

DDS-RTPS Message

Inline QoS

Data

Inline QoS

Orig. Writer Info

Other Metadata

Original Writer Info

Orig. Writer GUID

Orig. Writer Seq. #

Identity of original writer

Identity of original sample

Reader discards duplicates based on these



ReadMyData'

Transform and Forward

Router 1

Router 2

Writer 5AC2

Data

Seq. # 42

Writer 5AC2

Data

Seq. # 42

Writer 5AC2

Data

Seq. # 42

Writer 5AC2

Data

Seq. # 42

Writer 5AC2

Writer 5AC2

Seq. # 42

Seq. # 42

If your impl. doesn’t support, fall back to ownership.

WriteMyData


DDS Scalability: Two Aspects

1. Application data– How does performance fall off as # participants, writers,

readers increase?– Any single writer/reader pair can saturate gig-E:

achieving high aggregate throughput is not main issue– Interesting issue: Fan-out – number of readers per writer

2. Discovery– How many DDS applications can discover one another?– Interesting issue: How many applications need to discover

one another?

Hard limits, if any, very dependent on DDS implementation, machine configuration, network, etc.


1. DDS Scalability: Application Data

DDS-RTPS reliable multicast scales at least: – …to hundreds of readers per writer– …with very little degradation in throughput.– Larger testing facilities welcomed.

< 15% decrease1~900 readers

RTI Data Distribution Service 4.3Red Hat EL 5.0, 32-bit2.4 GHz processorsGigabit EthernetUDP/IPv4Reliable ordered delivery


2. DDS Scalability: P2P Discovery Scenarios

Single domain, symmetric discovery– Everyone discovers everyone else– Easiest to configure; most challenging wrt scalability– RTI test results: 1,800 participants; 3.2M endpoint matches

Single domain, asymmetric discovery– Each participant only knows of certain others in its domain– Good for partitioning domains in which not everyone talks

Multiple domains– Greatest separation for subsystems that rarely exchange data

• DDS-RTPS maps to different IP ports

1

…

n


Summary: Large-Scale P2P Scalability

Experimental results: DDS-RTPS allows participant to talk to ≥ 1-2K others in single symmetric domain

Implication: Compose arbitrarily large systems out of subsystems this size or smaller

Large (sub)systems often have requirements for:– Increased governance, e.g. over security boundaries– Data transformation– Protocol mediation– These typically require data brokering anyway


DDS Scalability: Stronger Measures

Not big enough?

1. Brokered application data– Covered this

2. Brokered discovery

…

…

m

…

n

n < m

m

n


DDS Scalability: Brokered Discovery

Can be built with DDS-RTPS-standard building blocks

Application Topic

Built-In Topic

RTPS Metadata

Data

RTPS Metadata

Data

Orig. Metadata

RTPS Metadata

Data

RTPS InfoSource submessage provides data forwarding capability:“Treat the following as if it came from that guy instead of from me.”


Future Direction: Enhanced Built-in Security

System of systems often have non-uniform trust– Multiple communities of interest w/ different

entitlements/authorization– Infrastructure components may (not) be trusted

(e.g. brokers, daemons, persistence, recording/playback, etc.)– There is no “system high”

Desirable new specifications:– Fine-grained access control: topic, instance, type member

levels– Sample-level tagging and labeling– Fine-grained signing and/or encryption: topic, member level


Future Direction: Standardized WAN Support

Site-to-site comms need DDS across WAN(s)– Including untrusted networks– Including firewall, NAT traversal– Challenge: UDP may not be routable, especially if multicast

DDS-RTPS protocol designed for layeringatop diverse lower-level protocols– Proven: UDP, TCP, serial, switched fabric, Infiniband, VME,

shared memory…– Blessed for interop by OMG: UDP

Additional OMG-recognized protocol layeringscan improve site-to-site interop– TCP for WAN routing– (D)TLS for transport-level security (a clarification, not new)– Could be standardized quickly


Recap

Large systems usually composed of smaller systems– Developed independently– …With different network environments– …And different data interest/entitlements

Not an accident: hierarchical design best practice at every scale– Subsystems guarded by DDS router/gateway

• Restrict access to internal topics/flows• Mediate internal/external protocols• Transform/cleanse internal/external data types• Impose governance: enforce policy, attach monitoring

– Inter-router integration space is itself a “subsystem” for further hierarchical composition


Recap

DDS uniquely qualified to meet these needs– Strong SLAs for governance– Multiple topologies for fault-tolerant subsystem isolation– High-performance, scalable, deterministic interoperability– What other standard technology offers this combination?

(Not JMS. Not CORBA. Not AMQP. Not web services. …)

Promising future directions:– DDS-RTPS/TCP protocol layering for WAN traversal– Enhanced security model

• Fine-grained access control• Sample-level tagging, signing, and encrypting


Thank You

large-scale system integration with dds for scada, c2, and finance

Technology