september 2.001

September 2.001

ENGINEERING ETHERNET BASED SOLUTIONS

This presentation is based in GTD experience integrating control and supervision architectures for Science, Space,

Defense and Industrial applications.

By the Software Support Team (GTD St.Genis) and the Cryogenics Team (GTD BCN)

Projects with relevant GTD involvement: XMM-Newton Space Telescope, Eurofighter, Arianne V launcher, Spaceport Control-room and Accelerators (Grenoble and CERN)

ENGINEERING ETHERNET BASED SOLUTIONSBy the Software Support Team (GTD St.Genis) and

the Cryogenics Team (GTD BCN)

1) SOLUTIONS: … to a problem. This presentation discuss suitable problems for ETHERNET application.

2) ENGINEERING: No general solution (at least for the most relevant problems). Criteria to create suitable solutions, identify major constraints and defined the proper integration strategy are also introduced.

PROBLEM OVERVIEW1.1DisplayingLarge amount of data in human real-time scale (i.e.: seconds)RegisteringNon real-time large amount of data with high time-stamp accuracy… … …Processing/ControlVariable amount of data in process real-time** scale (1ms – 500ms)*Data Supply for:

- Displaying- Registering

… … …

*) Ranges for industrial (PLC based) control systems in GTD**)REAL-TIME: known, accurate - i.e. 1ms - & controllable

AcquisitionVariable amount of data in process real-time scale** (1ms – 500ms)*Data Supply for:

- Processing/Control- Displaying- Registering

… … …

RequestingReduced amount of data in human

time scale (seconds)Configuring/Backing-up

Large amount of data non real-time event-driven (i.e. start-up)

… … …Commanding

Variable amount of data in process real-time** scale (1ms – 500ms)*

… … …

ActuationVariable amount of data in process

real-time** scale (1ms – 500ms)*… … …

the Processthe Process

the Controlthe Control

the Supervisionthe Supervision

Synchronous / Time[ms]

Synchronous / Time[s]

Asynchronous

Asynchronous

Asynchronous

requestsrequests

commandscommands

records: records: events, trends, …events, trends, …

sub-samplingsub-sampling

statusstatus

data supplydata supply

data supplydata supply

the Processthe Process

the Controlthe Control

the Supervisionthe Supervision

PROBLEM OVERVIEW1.2

synchronous

asynchronous

Max Data Quantity

Time Schedule

Max Data Quantity

Time Limits data

Real-TimeNon

Real-Time

Bw1 Bw i Bw j Bw n

Application Domain ofEthernet as Fieldbus

PROBLEM OVERVIEW1.3

data

TIMETIMEacquisition processing

known& fixedT ± T

THE ENGINEERING APPROACHTHE CONCEPT

2.1

PROBLEM

URD

… …

TOPOLOGY

ARCHITECTUREADD

For Processes affecting multiple devices:• Nature (synchronous/asynchronous)• Real Time Constraints• Bandwidth (Bw)

FUNCTIONALITIES

validated 1 on 1

ETHERNETapproach

+ complexity+ performance+ new functionalities+ data availability*/richness+ flexibility

may block (partially or totally) Ethernet applicability

Synchronous(1 (typically Closed Control Loops signals – “feedback” and “output”) Real Time Requirements

0< Tx <10ms Tiny Bandwidth Limited (Field)bus participants Local Processing (eventually by distributing processing effort)

10ms< Tx <50ms Limited Bandwidth Limited (Filed)bus participants Deterministic FieldbusDevice’s efficiency for TCP/IP packaging is limited under 50ms

50ms < Tx Ethernet Applicable (Bw occupation 1:10) for “determinisms” New topologies (system redesign!) Greater Bw Packets oriented transmission for Bw optimization (Protocol “Overhead”)

Fieldbus still ApplicableLess Bw availableLess flexible topologies (several distance constraints)

1) Synchronous: Deterministic time control on processes concurrently running at different devices.

THE ENGINEERING APPROACHTHE CONCEPT

2.2

AsynchronousReal Time Requirements

0< Tx <10ms [i.e. Open Loop Control – Alarm Handling, Time-stamping, …] Tiny Bandwidth

Limited (Field)bus participants * Deterministic Fieldbus > Polling / Packages > Slow! Local Processing (eventually by distributing processing effort)

10ms< Tx <50ms [i.e. Open Loop Information refresh]* “Messages” better than “token”* Token > Polling / Packages limited efficiency (overhead) TCP/IP with PROTOCOL + Tx Policy

50ms < Tx [i.e. (very) large systems for info refresh, buffered data, configurations’ up/download, …]* fully package oriented Ethernet Best Choice (Bw occupation 1:10) for “confidence” New topologies (system redesign!) Greater Bw TCP/IP with PROTOCOL + Tx Policy

THE ENGINEERING APPROACHTHE CONCEPT

2.3

Real TimeRequirements

10ms 100ms50ms0

SYNCHRONOUS

ASYNCHRONOUS

for Increasing Bandwidth

ETHERNETTCP/IP

DeterministicFieldBus

suit

abili

ty

Reco

mm

en

dable

… a

t th

e lim

it

Not

reco

mm

en

dable

THE ENGINEERING APPROACHTHE CONCEPT

2.4

THE ENGINEERING APPROACHENGINEERING A SOLUTION

3.1

ETHERNETTCP/IP

10MBits100MBits

IndustrialControl& I/O

Devices

But … … …• What is the effective Bandwidth occupation?• What are the constraints?• How shall the information exchange be organized?

Ethernet(IEEE 802.3)

TCP/IP

MiddlewareServices

1

2

3

4

5

6

7

THE ENGINEERING APPROACHENGINEERING A SOLUTION

3.2

Physical

Data Link

Network

Transport

Session

Presentation

Application Layer

High Level Protocols

Information Organization

APPLICATION

throughput

More data managed (maybe not more knowledge!)

ENGINEERING: Superprotocol + BW management

Some constraints: somehow inherited structures

Some immaturity adding TCP/IP services to certain PLC’s Operating Systems.

Some immaturity degrees in the firmware connected to proprietary architectures.(is it becoming a PLC weakness?!?) h

ard

ware

THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

3.3

supervision layer

control layer

field layer

FIRST IDEA ... Derived literally from Specification

SynchronousTime Stamping10ms<9.000 Binaries( 275.000 Binar.)

VisualizationSynchronous~2s<50.000 Binaries<22.000 Floats( 700.000 Binar.)( 300.000 Floats)

SynchronousProcessTs=500ms(150ms)< 9.000 Binaries<15.000 Floats( 275.000 Binar.)( 105.000 Floats.)

EventsAsynchronous<acceptable delayfrom 700.000 BinariesMax 500 events/s( 4.000 events/s)

SynchronousProcessTs=500ms(100ms)< 3.500 Binaries< 3.500 Floats( 155.000 Binar.)( 160.000 Floats.)

AsynchronousManual Requests~1s Human Feeling< 3.500 Binaries( 155.000 Binar.)

13

Bw~8MBitsBw~10,5MBits

2

THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

3.4

IMPLEMENTATION (as it is today!)

VisualizationAsynchronous(worst case 2s)<50.000 Binaries<22.000 Floats( 700.000 Binar.)( 300.000 Floats)

ProcessAsynchronous(worst case 150ms)< 9.000 Binaries< 5.000 Floats( 275.000 Binar.)( 105.000 Floats.)

EventsAsynchronous<acceptable delayfrom 700.000 BinariesMax 500 events/s( 4.000 events/s)

ProcessAsynchronous(worst case 100ms)< 3.000 Binaries< 2.000 Floats( 155.000 Binar.)( 160.000 Floats.)

AsynchronousManual Requests~1s Human Feeling< 3.500 Binaries( 155.000 Binar.)

TIME STAMPINGTIME STAMPING

TIME STAMPINGTIME STAMPING

X 165

X 64

X 8(redundant)

2 x TCP/IP cards

**

**

1

2

3

4

SAMPLINGSAMPLING

3.5THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

summarizing the chosen approach …

1) Distribution of the Time-stamping process. Time-stamping at the source eliminates synchronous communication.>>> It is feasible to move to a full ETHERNET TCP/IP solution>>> Requires specific Supervision system

2) Duplication of networks at the Control Layer to make synchronous exchanges “deterministic” and/or reduced collisions (moreover, dimensioning of the number of devices)>>> Measure performances of the Hardware

3) Eliminate unnecessary throughputs by directly addressing the information item to the target (favored by the flexible Ethernet architecture)

4) Bandwidth Management: Synchronous >>> Asynchronous!(further explained)

3.6

TCP/IP 100MBitsOpen Modbus CPS211 00

CPU534 14NOE771 00NOE771 00

0

100

200

300

400

500

600

1 Port 2 Ports 4 Ports 8 Ports 16 Ports 24 Ports 30 Ports

Kbits/s

100 ms70 ms50 ms40 ms

0

5

10

15

20

25

30

35

40

45

50

1 Port 2 Ports 4 Ports 8 Ports 16 Ports 24 Ports 30 Ports

Percentage

100 ms70 ms50 ms40 ms

Percentage of the PLC cycle time occupied by communications related tasks [%] for n parallel ports/channelsand different PLC cycles.

Performance (Bw)[Kbit/s] for n parallel managed communication’s ports/channels.Curves represent different PLC cycles (40ms .. 100ms)

THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

HARDWARE evaluation (example … at the Control Layer)

3.7

0

5

10

15

20

25

30

35

40

45

50

0 100 200 300 400 500 600

Band width

Du

ty c

ycle

100 ms70 ms

50 ms40 ms

0

5

10

15

20

25

30

35

40

45

50

0 100 200 300 400 500 600

Band width

Du

ty c

ycle

1 MASTER2 MASTERS4 MASTERS8 MASTERS16 MASTERS24 MASTERS30 MASTERS

Duty Cycle / Band Width ratio for different PLC Cycles.

Duty Cycle / Band Width ratio for different number of parallel opened

Ports/channels (MASTERS)

Good conditions found for 16 parallel managed ports/channels in a 50ms PLC cycle, thus resulting in approx. 256 Kbits/s of bandwidth and less than 20% PLC cycle occupation due to communication’s management (approx. 10ms).

THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

HARDWARE evaluation (example … at the Control Layer)

Event Driven Protocols / Tx Policies:>>> Oriented to Asynchronous strategies

a) People use to neglect “worst case” conditionsb) Steady conditions shall be carefully understood

>>> Event Driven use to be optimized for reduced pieces of INFOWorst conditions use to mean: large quantities!

>>> Optimization is achieved by the probabilistic fact of “little changes”“worst case” and “close to worst case” are not negligible

>>> Oriented to Synchronous strategiesIt is feasible by:a) Dimensioning for the worst caseb) Sampling to regularize data flow

>>> What is the advantage?Bandwidth Management

a) To support concurrent, less demanding, Asynchronous communication processes. b) To reduce collision probability in steady conditions

3.8THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

PROTOCOL & BANDWIDTH MANAGEMENT

Ti

Tk

Ts

process

Ts

THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

3.9

IMPLEMENTATION (as it is today!)

Maximum 8 Control Units

(PCUs) for each DS

Maximum 4 Field

Interfaces (FIs) for

each PCU

8 Redundant Data Servers(Bi-Pentium)

EventsVisual

Trends

Requests

EventsVisual

Requests

Process

ProcessEventsVisualTrends

Process

ProcessRequests

SAMPLINGSAMPLING

TIME STAMPINGTIME STAMPING

TIME STAMPINGTIME STAMPING

Example PCU:Quantum 534 14 + 2x NOE 771 00Event Driven (by GTD) on Open Modbus

Steady “worst conditions” (= Polling)Bw = 256 Kbit/s available- needed for Superv. 200KBit/s- needed for Process.160KBit/s

Better than “worst conditions”- Bandwidth availability for true Asynchronous processes.

ENGINEERING ETHERNET BASED SOLUTIONSconclusions

synchronous

asynchronous

Max Data Quantity

Time Schedule

Max Data Quantity

Time Limits data

Real-TimeNon

Real-Time

Bw1 Bw i Bw j Bw n

Application Domain ofEthernet as Fieldbus

PROBLEM OVERVIEW1.3

data

TIMETIMEacquisition processing

known& fixedT ± T

Ethernet(IEEE 802.3)

TCP/ IP

MiddlewareServices

1

2

3

4

5

6

7

THE ENGINEERING APPROACHENGINEERING A SOLUTION

3.2

Physical

Data Link

Network

Transport

Session

Presentation

Application Layer

High Level Protocols

Information Organization

APPLICATION

throughput

More data managed (maybe not more knowledge!)

ENGINEERING: Superprotocol + BW management

Some constraints: somehow inherited structures

Some immaturity adding TCP/IP services to certain PLC’s Operating Systems.

Some immaturity degrees in the firmware connected to proprietary architectures.(is it becoming a PLC weakness?!?)h

ard

ware

1. Synchronous-Asynchronous2. Data Quantity3. Time Frame

Our recommendation:

Try to use Ethernet for1. Synchronous, over 75ms2. Asynchronous, over 25ms

Real TimeRequirements

10ms 100ms50ms0

SYNCHRONOUS

ASYNCHRONOUS

for Increasing Bandwidth

ETHERNETTCP/ I P

DeterministicFieldBus

suit

abili

ty

Reco

mm

endable

… a

t th

e lim

it

Not

reco

mm

endable

THE ENGINEERING APPROACHTHE CONCEPT

2.4

Still difficulties:

1. Certain immaturities2. Inherited standards3. Large/Complex applications (more than necessary !?!)

> Engineering Effort a) To adapt resources b) To create Tx policies c) To benefit from Ethernet added values.

Case Study:

Characterized by … 1. Application “size”2. Information flow3. Synchronous/Asynchronous

Ethernet TCP/IP not well suited for synchronous, fast processes

THE ENGINEERING APPROACHENGINEERING A SOLUTION

A case study ... the Control System for the LHC Cryogenics

3.9

IMPLEMENTATION (as it is today!)

Maximum 8 Control Units

(PCUs) for each DS

Maximum 4 Field

Interfaces (FIs) for

each PCU

8 Redundant Data Servers(Bi-Pentium)

EventsVisual

Trends

Requests

EventsVisual

Requests

Process

ProcessEventsVisualTrends

Process

ProcessRequests

SAMPLI NGSAMPLI NG

TIME STAMPINGTIME STAMPING

TIME STAMPINGTIME STAMPING

Example PCU:Quantum 534 14 + 2x NOE 771 00Event Driven (by GTD) on Open Modbus

Steady “worst conditions” (= Polling)Bw = 256 Kbit/s available- needed for Superv. 200KBit/s- needed for Process.160KBit/s

Better than “worst conditions”- Bandwidth availability for

true Asynchronous processes.

THE ENGINEERING APPROACHTHE CONCEPT

2.1

PROBLEM

URD

… …

TOPOLOGY

ARCHITECTUREADD

For Processes affecting multiple devices:• Nature (synchronous/ asynchronous)• Real Time Constraints• Bandwidth (Bw)

FUNCTIONALITIES

validated 1 on 1

ETHERNETapproach

+ complexity+ performance+ new functionalities+ data availability* /richness+ flexibility

may block (partially or totally) Ethernet applicability

September 2.001