legnaro event builder prototypes
DESCRIPTION
Legnaro Event Builder Prototypes. Luciano Berti, Gaetano Maron. INFN – Laboratori Nazionali di Legnaro. GE Event Builder. Components: Hardware: switch: FoundryNet FastIron NIC: SysKonnect SK9821 PC: Supermicro PIII (i840) Software: vxWorks based. 15 x 15. Test conditions: - PowerPoint PPT PresentationTRANSCRIPT
G. Maron CPT Week CERN, 23 April 2001 1
LNL
CMS
Legnaro Event Builder PrototypesLegnaro Event Builder Prototypes
Luciano Berti, Gaetano MaronLuciano Berti, Gaetano Maron
INFN – Laboratori Nazionali di LegnaroINFN – Laboratori Nazionali di Legnaro
G. Maron CPT Week CERN, 23 April 2001 2
LNL
CMS
GE Event BuilderGE Event Builder
Test conditions:Test conditions:• No command or event aggregation (each packet No command or event aggregation (each packet transports a command or data frame relative to a single transports a command or data frame relative to a single event)event)• full data transfer from/to PC memoryfull data transfer from/to PC memory• recovery from packet lossrecovery from packet loss• fixed fragment sizes are varied 400-4000 bytesfixed fragment sizes are varied 400-4000 bytes
Components:Components:Hardware: Hardware: • switch: FoundryNet FastIron switch: FoundryNet FastIron • NIC: SysKonnect SK9821NIC: SysKonnect SK9821• PC: Supermicro PIII (i840) PC: Supermicro PIII (i840) Software: vxWorks basedSoftware: vxWorks based 15 x 15
G. Maron CPT Week CERN, 23 April 2001 3
LNL
CMS
Event builder layoutEvent builder layout
1 1515141413131212111110109988776655443322
1 1515141413131212111110109988776655443322
EVMEVM
RUsRUs
BUs
Slot 1Slot 1 Slot 2Slot 2 Slot 3Slot 3 Slot 4Slot 4
RUs and BUs distributed in all switch slots:RUs and BUs distributed in all switch slots:
– Part of the traffic localized within the slotPart of the traffic localized within the slot
– Reduces switch backplane utilizationReduces switch backplane utilization
RU performanceRU performance problem found withproblem found with this configuration this configuration
G. Maron CPT Week CERN, 23 April 2001 4
LNL
CMS
Modified Event Builder layoutModified Event Builder layout
11 1515141413131212111110109988776655443322
11 1515141413131212111110109988776655443322
EVMEVM
RUsRUs
BUsBUs
Slot 1Slot 1 Slot 2Slot 2 Slot 3Slot 3 Slot 4Slot 4
Request data Request data commandscommands
Fast EthernetFast EthernetSlotSlot
Request data Request data commandscommands
- RU fast control message over FE (PCI 32/33)RU fast control message over FE (PCI 32/33)- RU data transfer on GE (PCI 64/66) RU data transfer on GE (PCI 64/66)
G. Maron CPT Week CERN, 23 April 2001 6
LNL
CMS
EB protocolEB protocol
BUBU
allocateallocate
EVMEVM
confirmconfirm
11 22 33 nn
RUsRUs
sendsend
cachecache
G. Maron CPT Week CERN, 23 April 2001 7
LNL
CMS
Concurrent building threads in the same BUConcurrent building threads in the same BU
BUBU EVMEVM 11 22 33 nn
RUsRUs
BU thread 1BU thread 1BU thread 2BU thread 2BU thread 3BU thread 3
G. Maron CPT Week CERN, 23 April 2001 8
LNL
CMS
BU
allocate
EVM
confirm
Sequential readingSequential reading
1 2 3 n
RUs
BU
allocate
EVM
confirm
Random reading
1 2 3 n
RUs
4 5
send
cache
send
cache
Sequential vs Random readingSequential vs Random reading
G. Maron CPT Week CERN, 23 April 2001 9
LNL
CMS
BU
allocate
EVM
confirm
1 2 3 n
RUs
4 5
send
cache
““Sliding Window”Sliding Window”
• multiple send to Rus multiple send to Rus
• reduce the total reduce the total rebuilding timerebuilding time
• less events in the less events in the BusBus
• not yet testednot yet tested
G. Maron CPT Week CERN, 23 April 2001 10
LNL
CMS
• No difference on performanceNo difference on performance
• But more allocated event But more allocated event needed on BUs, needed on BUs,
• All the measurements with All the measurements with random readingrandom reading
Sequential readingSequential reading Random readingRandom reading
Sequential - random reading comparisonSequential - random reading comparison
G. Maron CPT Week CERN, 23 April 2001 11
LNL
CMS
Recovery from Packets lossRecovery from Packets loss
BUBU
Req. DataReq. Data
RURU timertimer
startstart
EvtDataEvtData
Req. DataReq. Data( retry ) ( retry )
startstart
cancelcancel
timeouttimeout
EvtDataEvtData
BUBU
Req. EvtIdReq. EvtId
EVMEVM timertimer
startstart
EvtIdEvtId
Req. EvtIdReq. EvtId(retry)(retry)startstart
cancelcancel
timeouttimeout
EvtIdEvtId
BU – EVM communicationBU – EVM communication BU – RU communicationBU – RU communication
TimeoutsTimeouts80 - 160 ms80 - 160 ms
G. Maron CPT Week CERN, 23 April 2001 12
LNL
CMS
EVB 15x15 performance - ThroughputEVB 15x15 performance - Throughput
0
20
40
60
80
100
120
140
0 500 1000 1500 2000 2500 3000 3500 4000
Fragment Size (Byte)
Thro
ughp
ut p
er n
ode
(MB/
s)
• Throughput up to 116 MB/s, ie 93% link speedThroughput up to 116 MB/s, ie 93% link speed• no packet loss observed (as expected)no packet loss observed (as expected)
15 x 15
G. Maron CPT Week CERN, 23 April 2001 13
LNL
CMS
EVB ScalingEVB Scaling
0
20
40
60
80
100
120
140
0 2 4 6 8 10 12 14 16
N
Th
rou
gh
pu
t p
er
no
de
(M
B/s
)
4000 bytes
2000 bytes
400 bytes
G. Maron CPT Week CERN, 23 April 2001 14
LNL
CMS
EVB Performance – Event RateEVB Performance – Event Rate
0
20
40
60
80
100
120
0 500 1000 1500 2000 2500 3000 3500 4000 4500
fragment size (bytes)
Fra
gm
en
t ra
te p
er
no
de
(k
Hz)
Nominal fragmentNominal fragmentsize 2kbytes:size 2kbytes:event rate = 52 kHzevent rate = 52 kHz
15 x 15
G. Maron CPT Week CERN, 23 April 2001 15
LNL
CMS
Conic Event BuilderConic Event Builder
EventEvent ManagerManager
Builder NetworkBuilder Network
RURU
BUBU
FUFU
FUFU
FUFU
FUFU
EventEvent ManagerManager
RURU
FUFU FUFU FUFU FUFU FUFU FUFU FUFU FUFU
Builder NetworkBuilder Network
FUFU FUFU FUFU FUFU
symmetric EVBsymmetric EVB conic EVBconic EVB
• faster ports at Rusfaster ports at Rus• slower ports at BUsslower ports at BUs
G. Maron CPT Week CERN, 23 April 2001 16
LNL
CMS
Conic Event Builder: LayoutConic Event Builder: Layout
11 443322
11 292927272525232321211919171715151313111199775533
EVMEVM
RUsRUs
FUsFUs
GE GE Slot 1Slot 1
Request Data Request Data CommandCommand
39393737353533333131
22 30302828262624242222202018181616141412121010886644 40403838363634343232
FE FE Slot 1Slot 1
FE FE Slot 2Slot 2
G. Maron CPT Week CERN, 23 April 2001 17
LNL
CMS
EVB throughput – Conic vs SymmetricEVB throughput – Conic vs Symmetric
0
20
40
60
80
100
120
140
0 500 1000 1500 2000 2500 3000 3500 4000
Fragment Size (Byte)
Th
rou
gh
pu
t p
er
no
de
(M
B/s
)
symmetric 15x15
conic 4x40
conic EVB: no performance degradation vs symmetric conic EVB: no performance degradation vs symmetric
4 x 40
G. Maron CPT Week CERN, 23 April 2001 18
LNL
CMS
EVB Conic – ScalingEVB Conic – Scaling
1 x 10
4 x 40
2 x 20
0
20
40
60
80
100
120
140
0 1 2 3 4 5
N
Th
rou
gh
pu
t p
er
no
de
(M
B/s
)
4000 bytes2000 bytes400 bytes
G. Maron CPT Week CERN, 23 April 2001 19
LNL
CMS
Conic: RU/FU Conic: RU/FU Throughput ratioThroughput ratio
1 x n
2 x n
4 x n
G. Maron CPT Week CERN, 23 April 2001 20
LNL
CMS
To be done and test To be done and test
• variable size eventsvariable size events
• EB performances with the new implemented “ window” EB performances with the new implemented “ window” mechanismmechanism
• latency times measurementslatency times measurements
• Fault generation with the new implemented Random Error Fault generation with the new implemented Random Error Generator to check the error recovery procedureGenerator to check the error recovery procedure
G. Maron CPT Week CERN, 23 April 2001 21
LNL
CMS
Multistage Event BuilderMultistage Event Builder
• All our results have been obtained with a single All our results have been obtained with a single switch event builder configurationswitch event builder configuration
• We propose to extend our tests to a multistage We propose to extend our tests to a multistage ethernet switches topology and to study the behavior ethernet switches topology and to study the behavior of this configuration.of this configuration.
G. Maron CPT Week CERN, 23 April 2001 22
LNL
CMS
Plain TopologyPlain Topology
• In the Event In the Event Builder application Builder application data flows in only data flows in only one directionone direction
• The inter-switch The inter-switch Gigabit Ethernet Gigabit Ethernet links are full-links are full-duplexduplex
• Result : half of the Result : half of the inter-switch inter-switch bandwidth bandwidth available is wastedavailable is wasted
RUsRUs
BUsBUs
G. Maron CPT Week CERN, 23 April 2001 23
LNL
CMS
Full Mesh TopologyFull Mesh Topology
• RU and BU RU and BU distributed in distributed in all the switchesall the switches
• Inter-switch Inter-switch links are used links are used in both in both directiondirection
• Same number Same number of ports of the of ports of the plain topologyplain topology
• Twice of the Twice of the bandwidth of bandwidth of the plain the plain topology in the topology in the inter-switch inter-switch linkslinks
RUsRUs
BUsBUs
G. Maron CPT Week CERN, 23 April 2001 24
LNL
CMS
Plain and mesh topology limitsPlain and mesh topology limits
• Each couple of switches is connected by a single linkEach couple of switches is connected by a single link• This is a bottleneck if the traffic is not uniformly randomThis is a bottleneck if the traffic is not uniformly random• The network is blocking for certain traffic patternsThe network is blocking for certain traffic patterns
G. Maron CPT Week CERN, 23 April 2001 25
LNL
CMS
Traffic with patternsTraffic with patterns
• If traffic has patterns (for example this could happens in the case If traffic has patterns (for example this could happens in the case the event builder is performed in steps) it could make sense to the event builder is performed in steps) it could make sense to introduce an artificial mechanism that randomise the traffic. introduce an artificial mechanism that randomise the traffic.
• This mechanism exist and it is called Universal Routing This mechanism exist and it is called Universal Routing
G. Maron CPT Week CERN, 23 April 2001 26
LNL
CMS
Universal Routing ReferenceUniversal Routing Reference
• Discovered by L.G. Valiant in 1980Discovered by L.G. Valiant in 1980
• See: M.D May, P.W. Thompson, P.H. Welch See: M.D May, P.W. Thompson, P.H. Welch NETWORKS,ROUTERS & TRASPUTER available NETWORKS,ROUTERS & TRASPUTER available on : http://www.pact.srf.ac.uk/macrame/papers/bluebook.htmlon : http://www.pact.srf.ac.uk/macrame/papers/bluebook.html
• Those papers describe the Universal Routing applied to Those papers describe the Universal Routing applied to Transputers Networks, a wormhole routing based networkTransputers Networks, a wormhole routing based network
• We adapted the same concept to a packet switched network We adapted the same concept to a packet switched network like Gigabit Ethernetlike Gigabit Ethernet
G. Maron CPT Week CERN, 23 April 2001 27
LNL
CMS
Universal Routing with GigaEthernet Universal Routing with GigaEthernet
• Based on Clos topologyBased on Clos topology• Multiple path available Multiple path available
between each couple of between each couple of switchesswitches
• Every packets is sent to Every packets is sent to a randomly chosen a randomly chosen intermediate switchintermediate switch
• The intermediate The intermediate switch send the packet switch send the packet to the final destinationto the final destination
• Full bandwidth Full bandwidth between each between each couple of switches couple of switches and uniform buffer and uniform buffer utilizationutilization
G. Maron CPT Week CERN, 23 April 2001 28
LNL
CMS
Universal RoutingUniversal Routing
• Transformation of the CLOS topology to a folded CLOS Transformation of the CLOS topology to a folded CLOS • The resulting number of ports is the same of the plain topologyThe resulting number of ports is the same of the plain topology
Half duplex links Full duplex links
RUs
BUs
RUs
BUs
G. Maron CPT Week CERN, 23 April 2001 29
LNL
CMS
Large (500x500) multistage GE network (1)Large (500x500) multistage GE network (1)
1111
22
2525
33
22
20 BUs20 BUs
2020
44
20 RUs20 RUs
20 Ports20 Ports 25 Ports25 Ports
40 Ports40 Ports
• 25 switches with 25 switches with 60 x 1Gb ports 60 x 1Gb ports
• 20 switches with 20 switches with 25 x 1 Gb ports 25 x 1 Gb ports
G. Maron CPT Week CERN, 23 April 2001 30
LNL
CMS
Large (500x500) multistage GE network (2)Large (500x500) multistage GE network (2)
• 25 switches with 25 switches with 40 x 1Gb ports + 40 x 1Gb ports + 2 x 10 Gb uplinks 2 x 10 Gb uplinks
• 2 switches with 2 switches with 25 x 10 Gb ports 25 x 10 Gb ports
11
11
22
2525
22
20 BUs
20 RUs
2 Ports 10G2 Ports 10G
25 Ports 10G25 Ports 10G40 Ports
G. Maron CPT Week CERN, 23 April 2001 31
LNL
CMS
Proposal for a multistage event builder demonstratorProposal for a multistage event builder demonstrator
• Multistage event builders can be emulated using the much Multistage event builders can be emulated using the much cheaper fast ethernet connections and switches. The GE speed cheaper fast ethernet connections and switches. The GE speed is not needed in these topological investigationsis not needed in these topological investigations
• The proposal is to have prototypes for:The proposal is to have prototypes for:– Full Mesh TopologyFull Mesh Topology
– Folded CLOS topology with (and without) Universal Routing Folded CLOS topology with (and without) Universal Routing mechanism mechanism
G. Maron CPT Week CERN, 23 April 2001 32
LNL
CMS
Full Mesh 64x64 Event Builder PrototypeFull Mesh 64x64 Event Builder Prototype
11
55
77
33
22
66
88
44
8 RUs8 RUs 8 BUs8 BUs- 1 host node 4 Rus / 4 BUs or - 1 host node 4 Rus / 4 BUs or a mix of thema mix of them- 32 hosts- 32 hosts- 128 FE NICS (56 + - 128 FE NICS (56 + 7272))- - 8 24 FE ports switch8 24 FE ports switch
Missing componentsMissing components
G. Maron CPT Week CERN, 23 April 2001 33
LNL
CMS
Folded CLOS 64x64 Event Builder PrototypeFolded CLOS 64x64 Event Builder Prototype
1111
22
44
33
22
16 BUs16 BUs
1616
44
16 RUs16 RUs
16 Ports16 Ports 4 Ports4 Ports
32 Ports32 Ports
33
FastIron with 3 24 FE ports modsFastIron with 3 24 FE ports mods
- 1 host node 4 Rus / 4 BUs or - 1 host node 4 Rus / 4 BUs or a mix of thema mix of them- 32 hosts- 32 hosts- 128 FE NICS (56 + - 128 FE NICS (56 + 7272))- - 4 48 FE ports switch4 48 FE ports switch- 2 24 FE ports FastIron module- 2 24 FE ports FastIron module+ 1 24 FE ports FastIron module+ 1 24 FE ports FastIron module
Missing componentsMissing components
- 1 host node 3 Rus / 3 BUs or - 1 host node 3 Rus / 3 BUs or a mix of thema mix of them- 32 hosts- 32 hosts- 96 FE NICS (56 + - 96 FE NICS (56 + 4040))- - 4 36 FE ports switch4 36 FE ports switch- 2 24 FE ports FastIron module- 2 24 FE ports FastIron module
64x6464x64
48x4848x48
G. Maron CPT Week CERN, 23 April 2001 34
LNL
CMS
Folded CLOS 80x80 Event Builder PrototypeFolded CLOS 80x80 Event Builder Prototype
11
11
22
44
20 BUs20 BUs
20 RUs20 RUs
2 GE Ports2 GE Ports 4 Ports4 Ports
40 FE40 FEPortsPorts
33
FastIron with 8 GE ports moduleFastIron with 8 GE ports module(1000 BaseT or (1000 BaseT or 1000 BaseSX1000 BaseSX))
- 1 host node 4 Rus / 4 BUs or - 1 host node 4 Rus / 4 BUs or a mix of thema mix of them- 40 hosts (32+- 40 hosts (32+88))- 160 FE NICS (56 +- 160 FE NICS (56 +104104))- - 4 48 FE ports + 2 GE links switch4 48 FE ports + 2 GE links switch- 1 8 GE (- 1 8 GE (Base SXBase SX) ports FastIron ) ports FastIron modulemodule
Missing componentsMissing components80x8080x80
22
G. Maron CPT Week CERN, 23 April 2001 35
LNL
CMS
8x80 Conic Event Builder Prototype8x80 Conic Event Builder Prototype
11
22
44
20 FUs20 FUs
2 GE Ports2 GE Ports
20 FE20 FEPortsPorts
33
RU1RU1
RU2RU2
RU3RU3
RU4RU4
RU5RU5
RU6RU6
RU7RU7
RU8RU8
FastIron with 2 8 GE ports modulesFastIron with 2 8 GE ports modules(1000 BaseT or (1000 BaseT or 1000 BaseSX1000 BaseSX))
- 1 host node 4 FUs - 1 host node 4 FUs - 20 hosts - 20 hosts - 80 FE NICS (56 + - 80 FE NICS (56 + 2424))- - 4 24 (48) FE ports + 2 GE up 4 24 (48) FE ports + 2 GE up links switchlinks switch- 2 8 GE (- 2 8 GE (Base SXBase SX) ports FastIron ) ports FastIron modulemodule
Missing componentsMissing components8x808x80
G. Maron CPT Week CERN, 23 April 2001 36
LNL
CMS
Material for the event builder multistage prototypesMaterial for the event builder multistage prototypes
Mesh 64x64Mesh 64x64- 72 FE NICs- 72 FE NICs- - 8 24 FE ports8 24 FE ports
Folded CLOS 80x80Folded CLOS 80x80- 8 PCs- 8 PCs- 104 FE NICS- 104 FE NICS- - 4 48 FE ports with 2 GE uplinks 1000 baseT4 48 FE ports with 2 GE uplinks 1000 baseT
if the 1000 baseT uplinks are not available:if the 1000 baseT uplinks are not available:
1) Folded CLOS 64x641) Folded CLOS 64x64::72 FE NICs72 FE NICs4 48 FE ports switch4 48 FE ports switch1 24 FE ports FastIron module1 24 FE ports FastIron module
2) Folded CLOS 48x482) Folded CLOS 48x48::40 FE NICs40 FE NICs4 >36 FE ports switch4 >36 FE ports switch