1b. bylsma, cern csc meeting july 2008 g. williams, csc commissioning, aug. 19, 2008 g. williams b....
DESCRIPTION
3B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 Switches should be Seen and not Heard Mounting Evidence of Communication Problems debugging VMECC hangs point to switch problems August 7th Hard Reset Stability Test a VMECC loses communications in 4 out of 15 hard resets ~1:300 resets/VMECC Two Prong Approach to Fixing Problem Make Switch Information Readily Available 1) premanently cable information port 2) write xdaq GUI to ease information access 3) tune switch setup Try to Reproduce problems in Bench TestTRANSCRIPT
1B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
G. WilliamsB. BylsmaS. Durkin
The Ohio State University
Exorcizing sWitches PCrate Communications
2B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
Peripheral Crate Communications
Netgear GLS7212 Cheap or Inexpensive? $600/switch $50/fiber transciever
2 DLINK Gigabit ports -> 8 DLINK Switch Fanout -> 60 VME PCrate Controllers
Gigabit Switches are Complex DevicesLayer 2 Services
IEEE 802.1Q static VLAN (512) IEEE 802.1p Class of Service (CoS) IEEE 802.1D Spanning Tree Protocol IEEE 802.1v protocol VLAN & port VLAN IEEE 802.1w Rapid Spanning Tree IEEE 802.1s Multiple Spanning Tree IEEE 802.3ad Link Aggregation (LACP) IEEE 802.1x port access authentication IGMP v1, v2 snooping support Network storm protection including broadcast, multicast and unicast traffic Static multicast filtering Weighted round robin (WRR) query technology
Required Reading: Command Line Reference 284 pagesAdministrator Quide 148 pagesHardware Manual 40 pages
3B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
Switches should be Seen and not Heard
Mounting Evidence of Communication Problems• debugging VMECC hangs point to switch problems
•August 7th Hard Reset Stability Test a VMECC loses communications in 4 out of 15 hard resets ~1:300 resets/VMECC
Two Prong Approach to Fixing Problem
Make Switch Information Readily Available 1) premanently cable information port 2) write xdaq GUI to ease information access 3) tune switch setup
Try to Reproduce problems in Bench Test
4B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
Switch XDAQ GUI (G. Williams, S. Durkin)
• access information thru Switch telnets• perl scripts automate access• switch access relatively slow• proven access does not effect switch operation• can be accessed during running
5B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
6B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
7B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
Multicast Packets ReceivedBroadcast Packets ReceivedTotal Packets Received with MAC ErrorsFragments/Undersize ReceivedFCS ErrorsOverrunsTotal Received Packets Not Forwarded802.3x Pause Frames ReceivedUnacceptable Frame TypeVLAN Membership MismatchVLAN Viable DiscardsReceived Frames DroppedMulticast Packets TransmittedBroadcast Packets TransmittedTotal Transmit ErrorsFCS ErrorsTx OversizedUnderrun ErrorsTotal Transmit Packets DiscardedSingle Collision FramesMultiple Collision FramesExcessive Collision FramesPort Membership DiscardsVLAN Viable Discards802.3x Pause Frames Transmitted
Each Port Tallies many Errors Types
This web page will report ports with the following problems
8B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
Switch Bench Test (B. Bylsma, S. Durkin)
Test Setup1 switch, 1 Crate, 1 VMECC, 1CCB, 1 DMB/TMB, 1 LVMB
Hard Reset, Readback and Check400 LVMB voltages, …
Results
no switch: 18000 reset/readbacks ok
switch (auto-negotiate): 36 errors/15000 resets
switch (1000 full-duplex) 18000 reset/readback ok
Hard Reset give 1:300 port auto-negotiation problems
9B. Bylsma, CERN CSC Meeting July 2008G. Williams, CSC Commissioning, Aug. 19, 2008
Conclusion
Have greatly improved Switch configuration
Hopefully VMECC Crate Access problems are Reduced