network monitoring in datagrid project · 1st scampi workshop – 27 jan. 2003 – datagrid network...
TRANSCRIPT
![Page 1: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/1.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring
Network monitoring in DataGRID project
Franck Bonnassieux (CNRS)[email protected]
![Page 2: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/2.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 2
Outline
DataGRID network
Specificity of Grid network monitoring
Network metrics and measures
Architecture of DataGRID network monitoring
Network sensors and tools
High level tools
Perspectives
![Page 3: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/3.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 3
DataGRID http://www.eu-datagrid.org/
• 7 applications distributed among 6 virtual organisations• 11 organisations over 15 countries• 40 sites in Europe• 12 work packages • WP7 : network work package
– provisioning of infrastructure – Network and Transport Services – Network and Grid traffic monitoring – Grid Security
![Page 4: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/4.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 4
SuperJANET4
NIKHEF
RAL
SURFnetEuropean Topology : Geant, NRNs, Sites
CERN
![Page 5: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/5.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 5
Why network performances monitoring in grid environments ?
For Provisioning:To be available, via visualization to human observer (user, network/system administrators)To provide tools for network performances measurement, problems identification and resolution (bottlenecks, point of unreliability, quality of service needs, topology…)To achieve network performance forecast and optimization –Capacity planning
For Resource Brokers:Network performance parameters are used for optimizing resource allocation (replication, Remote file access…)
![Page 6: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/6.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 6
“Physical” view of the Network
Public NetworkNo securityNo predictable performancesNo control on the traffic
The flat INTERNET
Resource = CE (computing element) or Resource = SE(storage element)
![Page 7: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/7.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 7
“Logical” view of the Grid Network
![Page 8: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/8.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8
Measurement methods
Active methodsInjection of traffic inside the network for testing performances between two pointsproblem: may be intrusive (TCP/UDP throughput)
Passive methodsCollect of traffic information in one point of the network : router, switch, dedicated passive host, computing element…
Problem : give network usage, not capacity
![Page 9: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/9.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 9
Network metrics and tools
One Way Delay => Ripe Boxes
Round Trip Delay => PinGEr
Packet Loss => PinGEr
TCP throughput => IPerfEr
UDP throughput
Jitter => UDPMon
Routers traffic => NetLoad Agent
![Page 10: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/10.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 10
Ripe boxes
RIPE box or Surveyor
one-way delay + lossUDP protocol
• specialized boxes
• Measurement between 2 RIPE boxes
• GPS time synchronization
Time
GPS
RIPE 2RIPE 1
![Page 11: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/11.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 11
RIPE output entre UK et CERN
![Page 12: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/12.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 12
Network Monitoring Architecture
PCP
TopoGrid RTPL
Distributed Data Collector
Raw
IPerfEr UDPmon NetLoad AgentPingEr
Measure on each site
CollectAndStorage
Visualization
MapCenter
Replica Managers & resources brokersNetwork Managers
Forecaster
Processing
ArchiveGlobus MDSR-GMA
Globus MDSR-GMA
NetworkCost
![Page 13: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/13.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 13
PCP (Probe Coordination Protocol)
2
1
6
3
4
5
7
Scheduling of measuresby clique
![Page 14: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/14.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 14
PingEr
RTT
LOSS
![Page 15: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/15.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 15
IperfEr
TCP Throughput
![Page 16: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/16.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 16
UDPmon
UDP Throughput
Loss
![Page 17: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/17.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 17
NetLoad Agent
In/Out Traffic
![Page 18: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/18.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 18
Metrics stored in LDAP (Globus MDS)
• R-GMA and Globus MDS Producer for storage of network metrics• Storage and aggregation of all historical data in archives
![Page 19: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/19.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 19
rTPL
• High-level performance presentation interface• Built-on-demand network metrics matrixes• Customizable views for network administrators of all testbed sites
![Page 20: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/20.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 20
TopoGRID http://ccwp7.in2p3.fr/topogrid
• Automatic discovery of routers and end nodes• Identification of routes and bottlenecks• Clickable Java Applet for in deep analysis of network paths
![Page 21: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/21.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 21
MapCenter http://ccwp7.in2p3.fr/mapcenter
•• Open and Flexible tool to Open and Flexible tool to visualize in realvisualize in real--time Grid time Grid status and Network status and Network performancesperformances• Automatic discovery and positioning of resources in graphical maps• Advanced stealth monitoring techniques (ICMP, TCP, UDP, HTTP, LDAP)
![Page 22: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/22.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 22
NetworkCost functions
13,084,046,534,5CNAF
7,086,2410,385,03IN2P3
2,6611,863,2511,13NIKHEF
4,357,122,447,46RAL
35,4444,8777,7846,75 CERN
CNAFIN2P3NIKHEFRALCERN
CERNRALNIKHEFIN2P3CNAF
CERNRALNIKHEFIN2P3CNAF
getNetworkCost
FileSize = 10 MBResults = time to transfer (sec.)
![Page 23: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/23.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 23
NetworkCost model
The current cost model is designed for data intensive computing and especially large files transfers
The most relevant metric for that cost model is available throughput
ImplementationIperf Measurements (current)
GridFTP Logs (future)
Other metrics (future) : UDP, RTT, Jitter, ...
Synchronisation (PCP)
![Page 24: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/24.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 24
MapCenter – Network Monitoring Page
![Page 25: Network monitoring in DataGRID project · 1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 8 Measurement methods Active methods Injection of traffic inside the](https://reader036.vdocuments.site/reader036/viewer/2022071010/5fc7d764f1d03a755141ef53/html5/thumbnails/25.jpg)
1st SCAMPI Workshop – 27 Jan. 2003 – DataGRID Network Monitoring - 25
Perspectives
GRID -> SCAMPITestbed infrastructure
Huge traffic generation on gigabits links
End-to-end metrics
…
SCAMPI -> GRIDTraffic analysis (protocol distribution)
GridFTP, Remote file access, …
Grid traffic measurement (per protocol or/and per address domains)
…