a scalable network monitoring system as a public service on cloud
TRANSCRIPT
![Page 1: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/1.jpg)
A Scalable Network Monitoring System as a Public Service on Cloud
Chavee Issariyapat Network Technology Lab (NTL), NECTEC, THAILAND [email protected]
Network Technology Lab (NTL) NECTEC, THAILAND
![Page 2: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/2.jpg)
2
Network Monitoring System
o A system that constantly monitors network for failing components and notifies the network administrator in case of outages.
We have a problem!
![Page 3: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/3.jpg)
Network Monitoring System
• There are several open source NMS e.g. Nagios, Cacti, OpenNMS, GroundWork Monitor, MRTG
![Page 4: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/4.jpg)
Our first attempt : NetHAM
• A software that runs on a dedicated Linux server.
• The core engine is based on open source software.
• Provides a user-friendly web interface.
• Summarized the status of the network using graphic.
• Has been used by 100s of users including schools, libraries, hospitals, small enterprises.
![Page 5: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/5.jpg)
Lessons learned
o NetHAM still has limitations v Need a dedicated server.
may not worth for very small networks v Scaling limitation.
cannot scale out v Single point of failure.
no redundancy
![Page 6: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/6.jpg)
Basic Idea
INTERNET
![Page 7: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/7.jpg)
Basic Idea
INTERNET
!
Oh really!?
![Page 8: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/8.jpg)
Basic Idea
INTERNET
! ! ! ?
But If all the gateways are down, the monitoring server will no way be able to send out a notification.
![Page 9: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/9.jpg)
Basic Idea
INTERNETSo, in stead of this
![Page 10: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/10.jpg)
Network
Network
Network
Basic Idea
We do something like this..
![Page 11: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/11.jpg)
Network
Network
Network
Basic Idea
With this approach we will never miss the bad news.
Huh
Sir, you got to see this!! !
![Page 12: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/12.jpg)
Network
Network
Network
Basic Idea
This leads to a Cloud monitoring platform…
Huh
Sir, you got to see this!! !
![Page 13: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/13.jpg)
Our New approach
NetHAM nano • SaaS. No installation! • Extremely user-friendly • Highly scalable • Self-managed • Redundant / Fault tolerant • Zero downtime
![Page 14: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/14.jpg)
NetHAM nano cloud
NetHAM nano Cloud
SMS, Push
Notification
E-‐Mail
Control
Probe
API
i
![Page 15: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/15.jpg)
Inside of the cloud
• Scalable • Fault-tolerance • Self-managed
NetHAM nano Cloud
User
Data Store
Web Server Web Server
Push Server
Monitoring Agent Monitoring Agent
Agent Gateway
Monitoring Agent Monitoring Agent
Agent Gateway
Notification ServerSystem Manager
User
![Page 16: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/16.jpg)
Homogeneity of NetHAM nano cloud
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
Application
Core
All NetHAM nano nodes are exactly the same! NetHAM nano also works on a single node. Scale out by adding nodes!
![Page 17: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/17.jpg)
Homogeneity of NetHAM nano cloud
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
When pooling as a cluster, some nodes may have some functionalities disabled.
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
node A node B node C
![Page 18: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/18.jpg)
Failover : Agent Layer
• Failover mechanism is implemented on every layer.
1,2,4 5,7
3,6
Manager
Agent A
Agent B
Agent C
1 2 3 4 5 6 7
1,2,4,5 R U OK
?
3,6,7
Manager
Agent A
Agent B
Agent C
1 2 3 4 5 6 7
![Page 19: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/19.jpg)
Failover : Data Store Layer
• Database redundancy à no down time for the data store
Replica Set
Primary Secondary Secondary
Shard 1
Replica Set
Primary Secondary Secondary
Shard 3
Replica Set
Primary Primary Secondary
Shard 2
.. scale out by adding shard
Shard RouterShard Router Shard Router Shard Router
.. scale out by adding shard router
Web Server
![Page 20: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/20.jpg)
Failover : Node Layer
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App C
ore
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
![Page 21: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/21.jpg)
Failover : Node Layer
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App C
ore
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
![Page 22: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/22.jpg)
Failover : Node Layer
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App C
ore
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
![Page 23: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/23.jpg)
The current testbed
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
5 VMs at Bangkok-Thai Tower
2 VMs at CAT Bangrak
2 VMs at CAT Nonthaburi
1 PC at NTL 1 PC at OQC ( private IP )
1 Android Tablet mobile IP (assigned dynamically)
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
2 Raspberry Pi
Agent
Agent Gateway
Data Store
Manager
Messenger
Push Server
App
Core
![Page 24: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/24.jpg)
Self Monitoring
While monitors others, NetHAM nano is also monitoring itself! Self monitoring performs in an auto-configured & scalable & fault tolerant manner. Every node is monitored by at least one neighbor node. Like a usual host check, cross monitoring can be adapted dynamically according to a cloud manager.
![Page 25: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/25.jpg)
Possible NetHAM nano Nodes
Android Phones / Tablets
Embed Linux devices e.g. Raspberry Pi
Server / Desktop PC
Virtualization nodes e.g. Virtualbox, VMWare, Amazon EC2
![Page 26: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/26.jpg)
Platform-Specific plugins
• Server – Network service : PING, HTTP, FTP, SNMP, TCP/UDP, DNS – Device statistic : CPU, memory load – Network Interface : bytes in/out, operational status
• Home Agent – Environment : Temperature, Humidity – Network : wifi, devices identified by mac addresses
• Mobile – GPS coordinate – Accelerometer , Gyrometer , Proximity sensor , Camera
![Page 27: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/27.jpg)
NetHAM nano Front End
• SSL web integrate with Facebook Application Platform.
• Real-time status update via the push engine.
• Hassle-free user interface.
https://apps.facebook.com/nethamnano
![Page 28: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/28.jpg)
Front End
• Network topology with auto graph layout • Interactive chart component with scroll & zoom capability
![Page 29: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/29.jpg)
Some future use cases : Network monitoring system
• To monitor network devices such as switches, routers, servers. • To monitor remote service e.g. PING,HTTP,SNMP (more to be added) • Support distributed monitoring via multiple agents. • Suitable for small networks, small offices, websites etc.
![Page 30: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/30.jpg)
Some future use cases : Home applications
• NetHAM nano can be used to monitor home networks via usual ADSL modems.
• By doing a port forwarding, NetHAM nano could answer these questions :
Will be even more exciting if we have a home agent!
![Page 31: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/31.jpg)
Some future use cases : Smart energy
• By using a smart meter, it is possible to monitor home electricity consumption the same way as Google Power meter does.
• Monthly electric bill can be predicted. • Alerts when there’s anything wrong
with the electric usage patterns.
![Page 32: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/32.jpg)
Some future use cases : Mobile applications
• By imitating the standard NetHAM nano agent, we can have a compact agent run on mobile phones and tablets.
• In stead of checking the remote services, a compact agent checks their sensors.
• NetHAM nano can track device’s position, speed, acceleration etc and display on the current UI with a map view.
Somchai
Preecha
![Page 33: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/33.jpg)
Some future use cases : Information monitoring
• NetHAM nano can gather information from several sources and notify users according to their current locations.
NetHAM nano • “It’s going to rain in half an hour!” • “Leave soon if you don’t want to
be caught in a traffic jam.” • “Diesel price increases tomorrow
at 5:00AM.”
![Page 34: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/34.jpg)
Project Status
• NetHAM nano project is under development. • We are looking for places to deploy nodes to support
multiple-point monitoring from different regions/countries. • We are looking for innovative applications e.g. home
monitoring, location-based service, specific information retrieval for end users e.g. weather alert, traffic alert, disaster report etc.
• Contact us at : [email protected]
![Page 35: A Scalable Network Monitoring System as a Public Service on Cloud](https://reader031.vdocuments.site/reader031/viewer/2022030322/5883f7661a28ab0b1a8bf4ea/html5/thumbnails/35.jpg)
Thank you