infiniflux case study of firewall
TRANSCRIPT
InfiniFlux Case Study
www.infiniflux.com
Requirements from Clients Using Firewall
2
Clients request the firewall that provides high level of security performance and easy log analysis and reporting.
Why does the search for logs takes long in the firewall?
Why the dashboard of the firewall has only simple features?
It requires various statistical information for preparing reports based on logs, but…
Is it necessary to buy external solutions for log analysis and reporting?
Looking for more efficient and better ways to use firewall!
Hardware Environment
3
Unlike conventional servers, it is impossible to expand hardware resources, and the firewall needs to divide limited resources into several daemons.
Hardware Performance
Hardware specifications are set when firewall was installed and cannot be expanded later.
Hardware Stability
When the firewall is installed, physical stability takes priority over the best performance.
Storage Space
Require to divide storage space efficiently as it is impossible to expand hardware resources.
• Need to divide limited resources due to the limitations on hardware resources
• Make the effective use of divided resources
• Challenge over data management issue due to lack of storage space
Requirements
4
Responding to the demands of the clients, it must operate smoothly and efficiently with limited resources.
• Provide quick search on stored logs
• Extract various statistical data over logs
• Able to analyze logs for the dashboard and reporting
• Operate with the limited resources smoothly
Existing Architecture for Storing Log Data
5
Store logs created from each service daemon as files, and separately store main statistical information as META data.
Store logs based on date as filesStore statiscal data
log
Service Daemon
Firewall IPS Web Filter VPN DLP
Log Daemon
log log log
META Table
Architecture for Using Existing Logs
6
Dashboard uses RRD, and search for reports and logs use META data
Create required data through complex process
META Table
RRD
log log log
Dashboard
Report
Search log
Improvement of Log Storage Architecture
7
Store logs into InfiniFlux and create and utilize META table
Log Table META Table
Service Daemon
Firewall IPS Web Filter VPN DLP
Log Daemon
Improvement of Log Usage Architecture
8
Dashboard uses RRD, and search for reports and logs use META data
Dashboard
Report
Search log
Reporting tool
META Table
RRD
Log Table
Configure System Resource Settings
9
InfiniFlux is required to use limited resources in order to maintain the primary purpose of the firewall, security features.
• Using environment configuration file of InfiniFlux, “iflux.conf”, to control CPU and Memory usage- CPU_COUNT : specify the number of CPU for use- CPU_AFFINITY_BEGIN_ID : operate specified CPU ID first- PROCESS_MAX_SIZE : the size of memory for available to use
• InfiniFlux stops storing data when the storage space is full, and re-store the data when there is available space in order to prevent the DB stoppage due to insufficient storage space.
Methods for Storing Logs
10
Log daemon receives logs generated from the service daemon and creates thread for storing logs. Then, threads store logs into log tables based on the type.
Regularly create and store data
Log Daemon
Thread 1 Thread 2 Thread 3 Thread 4 Thread 5
Log1 Table Log2 Table META1 Table META2 Table META3 Table
Service Daemon
Firewall IPS Web Filter VPN DLP
Meta data processing
Fast Response
11
MinMax cache is used for storing data based on the type.Adjust the size of HashBucket to speed up query response
• With MinMax Cache, costs for unnecessary file scan can greatly be reduced by checking minimum & maximum values of data.- Data should be stored in sequence.- High level of data dispersion- As a result, clients able to specified minimum and maximum values to each column when table was
created.
• InfiniFlux creates and uses "Hash table" to run statements including GROUP BY, DISTINCT or COUNT (Distinct Column). When the result values are larger than the size of "Hash table", it slows down the query speed, thus, proper size should be allocated to "Hash table“.- It can be set in the Iflux.conf, but requires to use the memory which is set to HashBucket every time a
new session is established.- The size of hash table can be set by using “ALTER SESSION SET hash_bucket_size” if necessary and the
default will be used whenever a new session is established.- The default values is 20011, and can be set up to 100MB.
Backup
12
For the firewall, difficult to use backup & restore features of conventional DB due to its characteristics. Thus, back up data based on date and use "Mount"
feature that is suitable for firewalls.
DB
Send backup image
Table1 Table2 Table3 Table4 Table5
date1 date1 date1 date1 date1
date2 date2 date2 date2 date2
Backup targets based on date
date3 date3 date3 date3 date3
date2 date2 date2 date2 date2
Restore data
Backup file
Table1 Table2
Table3 Table4 Table5
Firewall External Storage Device
date2 date2
date2 date2 date2
Positive Effects
13
Increase the customer satisfaction by enabling the search for large volume of logs and providing reports on various items.
Enabling high performance data input and analysis with limited hardware environments
Able to search logs quickly by storing logs into DB rather than files
Create dashboard and reporting on various items based on diverse statistical information
Correlation analysis of security events based on the combination of various logs
Website : www.infiniflux.comEmail : [email protected]
The World's Fastest Time Series DBMS
for IoT and Big Data
InfiniFlux