prediction experience and new model - zabbix...agenda 2 brief history of prediction in zabbix...
TRANSCRIPT
![Page 1: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/1.jpg)
Prediction Experience and New Model
Serg Mescheryakov, D.Sc., Professor
St. Petersburg Polytechnic University, Russia
Genesys Telecommunications Laboratories, USA
Dmitry Shchemelinin, Ph.D.
RingCentral, USA
Kristina Kucherova
Distillery, USA
![Page 2: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/2.jpg)
Agenda 2
Brief history of prediction in Zabbix
![Page 3: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/3.jpg)
Agenda 2
Brief history of prediction in Zabbix
Prediction experience – focus on cases of rapid data change
1 – Prediction of free disk space for database at Distillery
(Kristina Kucherova)
2 – Prediction of available memory at Genesys
(Serg Mescheryakov)
3 – Prediction of cyclic workload at Genesys and RingCentral
(Serg Mescheryakov and Dmitry Shchemelinin)
![Page 4: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/4.jpg)
Agenda 2
Brief history of prediction in Zabbix
Prediction experience – focus on cases of rapid data change
1 – Prediction of free disk space for database at Distillery
(Kristina Kucherova)
2 – Prediction of available memory at Genesys
(Serg Mescheryakov)
3 – Prediction of cyclic workload at Genesys and RingCentral
(Serg Mescheryakov and Dmitry Shchemelinin)
New data model for cyclic workload
(Kristina Kucherova and Serg Mescheryakov)
Examples of the anomalies detected in cyclic workload
![Page 5: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/5.jpg)
Agenda 2
Brief history of prediction in Zabbix
Prediction experience – focus on cases of rapid data change
1 – Prediction of free disk space for database at Distillery
(Kristina Kucherova)
2 – Prediction of available memory at Genesys
(Serg Mescheryakov)
3 – Prediction of cyclic workload at Genesys and RingCentral
(Serg Mescheryakov and Dmitry Shchemelinin)
New data model for cyclic workload
(Kristina Kucherova and Serg Mescheryakov)
Examples of the anomalies detected in cyclic workload
Recommendations and expectations
![Page 6: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/6.jpg)
History of Prediction in Zabbix 3
2014 – Prediction models
are presented to
Zabbix Conference
![Page 7: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/7.jpg)
History of Prediction in Zabbix 3
2014 – Prediction models
are presented to
Zabbix Conference
2015 – Prediction functionality
is implemented and
released in Zabbix 3.0
![Page 8: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/8.jpg)
History of Prediction in Zabbix 3
2014 – Prediction models
are presented to
Zabbix Conference
2015 – Prediction functionality
is implemented and
released in Zabbix 3.0
2015 – Zabbix is in top 5
of world popular
monitoring solutions
Monitoring Tools Deployed in 2015
![Page 9: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/9.jpg)
History of Prediction in Zabbix 3
2014 – Prediction models
are presented to
Zabbix Conference
2016 – Workshops on trend
prediction are run at
Zabbix Conference
2015 – Prediction functionality
is implemented and
released in Zabbix 3.0
2015 – Zabbix is in top 5
of world popular
monitoring solutions
2017 – Prediction experience and
new models are shared with
Zabbix Conference
Monitoring Tools Deployed in 2015
![Page 10: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/10.jpg)
Example 1: Prediction of Free Disk Space for Database 4
Problem –
Forecast is dropped dramatically
below critical threshold
Specifics of DB monitoring –
Auto-growth of DB extent is stepwise
(long time intervals with random jump)
![Page 11: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/11.jpg)
Example 1: Prediction of Free Disk Space for Database 5
Zabbix recommendation –
Look back period is long enough
when covers 5-10 extent stepsSolution –
Extend historical look back period
to make forecasting line smooth
![Page 12: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/12.jpg)
Example 2: Prediction of Available Memory 6
Metrics –
Available memory on host
running Java application
Prediction data model –
Linear
Look back history – 6h
Forecast horizon – 1h
Specifics –
Due to Java memory leak
the app is auto-restarted
by Zabbix trigger
Problem –
After Java service restarted,
the memory is restored and
the forecast is not reliable
Solution –
Ignore the history and
reset look back period
Problem –
Incorrect forecast after app restarted
Java memory leak
Auto-restart app
![Page 13: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/13.jpg)
Example 3: Prediction of Cyclic Workload 7
Business metrics of cyclic workload at Genesys:
SIP calls count
Number of web sessions
Number of processes
Etc.
Daily workload cycleDaily workload cycle
Problem –
Existing prediction models produce false alarms
on rapid data change in the end of working day
Specifics –
Each cycle has specific form, height, width, etc.
Specifics –
Each cycle has specific pattern, height, width, etc.
![Page 14: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/14.jpg)
Example 3: Prediction of Cyclic Workload 8
Business metrics of cyclic workload at RingCentral:
Calls connections count
Number of HTTP requests
Fax queue size
Daily workload cycle
Specifics –
Each cycle has specific pattern, height, width, etc.
![Page 15: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/15.jpg)
New Data Model for Cyclic Workload 9
The idea is to predict future workload based on the pattern of historical cycles
The goal is to detect potential anomaly and prevent capacity problems Anomaly
Repetitive pattern of a data series
(not single values like in Zabbix doc)
Normal cyclesNormal cycles
Parameters of a pattern
![Page 16: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/16.jpg)
Examples of the Anomalies Detected in Cyclic Workload 10
![Page 17: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/17.jpg)
Examples of the Anomalies Detected in Cyclic Workload 11
![Page 18: Prediction Experience and New Model - Zabbix...Agenda 2 Brief history of prediction in Zabbix Prediction experience –focus on cases of rapid data change 1 –Prediction of free disk](https://reader033.vdocuments.site/reader033/viewer/2022052800/5f10e3a57e708231d44b4c72/html5/thumbnails/18.jpg)
Recommendations and Expectations 12
Recommendations for prediction improvement:
If metric is changed stepwise, we need to extend historical look back period
to cover at least 5 extent steps (Zabbix doc says 5-10)
Ratio 5:1 between look back history and forecast horizon is recommended
for more accurate forecast (Zabbix doc says to start with 1:1)
After application is auto-restarted or the host is rebooted, we definitely need
to reset look back history, otherwise the forecast is no longer reliable
Expectations for future:
Allow using trends in the forecast calculation, at least for linear model
(currently the prediction is limited by history data and no trend data is used)
Implement new data model for prediction of cyclic processes and events
(currently is not supported in Zabbix)
Thank you!