federator.ai data collection methods and functionalities · virtual san (vsan) is a hyper-converged...
Post on 22-May-2020
10 Views
Preview:
TRANSCRIPT
© 2017 ProphetStor Data Services, Inc. 0
Federator.ai® Data Collection Methods and Functionalities
1 © 2018 ProphetStor Data Services, Inc. All Rights Reserved
Federator.ai® Data Collection Methods and Functionalities
Federator.ai® Data Collection Methods and Functionalities
Contents
Introduction ...................................................................................................................................... 2
Federator.ai® overview ..................................................................................................................... 2
Different types of data collection ....................................................................................................... 2
Metadata ................................................................................................................................................................ 2
Performance metrics .............................................................................................................................................. 2
Events ..................................................................................................................................................................... 2
Logs ........................................................................................................................................................................ 2
S.M.A.R.T. ............................................................................................................................................................... 2
Collecting data from multiple layers .................................................................................................. 2
Application layer .................................................................................................................................................... 2
Virtualization layer ................................................................................................................................................. 3
Infrastructure layer ................................................................................................................................................ 3
Incomplete/missing data handling ......................................................................................................................... 5
Analytics of data collected ................................................................................................................. 5
Cross-layer correlation ........................................................................................................................................... 5
Performance anomaly detection ........................................................................................................................... 5
Performance prediction ......................................................................................................................................... 5
Disk health trending prediction .............................................................................................................................. 6
Memory error detection/prediction ...................................................................................................................... 6
Log/event noise reduction ..................................................................................................................................... 6
2 © 2018 ProphetStor Data Services, Inc. All Rights Reserved
Federator.ai® Data Collection Methods and Functionalities
Introduction
In the big data era, IT operations are the de facto
keys to augment business growth, and machine data
plays a crucial role in IT operation efficiency. This
document describes what and how data is collected
and represented for IT operations using
Federator.ai®.
Federator.ai® overview
ProphetStor's Federator.ai® is an Artificial
Intelligence for IT Operations (AIOps) platform
providing insights and foresights by collecting
machine data from application, virtualization, and
infrastructure layers. It uses state-of-the-art artificial
intelligence (AI) technology to bring insights into
data centers and lets enterprises make better
decisions from the foresights it provides.
Different types of data collection
Federator.ai® uses the agents installed in different
environments and platforms to collect data. The
different types of data collection include the
following:
Metadata
Metadata are data that provide descriptions of and
information about data sources and may indicate the
relationships among different IT devices
Performance metrics
Performance metrics describe the loading and
utilization of system resources, including availability,
response time, latency, throughput, service time,
completion time, power consumption, processing
speed and so on.
Events
An "event" is the description of a user's actions or
system occurrences detected by software. Events are
triggered when an operation runs, or the system
processes meet its' criteria. They usually identify
operational anomalies.
Logs
Logs record system event occurrences, application
processes, service transitions, and the
communication information between different
software and systems.
S.M.A.R.T.
S.M.A.R.T. (or SMART) stands for Self-Monitoring,
Analysis, and Reporting Technology and is a
monitoring mechanism included in hard drives, SSDs,
and NVMes. SMART data indicates the various
attributes representing the health state of a given
hard drive and is used by Federator.ai® as the key
parameter set for predicting imminent disk failure.
Collecting data from multiple layers
Federator.ai® supports cross-layer data collection
from application, virtualization, and infrastructure
layers. Federator.ai® uses agents installed on the
monitored platforms from different layers to collect
data and sends the data to the Federator.ai® server
for data analysis, monitoring and prediction. The
following describes the details of supported systems
from different layers and how the data is collected.
Application layer
APM
Federator.ai® seamlessly integrates other third-party
Application Performance Monitoring & Management
(APM) tools including AppDynamics, New Relic, and
BMC. The following shows the collected data types
and collection methods.
Collected data types: Metadata, events, and
logs.
Data collection: Install a Federator.ai® agent to
communicate with third-party APM's APIs to get
metadata and events.
Log data: An agent installation is required on
each monitored node to collect log data.
3 © 2018 ProphetStor Data Services, Inc. All Rights Reserved
Federator.ai® Data Collection Methods and Functionalities
Virtualization layer
Containers
Federator.ai® supports container-base virtualization
including Kubernetes and OpenShift. The following
describes the collected data types and collection
methods.
Collected data types: Metadata, performance
metrics, events, and logs.
Data collection: Install a Federator.ai® agent to
communicate with Kubernetes and OpenShift
APIs to get metadata, performance metrics, and
events.
Log data: An agent installation is required on
each monitored node to collect log data.
Hypervisor
Federator.ai® supports VMware hypervisor, and
utilizes VMare’s vCenter, which is a unified platform
that manages the vSphere environment. The
following describes the collected data types and
collection methods.
Collected data types: Metadata, performance
metrics, events, logs, and S.M.A.R.T.
Data collection: Install a Federator.ai® agent to
communicate with a vCenter via its API to get
the metadata, performance metrics, events, and
logs.
S.M.A.R.T. data: Each ESXi host needs to be
installed with smartmontools (SMART
Monitoring Tools), which collects SMART data
from disks. Federator.ai® agent communicates
with smartmontools via ESXi’s SSH. Thus, the
SSH access needs to be enabled on each ESXi
host.
HCI
Hyper-converged infrastructure (HCI) is the
convergence of a hypervisor, server, and storage into
a single system. Nutanix is software-based HCI that
runs cross-virtual hypervisors on one platform. The
following describes the collected data types and
collection methods.
Collected data types: Metadata, performance
metrics, events, and S.M.A.R.T.
Data collection: Install a Federator.ai® agent to
communicate with Nutanix CVM API and get the
metadata, performance metrics, and events.
S.M.A.R.T. data: Except for Federator.ai® agent,
smartmontools needs to be installed on each
host.
Infrastructure layer
The infrastructure layer includes bare metal servers
and storage systems.
Linux/Windows hosts
Federator.ai® supports OS-based infrastructures
including Linux and Windows. The following
describes the collected data types and collection
methods.
Collected data types: Metadata, performance
metrics, logs, and S.M.A.R.T.
Data collection: Install one agent on each
monitored host to collect metadata,
performance metrics, and logs.
S.M.A.R.T. data: Except for Federator.ai® agent,
smartmontools needs to be installed on each
host.
vSAN
Virtual SAN (vSAN) is a hyper-converged software-
defined storage infrastructure developed by
VMware. VMware vSAN abstracts physical storage
into virtual pools and unifies resources under its
policy-based management. The following describes
the collected data types and collection methods.
Collected data types: Metadata, performance
metrics, events, and logs.
Data collection: Install a Federator.ai® agent to
communicate with vCenter’s API to get
metadata, performance metrics, events, and logs.
4 © 2018 ProphetStor Data Services, Inc. All Rights Reserved
Federator.ai® Data Collection Methods and Functionalities
Ceph
Ceph is a distributed storage system designed to run
on commodity hardware and provides highly
scalable object-, block- and file-level storage. The
following describes the collected data types and
collection methods.
Collected data types: Metadata, performance
metrics, and S.M.A.R.T.
Data collection: Install Ceph built-in
diskprediction plugin or Federator.ai® agent to
collect data from Ceph clusters.
The following table shows the overview of data collection from different layers and platforms.
Layer Category Platform/System Collected data types How data is collected
Applications APM AppDynamics
New Relic
BMC
Metadata
Events
An agent is installed to connect to third- party APM API.
Logs An agent needs to be installed on each monitored node to collect logs.
Virtualization Containers Kubernetes
OpenShift
Metadata
Performance Metrics
Events
An agent is installed to connect to the container API.
Logs An agent needs to be installed on each monitored node to collect logs.
Hypervisor VMware vSphere Metadata
Performance Metrics
Events
Logs
An agent is installed to connect to a vCenter via API.
S.M.A.R.T. Each ESXi host needs to be
installed with
smartmontools.
SSH needs to be enabled on
each ESXi host.
An agent is installed to
connect to smartmontools
via SSH.
HCI Nutanix Metadata
Performance Metrics
Events
An agent is installed to connect to Nutanix CVM API.
S.M.A.R.T. Each host needs to be
installed with
smartmontools.
Each host needs to be
installed with an agent.
5 © 2018 ProphetStor Data Services, Inc. All Rights Reserved
Federator.ai® Data Collection Methods and Functionalities
Layer Category Platform/System Collected data types How data is collected
Infrastructure OS Linux/Windows Metadata
Performance Metrics
Logs
Each host needs to be
installed with an agent.
S.M.A.R.T. Each host needs to be
installed with
smartmontools.
Each host needs to be
installed with an agent.
Storage vSAN Metadata
Performance Metrics
Events
Logs
An agent is installed to connect
to a vCenter via API.
Ceph Metadata
Performance Metrics
S.M.A.R.T.
Enable the Ceph built-in
diskprediction plugin or install
Federator.ai® agent to collect
data.
Incomplete/missing data handling
Incomplete/missing data refers to the situation when
the data sent from an agent has insufficient data or
some of the fields in the data do not exist.
Federator.ai® can still generate a prediction when
this situation occurs. However, the predicted results
cannot provide the same degree of accuracy as would
sufficient data. Federator.ai® uses a confidence value
(a five-star rating system) to indicate the quality of
the prediction results.
For example, when some of the S.M.A.R.T. data from a
disk do not exist, the corresponding confidence
values will have fewer stars.
Analytics of data collected
The following paragraphs describe how collected
data is used and how Federator.ai® brings out the
insightful information from this data to enhance IT
operation efficiency.
Cross-layer correlation
Federator.ai® analyzes metrics and resource
utilization across the application, virtualization, and
infrastructure layers. It also provides visualization of
the relationships between entities within each layer.
This cross-layer correlation helps IT administrators
proactively act on an entity that may be adversely
affected by another entity. The benefits of this feature
are to reduce MTTR (Mean Time To Repair) and
increase MTBF (Mean Time Between Failures).
Performance anomaly detection
Federator.ai® uses performance metrics to analyze
the performance use and resource utilization status
and then detect the anomalies. Performance anomaly
detection provides IT administrators with warnings on
unusual behavior that may adversely affect system
operations, allowing for a quick diagnosis of and
resolution for the problem.
Performance prediction
Using the state-of-the-art algorithms, Federator.ai®
predicts future performance trends and enables users
to optimize resource utilization.
6 © 2018 ProphetStor Data Services, Inc. All Rights Reserved
Federator.ai® Data Collection Methods and Functionalities
Disk health trending prediction
With the patented AI technology, Federator.ai® uses
S.M.A.R.T. and disk metadata for disk health trending
prediction. It also uses performance metrics of disks
and hosts as supplementary data to reinforce
prediction accuracy.
Memory error detection/prediction
Federator.ai® also applies AI to detect and predict
potential memory errors from events and logs of
physical hosts.
Log/event noise reduction
IT administrators typically have problems in
understanding system logs and events as many of
them are redundant messages. In addition, too much
redundant information may cause operation
inefficiency and waste resources. Federator.ai® can
filter out the log/event noise and extract the
significant information from logs and events to
benefit IT operations.
The table below shows how data is used in Federator.ai®.
Data sources
AI features Metadata
Performance metrics
Disk S.M.A.R.T. Log/event
Cross-layer correlation
Performance anomaly detection from baseline
Performance prediction
Disk health trending prediction
Memory error detection/prediction
Log/event noise reduction
top related