ibm operations analytics log analysis · pdf filelogstash is required for both collecting the...

31
Windows Events Securely to Operations Analytics Log Analysis Send Windows Events Securely to IBM Operations Analytics Log Analysis Version 1.0 Luke Murphy ITSM - Hybrid Cloud Technical Consultant IBM Cloud

Upload: dangcong

Post on 07-Mar-2018

229 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

Send Windows Events Securely to IBM Operations Analytics Log Analysis

Version 1.0

Luke Murphy

ITSM - Hybrid Cloud Technical Consultant IBM Cloud

Page 2: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

Edition notice This white paper edition applies to IBM Operations Analytics Log Analysis and to all subsequent releases and modifications until otherwise indicated in new editions. Also reference the IBM Operations Analytics Development Community (https://developer.ibm.com/itoa/) for additional information on Predictive Insights.

Page 3: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

IBM Operations Analytics Log Analysis

iii

CONTENTS

Overview of the required steps ........................................................................................... 4

1. Install OpenJDK (Java 8) on the IOALA Server ............................................................. 5

2. Install IBM Operational Analytics – Log Analysis ......................................................... 6

3. Installing Logstash .................................................................................................. 8

4. SSL Certificate Configuration.................................................................................... 9

5. Install Winlogbeat on Windows client server(s) ........................................................ 12

6. Copy certificates to Windows clients ...................................................................... 13

7. Winlogbeat Configuration ..................................................................................... 14

8. Logstash Configuration ......................................................................................... 16

9. Creating and Publishing the data source for Log Analysis .......................................... 19

10. Testing the Application ......................................................................................... 25

Page 4: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

4

In this how-to tutorial, I will explain the deployment of IBM Operational Analytics – Log Analysis (IOALA), to be used as a centralized logging interface for a multi-server windows based environment. The content of this document shows you how to configure the Logstash component of the Log Analysis server to collect and visualize logs of your Windows systems in a centralized location, using a lightweight data shipping tool called Winlogbeat.

Below is a summary of all the components needed to execute this solution. • Red Hat Enterprise Linux 6.x Server • 1+ Windows 7/10 Servers • IBM Operational Analytics – Log Analysis 1.3.5 • Logstash 2.2.1 - This is the data processing pipeline that allows you to pull data from a wide

variety of sources • Winlogbeat 5.1 - This tool reads from one or more event logs using Windows APIs

Winlogbeat will be installed on a Windows 10 server (the client server). IBM Log Analysis and Logstash will be installed on Red Hat Enterprise Linux Server release 6 (the IOALA server).

Overview of the required steps To implement this solution, the following 10 steps will need to be completed.

• Install OpenJDK (Java 8) on the IOALA Server • Install IBM Operational Analytics – Log Analysis • Installing Logstash • SSL Certificate Configuration • Install Winlogbeat on Windows server(s) • Copy certificates to Windows clients • Winlogbeat Configuration • Logstash Configuration • Creating and Publishing the data source for Log Analysis • Testing the Application

Page 5: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

5

1. Install OpenJDK (Java 8) on the IOALA Server Java 8 is available to download through “yum” or “apt-get” tools. For this demo, ibm-yum will be used as the repository for downloading packages. Java 8 was installed using the following command:

After a successful installation, this version of java should automatically be set as the JRE used in your Linux environment. We can check that this is the case by implementing the following commands:

Once the correct version of Java is installed, typing java –version will show the version of Java being used by the OS: The command should return something like the following.

If the above is returned, you have successfully installed OpenJDK 1.8 and can move on to the install of IBM Operational Analytics – Log Analysis.

$ Ibm-yum.sh install java-1.8.0-openjdk-headless.x86_64

$ which java /usr/bin/java

$ ls -l /usr/bin/java lrwxrwxrwx 1 root root 22 Jan 25 11:51 /usr/bin/java -> /etc/alternatives/java $ ls -l /etc/alternatives/java lrwxrwxrwx 1 root root 46 Jan 25 11:51 /etc/alternatives/java -> /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

$ java -version openjdk version "1.8.0_111" OpenJDK Runtime Environment (build 1.8.0_111-b15) OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)

Page 6: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

6

2. Install IBM Operational Analytics – Log Analysis

Skip this step if you have already installed IOALA. Download the relevant software package for IOALA from Passport Advantage. If you are evaluating the solution, you can obtain the IOALA Entry Edition from Developer Works for IOA. Installation and pre-requisites for IOALA is extensively documented here. For a small pre-production system, we used a single monolithic server in our example with a user account “ioala” with the server pre-requisites configured.

- Save the download into an appropriate location: e.g. /opt/IBM/Install/

- You may need to extract the files into a location: e.g. /opt/IBM/Install/LA At the command prompt: cd into the above directory (cd /opt/IBM/Install/LA) and type ./install.sh which will start the install.

This will guide you through the process of installing IOALA with IBM Installation Manager. To install IOALA using the command line or silently, you can follow the instructions in the following link:

https://www.ibm.com/support/knowledgecenter/SSPFMY_1.3.5/com.ibm.scala.doc/install/iwa_install_oview_c.html

After “./install.sh” the UI in the popup will ask you to select the packages to install. Select both IBM installation Manager 1.8.2 and IBM Operations Analytics – Log Analysis 1.3.5. Then accept the license agreement and select the locations for the shared resources directory and the Installation manager directory. For example:

/opt/IBM/IM/IBMIMshared /opt/IBM/IM/InstallationManager/eclipse

Then select ‘create a new package group’ for IBM Operations Analytics and specify the installation directory. E.g.

/opt/IBM/LogAnalysis

$ cd /opt/IBM/Install/LA $ ./install.sh

Page 7: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

7

On the page that says ‘select features to install’, ensure that the “Apache Solr” feature is selected and click next On the next page, leave the default configuration for the IOALA ports and click next. Review the summary information and click install to complete the installation. The installation should only take a couple of minutes to complete and IOALA will start automatically if there are no errors with the installation. We can test the installation by navigating to the following URL:

https://<your_hostname_goes_here>:9987/Unity/ We should see a UI like the following

Once IOALA is installed and running on the Red Hat server, the next step is to install Logstash. Logstash will act as the central hub for storing and processing the incoming logs from the Windows clients. Logstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics – Log Analysis. Logstash and IOALA can run on the same server, or separate servers on the same network. For this workaround, the instructions provided are based on a single server/multiple client environment where IOALA and Logstash are deployed on the same machine.

Page 8: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

8

3. Installing Logstash Change directory to the remote_install_tool configuration directory inside the IOALA install directory and edit the “ssh-config.propeties” file as follows:

Where REMOTE_HOST is the hostname of the system where you are remote installing Logstash, PORT is which port the host system is configured to listen for SSH, USER & PASSWORD is the username and password credentials used to log into the host system.

After this configuration stage is completed, type the following commands to install Logstash on the target system.

After the ./install.sh command, you will be asked a number of questions:

• Enter the location where you want to install Logstash (e.g. /home/ioala/LogAnalysis) • Install EIF Receiver Instances? No (N) • Install LFA 6.3? No (N) • Install Logstash 2.2.1? Yes (Y)

Logstash will then be installed on the target server under the directory: “/<LogAnalysis_Home_Directory>/Logstash”

####################################################################################### # SSH Configurations for eif-remote-deployer-tool ####################################################################################### REMOTE_HOST=<hostname/FQDN> PORT=22 TIME_OUT=60000 USER=ioala #PASSWORD can be commented while using Public key based authentication PASSWORD=*****

$ cd /<LogAnalysis_Install_Directory>/remote_install_tool $ ./install.sh

$ cd /<LogAnalysis_Home_Directory>/remote_install_tool/config $ vi ssh-config.properties

Page 9: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

9

4. SSL Certificate Configuration Winlogbeat Logstash Certificate

Before we begin with the configuration of Logstash, we need to generate an SSL Certificate and key pair for the secure shipping of data from our Windows clients to Logstash, using Winlogbeat and Logstash. The certificate is used by Winlogbeat (on the Windows machine) to verify the identity of the Logstash server. Complete the following steps to generate the SSL certificate and key pair. Note: If you are using separate Logstash and IOALA servers, then you must complete this step on the Logstash server, because this is the server that will need to communicate with the Winlogbeat API on the Windows machines.

Note: these directories may already exist, if so, skip to the next command.

Here, the fully qualified domain name (FQDN) of the Logstash server is used to generate the SSL certificate. It is strongly recommended that you have this DNS setup with your private networking.

The above command will generate the certificate and key into the “certs” and “private” directories on the Logstash server. Before we copy the certificate to our windows clients, we need to import it into our Java Runtime Environment (OpenJDK) key-store. This is because this “JRE” is what Logstash uses to compile and run. We have already located where our “OpenJDK” version of “JRE” is installed. The following command will import the required certs into the JRE key-store. You will need to run this command from inside the “bin” directory of where your “Logstash server instance JRE” is installed. i.e, if you’ve installed the JRE in:

“/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el6_8.x86_64/jre”

$ sudo mkdir /etc/pki/tls/certs (directory for certificate) $ sudo mkdir /etc/pki/tls/private (directory for key)

$ cd /etc/pki/tls; sudo openssl req -subj '/CN=<LogstashServer_FQDN>/’ -x509 -days 3650 - batch -nodes -newkey rsa:2048 -keyout private/win-logstash.key -out certs/win-logstash.crt

Page 10: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

10

then you would type the following command before issuing the “keytool” command.

If prompted for a password for the key store, it will usually be “changeit”. You will be asked if you trust the certificate to be imported, type “yes” and the certificate will be imported into the keychain. The certificate we generated above will be copied to all of our Windows clients, imported to the Trusted root CA, and added to the Winlogbeat configuration in a later step. However, before that, we need to import another certificate into the same Java key-store on our Logstash server.

Logstash Log Analysis Certificate For our Logstash server to be able to identify and trust our IOALA server, we also need to import the IOALA certificate. After IOALA 1.3.5 was installed in the pre-requisites, a certificate would have been generated in the security folder. We need to import this certificate into our Java key store. Firstly, check to see if an older certificate already exists in the Logstash server instance JRE key-store:

After the command above, if you get an entry in the key-store for the selected alias (scala), you will most likely need to delete this certificate before you import the new one. We delete the existing certificate by issuing the following command:

$ keytool -import -file /etc/pki/tls/certs/win-logstash.crt -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el6_8.x86_64/jre/lib/security/cacerts

$ keytool -list -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el6_8.x86_64/jre/lib/security/cacerts -alias scala –storepass changeit

$ keytool -delete -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el6_8.x86_64/jre/lib/security/cacerts -storepass changeit -alias scala

## Example of changing directory to Logstash JRE ## $ cd /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.1111.b15.el6_8.x86_64/jre/bin

Page 11: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

11

Now we can import the new certificate into the key-store:

The new scala certificate should now be imported into the Java key-store. If we type the “keytool –list” command again, we should see something like the following:

$ keytool -import -file /<LogAnalysis_Home_Directory>/wlp/usr/servers/Unity/resources/security client.crt -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el6_8.x86_64/jre/lib/security/cacerts -storepass changeit

Page 12: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

12

5. Install Winlogbeat on Windows client server(s) Winlogbeat is an open-source product from Elasticsearch and can be downloaded from the following link https://www.elastic.co/products/beats/winlogbeat The version of Wingbeat used in this practice is 5.1.2. Once downloaded, unzip Winlogbeat into an appropriate folder on your Windows machine (e.g. “C:\Program Files\Winlogbeat”) 1. Extract the contents into C:\Program Files 2. Rename the winlogbeat-<version> directory to Winlogbeat 3. Open a PowerShell prompt as an Administrator (right-click on the PowerShell icon and select Run

As Administrator) 4. Then type the following commands to install Winlogbeat. When the security warning appears after

the second command, type ‘R’ to run the command.

• •

You should get something like the following when Winlogbeat is installed:

$ cd 'C:\Program Files\Winlogbeat' $ PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-winlogbeat.ps1

Page 13: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

13

6. Copy certificates to Windows clients For this step, I have used WinSCP, which allows us to remotely connect to our Linux server in order to collect and import the certificate that we generated earlier. Any other tool that allows an SSH connection to the Linux server may be used for this step, as long as the correct certificate is copied and imported onto all Windows clients. Once you have initiated an SSH remote connection to the Logstash server. Locate the certificate (in /etc/pki/tls/certs) and key (in /etc/pki/tls/private) that we created and copy these two files to any directory on our Windows client (I have saved these in “C:/winlogbeat” for ease of use). The certificate then needs to be imported into the Trusted Root CA on the Windows clients. To do this, follow the steps below.

1. Locate the “win-logstash.crt” on your windows machine 2. Double click the certificate and select install certificate 3. Choose “Local Machine” as the store location and click next 4. Click “Yes” to the user account control pop up 5. Select the option “Place all certificates in the following store”, click browse, select the “Trusted

Root Certification Authorities” and click “OK”. 6. Click next to review the settings you have chosen and then “Finish” to complete the import of

the certificate.

Page 14: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

14

7. Winlogbeat Configuration Now that we have installed Winlogbeat and imported the certificates for the enablement of secure data communication, we can configure Winlogbeat to ship logs to our Logstash server.

• Under your Winlogbeat directory, open up the file “Winlogbeat.yml” with an editor of your choice.

• In the “event_logs” section under “Winlogbeat Specific Options”, you can specify the event logs that you want to monitor. By default, Winlogbeat is set to monitor application, security, and system logs. For this demo, the types of logs to monitor have been left to default. Your configuration for this section should look like this:

• In the output section, we are going to use the Logstash output. Here, we specify the hostname of the Logstash server, the port on which we will (in a later step) configure Logstash to listen on, the SSL certificate authorities (this defaults to the Trusted Root CA when left blank, and also the location of the certificate and key pair used for SSL. The output config should look like this:

Page 15: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

15

• All other sections of the config file were left as default. Once we have set the configuration as above, we can test this by initiating the following command under the winlogbeat directory:

The Winlogbeat Configuration is now complete, and we can start this service by typing “Start-Service winlogbeat” in the PowerShell window. But, let’s not start anything until we finish all the configuration.

Page 16: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

16

8. Logstash Configuration Logstash Configuration files are in JSON format. In our case, we will be using the configuration files based on the SCALA output plugin, which are stored in the directory “/<LogAnalysis_Home_Directory>/Logstash/logstash-2.2.1/logstash-scala/logstash/config. There are 3 parts to a Logstash config file, which are the inputs, the filters and the outputs. The first step is to create a new config file called “windows.conf”. We do this by typing: “vi windows.conf”. • The Input configuration is:

This specifies a “beats” input configuration that will listen on port 15738 (i.e. the port we specified Winlogbeat to send it’s logs to). We also specify the SSL certificate and key pair that we created earlier. • The filter configuration is:

input { beats { port => 15738 type => wineventlog ssl => true ssl_certificate => "/etc/pki/tls/certs/win-logstash.crt" ssl_key => "/etc/pki/tls/private/win-logstash.key" } }

filter {

# extract the path fields to map physical data source attributes to master datasource if [type] == "wineventlog" { mutate { replace => [ "timestamp", "%{@timestamp}"] replace => [ "path", "winlogbeat"] replace => [ "host", "winlogbeat"] } # end mutate } # end if-type==was

Page 17: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

17

This filter looks for logs that are of type “wineventlog” and performs the filtering before the logs are output to Log Analysis. The 2nd and 3rd “replace” statements inside the mutate field above are simply mapping the host and pathname properties for the data source that will be created in step 9, this will allow successful ingestion of formatted data into IOALA. The “replace” timestamp statement is mapping the timestamp value that we receive from the Windows event logs, to the correct value that is used in the source type properties, also created in step 9. The output configuration on the next page does a number of things:

It firstly saves the command line logs to file (rubydebug.log).

It then transmits the events to the IBM Operational Analytics – Log Analysis Server (at https://<your_hostname_goes_here>:9987/Unity/DataCollector).

Here, you will need to specify your own Log Analysis server with its login credentials. In the section that says “scala_fields”, here is where we specify the target data source and properties of the data to be used with IOALA.

Lastly, it prints the output of the script to the command line using the “stdout” plugin. This simply enables us to see what is happening at the Logstash command line.

Note: Before we run this script, we need to create a properties file using the DSV toolkit, enter the values or fields (e.g. timestamp, host, message etc..) that we want our data to comply with, then publish this properties pack into Log Analysis. WE then create a new data source based on this properties pack in order to ingest the data that we are loading from Winlogbeat and Logstash. After we configure the Logstash config file with the specified inputs, filters & outputs, we MUST move to step 9 before running the Logstash script.

Page 18: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

18

• The output configuration is:

output { file { path => "./rubydebug.log" codec => rubydebug } scala { scala_url => "<hostname/fqdn>:9987/Unity/DataCollector" # Change the creds if the defaults are changed scala_user => "unityadmin" scala_password => "<your_LA_Password>" # You can either use encrypted password or use a plain text one. # If encrypted password is used, provide the keystore path below. # If plaintext password is used, just comment the below line for keystore path scala_keystore_path => "" batch_size => 500000 idle_flush_time => 5 sequential_flush => true num_concurrent_writers => 20 use_structured_api => false # Ensure you create a cache dir. And configure the path below - disk_cache_path => "./cache" scala_fields => { # An example below - "winlogbeat@winlogbeat" => "timestamp,log_name,level,source_name,computer_name,process_id,event_id,record_number,thread_id,message,logRecord" } date_format_string => "yyyy-MM-dd HH:mm:ss Z" log_file => "./scala-debug.log" log_level => "debug" } stdout { codec => rubydebug} }

Page 19: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

19

9. Creating and Publishing the data source for Log Analysis This section describes how to create the new source type and data source with the correct properties that will be used to ingest the data from Logstash. To do this, we will use the DSV Toolkit. For more information on the DSK Toolkit for IOALA, please follow the link below. https://www.ibm.com/support/knowledgecenter/SSPFMY_1.3.5/com.ibm.scala.doc/extend/iwa_extend_DSVovw.html Complete the following steps to create the Winlogbeat data source.

Create a new properties file called “winevents.properties” in this directory which includes all the fields that you need to organise your data. The file should look something like this: ** The properties file for Winlogbeat starts on the next page**

$ cd “/<LogAnalysis_Home_Directory>/unity_content/DSVToolkit_v1.1.0.4

Page 20: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

20

[SCALA_server] scalaHome: /opt/IBM/LogAnalysis [DSV_file] delimiter: , quoteChar: " moduleName: winlogbeat version: 1.0.0.0 [field0_indexConfig] name: logRecord dataType: TEXT retrievable: true retrieveByDefault: true sortable: false filterable: false searchable: true path_1: content.text combine: FIRST [field1_indexConfig] name: timestamp retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true dataType: DATE dateFormat: yyyy-MM-dd'T'HH:mm:ss.SSSX [field2_indexConfig] name: log_name dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true [field3_indexConfig] name: level dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true [field4_indexConfig] name: source_name dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true **File Continues on Next Page**

Page 21: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

21

**Continuation of File** [field5_indexConfig] name: computer_name dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true [field6_indexConfig] name: process_id dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true [field7_indexConfig] name: event_id dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true [field8_indexConfig] name: record_number dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true [field9_indexConfig] name: thread_id dataType: TEXT retrievable: true retrieveByDefault: true sortable: true filterable: true searchable: true [field10_indexConfig] name: message dataType: TEXT retrievable: false retrieveByDefault: false sortable: false filterable: false searchable: true

Page 22: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

22

Save the file and run the following command

-o: this overwrites what is already there (there shouldn’t be anything but good practise to put it in for future reference) -d: this deploys “winevents.properties” to Log Analysis.

The above command should successfully deploy the new insight pack (winlogbeat) into Log Analysis. If you do not get “Build Successful” after the operation, you will need to check the configuration, and make sure it does not contain any errors. To be able to ingest the data into Log Analysis, we need to create a data source on the IOALA UI and map it to the insight pack that we have just deployed. 1. Log into IOALA using the link below, as the administrator (unityadmin) and click

administrative settings on the top right hand side.

https://<your_hostname_goes_here>:9987/Unity/ (provided for illustration purposes) 2. Select the “Data Sources” tab and then press the “+” icon to create a new data source.

$ python dsvGen.py winevents.properties –o –d

Page 23: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

23

3. The following popup will appear. Follow the exact configuration in the screenshots to add the new data source.

Page 24: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

24

Click “Finish” and the data source will be configured. To confirm that the newly created data course does in fact map to the properties file that deployed earlier; we can click on “Data Types” in the administrative settings home page in LA, click “source types” down the right hand side, double click “winlogbeat” to edit the source type, and then click “view index configuration”. Here, we should see all the properties that we are to include for the events that we collect from our Windows clients (as below):

Page 25: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

25

10. Testing the Application Testing that the application works will consist of three main tasks. The first task will be to start the Winlogbeat service on our Windows clients, in order to initialize the shipping of logs over TCP to our Logstash Server. The second task is to start the Logstash service on our Linux server and ensure that there are no errors at the standard output. Finally, we will navigate to the browser where Log Analysis is running and we should be able to visualize the incoming logs from our windows clients. Follow the steps below to complete the testing of the application. Starting the Winlogbeat service on the Windows client(s)

We can confirm that the Winlogbeat is running by checking the services desktop app for winlogbeat. It should say “Running”, as below:

Once we have started Winlogbeat, we can start the Logstash script by issuing the following commands on the Logstash server.

$ cd /opt/IBM/LA/IOALA/Logstash/logstash-2.2.1/logstash-scala/ $ ../bin/logstash agent -f logstash/config/windows.conf

Page 26: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

26

At the output, you should see “Logstash startup completed” at the standard output of the command line. After some time, you will see the outgoing logs from the Windows clients appear at the standard output of the Logstash script. The standard output will look something like this:

We have successfully shipped the logs securely from our Windows clients to Logstash on our Log Analysis server. Now, the last thing we need to check is if the logs were successfully published to Log Analysis.

Visualizing the logs from our Windows clients on the Log Analysis server

Log into the Log Analysis server and select “New Search”, and follow the steps below to search for:

*: this searches for everything Last Hour: Searches all incoming events in the past hour “winlogbeat”: Searches events from the winlogbeat data-source

Page 27: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

27

Then press search: you should see all the logs from the windows machines appear under the search bar, as below.

Once we have the logs continuously coming onto Log Analysis, we can perform multiple methods of Analysis on these logs. For example, I have plotted a simple line chart from a certain chunk of the data. The data shown below is the data that “timestamp” ranges from 3:00pm on 30/01/2017 to 00:00 on 31/01/2017. Please note that Log Analysis uses the American time format.

Page 28: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

28

The graph shows us how many events are being populated each hour in the specified time range.

So if you get to the final piece above, you have successfully implemented a solution to gather all types of logs from Windows servers (Winlogbeat), store and process them with a centralized logging tool (Logstash) and visualize the results on IBM Operational Analytics – Log Analysis.

Page 29: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

29

® © Copyright IBM Corporation 2017 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A.

For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan, Ltd. 1623-14, Shimotsuruma, Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation 2Z4A/101 11400 Burnet Road Austin, TX 78758 U.S.A.

Page 30: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

30

Such information may be available, subject to appropriate terms and conditions, including in some cases payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. If you are viewing this information in softcopy form, the photographs and color illustrations might not be displayed. Copyright License This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not be liable for any damages arising out of your use of the sample programs. Privacy Policy Considerations IBM Software products, including software as service solutions, (“Software Offerings”) may use cookies or other technologies to collect product usage information, to help improve the end user experience, to tailor interactions with the end user, or for other purposes. In many cases, no personally identifiable information is collected by the Software Offerings. Some of our Software Offerings can help enable you to collect personally identifiable information. If this Software Offering uses cookies to collect personally identifiable information, specific information about this offering’s use of cookies is set forth below. Depending on the configuration that is deployed in the application server, this Software Offering may use session cookies for session management. These cookies can optionally be enabled or disabled, but disabling the cookies will also disable the functionality that they enable. See the application server documentation for additional information. Optionally, you can also use functionality, such as typeahead, to store data locally on the browser. The data that is stored locally could contain personally identifiable information: Data validation. If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies, you should seek your own legal advice about any laws that are applicable to such data collection, including any requirements for notice and consent. For more information about the use of various technologies, including cookies, for these purposes, see IBM’s Privacy Policy at http://www.ibm.com/privacy and IBM's Online Privacy Statement at http://www.ibm.com/privacy/details in the section entitled “Cookies, Web Beacons and Other Technologies” and the "IBM Software Products and Software-as-a-Service Privacy Statement" at http://www.ibm.com/software/info/product-privacy.

Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.

Page 31: IBM Operations Analytics Log Analysis · PDF fileLogstash is required for both collecting the logs from the Windows clients, and also outputting the results to IBM Operational Analytics

Windows Events Securely to Operations Analytics Log Analysis

31

Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and service names might be trademarks of IBM or other companies.