hortonworks sqrrl webinar v5.pptx

© Hortonworks Inc. 2014

Cyber Pattern Discovery using Linked Data Analysis

November 12, 2014

Webinar with Hortonworks and Sqrrl


Webinar with Hortonworks and Sqrrl Cyber Pattern Discovery using Linked Data Analysis November 12, 2014

Joe Travaglini Director of Products Sqrrl

John Kreisa VP Strategic Marketing Hortonworks


The Modern Data Architecture Hortonworks. We do Hadoop.

John Kreisa, VP Strategic Marketing Hortonworks


Agenda

•  Apache Hadoop and a Modern Data Architecture •  Security in a comprehensive data management platform

•  Security Analytics using (Big) Cybersecurity Data

•  Case study: Internal network breach


Our Mission: Power your Modern Data Architecture with HDP and Enterprise Apache Hadoop

Who we are June 2011: Original 24 architects, developers, operators of Hadoop from Yahoo! June 2014: An enterprise software company with 420+ Employees

Key Partners

Our model Innovate and deliver Apache Hadoop as a complete enterprise data platform completely in the open, backed by a world class support organization


LIMITATIONS Silos & Expensive

Single Purpose

APP

LIC

ATIO

NS

DAT

A S

YSTE

M

Business Analytics

Custom Applications

Packaged Applications

Why a Modern Data Architecture?

RDBMS EDW MPP

MDA: Key Drivers

1.  Leverage new types of data 2.  IT optimization 3.  Enable a data lake GOALS •  Extend new data sets across

existing data platforms •  Common data platform, multiple

processing engines •  Batch, interactive and real time

on a single data platform EXISTING Systems

Clickstream Web &Social

Geoloca9on Sensor & Machine

Server Logs

Unstructured

SOU

RC

ES


A Modern Data Architecture Includes Hadoop

Hadoop compliments and enhances existing technologies

Common data set, multiple applications •  Optionally land all data in a single cluster

•  Batch, interactive & real-time use cases

•  Support multi-tenant access, processing & segmentation of data

YARN: Architectural center of Hadoop •  Consistent security, governance & operations •  Ecosystem applications certified �

by Hortonworks to run natively in Hadoop

SOU

RC

ES

EXISTING Systems

Clickstream Web &Social

Geoloca9on Sensor & Machine

Server Logs

Unstructured

APP

LIC

ATIO

NS

DAT

A S

YSTE

M

Business Analytics

Custom Applications

Packaged Applications

RDBMS EDW MPP YARN: Data Operating System

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° N

HDFS (Hadoop Distributed File System)

Interactive Real-Time Batch


Unlock New Applications from New Types of Data INDUSTRY USE CASE Sentiment

& Web Clickstream & Behavior

Machine & Sensor Geographic Server Logs Structured &

Unstructured

Financial Services New Account Risk Screens ✔ ✔ Trading Risk ✔ Insurance Underwriting ✔ ✔ ✔

Telecom Call Detail Records (CDR) ✔ ✔ Infrastructure Investment ✔ ✔ Real-time Bandwidth Allocation ✔ ✔ ✔

Retail 360° View of the Customer ✔ ✔ ✔ Localized, Personalized Promotions ✔ Website Optimization ✔

Manufacturing Supply Chain and Logistics ✔ Assembly Line Quality Assurance ✔ Crowd-sourced Quality Assurance ✔

Healthcare Use Genomic Data in Medial Trials ✔ ✔ ✔ Monitor Patient Vitals in Real-Time ✔ ✔

Pharmaceuticals Recruit and Retain Patients for Drug Trials ✔ ✔ Improve Prescription Adherence ✔ ✔ ✔ ✔

Oil & Gas Unify Exploration & Production Data ✔ ✔ ✔ ✔ Monitor Rig Safety in Real-Time ✔ ✔ ✔

Government ETL Offload/Federal Budgetary Pressures ✔ ✔ Sentiment Analysis for Government Programs ✔


Break Down Silos with a Security Data Lake

SCA

LE

SCOPE

Unlocking the Data Lake

RDBMS

MPP

EDW

• Data Lake Enabled by YARN

•  Single data repository, shared infrastructure

•  Multiple security apps accessing all the data

•  Enable a shift from reactive to proactive interactions

•  Gain new insight across the entire enterprise

New Analytic Apps or IT Optimization

HDP 2.1

Gov

erna

nce

&

Inte

grat

ion

Secu

rity

Ope

ratio

ns

Data Access

Data Management

YARN


Big Data is Changing Cyber Security

“By 2016, more than 25 percent of global firms will adopt big data analytics for at least one security and fraud detection use case, up from current eight percent.”

– Gartner Cyber Security report Feb 2014

Gartner recommendations •  Align security capabilities in a holistic security strategy tailored to the threats and risks •  Target a single architecture to collect, index, normalize, analyze and share all information •  Organizations should look for profile accounts, users or other entities, and look for

anomalous transactions against those profiles


How Can Big Data Analytics Help Cyber Security?

•  To prioritize threats, vulnerabilities, and attacks •  To control endpoints and mobile connections /devices •  To prevent insecure devices from accessing secure systems •  To provide intelligence about the threat land- scape •  To reduce false positives


Securely explore your data

CYBER PATTERN DISCOVERY USING LINKED DATA ANALYSIS

A Big Data Solution with Hortonworks and Sqrrl Joe Travaglini, Director of Products, Sqrrl

© 2014 Sqrrl Data, Inc. | All Rights Reserved


Who We Are



Agenda

•  Security Analytics using (Big) Cybersecurity Data •  Dealing with the new security dilemma •  Why Hadoop and HDP are the perfect fit •  The ‘Linked Data’ Approach

•  Case study: internal network breach •  Overview of scenario •  Data modeling with Sqrrl •  Visual, contextual research and analysis



229 87% 90% $12.7M

Source: Mandiant Source: Verizon

Source: Verizon Source: Ponemon


The Numbers Don’t Lie


Targeted Attacks Have Changed the Game


Source: Battery Ventures


What Does This Mean For Us?

•  You’ve been breached. Deal with it. •  Empower the investigator •  Research and respond: better, faster, smarter •  It’s all about speed to understanding


Dissolution of the Secure Perimeter


The Security Data Dilemma


Detecting attacks requires more (i.e. BIG) data

But your tools can’t handle the big data wave

So attackers are spilling in


A Modern Data Architecture

•  Hortonworks Data Platform at the core

•  Sqrrl Enterprise stack at the app layer

Hadoop enables us to look at data differently


Hortonworks and Sqrrl Solution


Sqrrl Enterprise Architecture


Commodity Hardware HDFS + Accumulo

Raw Events Linked Data Model

Query Engine Bulk/Graph Processing

Visualization / API ML + Anomaly Detection

Physical

Data Storage

Data Model

Processing

Interface

Audit

Cryptography

Labeling + Policy

Security


Big Data Transformed


Security Data

VPN FW

Network Data

Proxy NetFlow

Application Data

HR USB

Email

Data Sources Linked Contextual Knowledge Analysis


Linked Data Analysis Adding structure to the noise



Case Study: Compromised Network



Breach Detection Scenario




Case Study Model


Data Sources

Users

Hosts

login

Linked Meta Model

flow

login

DNS records

Netflow

Host logs

Database logs

External Alerts


Case Study Example Mapping


Netflow Records

startTime endTime sourceIP destIP sourcePort destPort protocol tcpFlags bytesIn bytesOut

10/22/14 8:58 10/22/14 8:58 10.0.2.15 192.168.0.123 37051 139 TCP ...RS. 100 3355

10/22/14 8:45 10/22/14 8:45 10.0.2.15 192.168.0.6 0 3328 ICMP ...... 40 100

10/22/14 8:59 10/22/14 8:59 192.168.0.119 10.0.2.15 139 60071 TCP .A..S. 46 351

10.0.2.15

192.168.0.123

Class=Flow, totalBytes = 3455

192.168.0.6 Class=Flow,

totalBytes = 140


Case Study Example Data



Investigation Process


1. Set the Stage 2. Enable Search and Discovery

3. Automate Analysis

•  Define the security-centric entity/relationship model

•  Extract and maintain the model

•  Visually navigate assets and actors in the network

•  Drill down to the raw data seeding the model

•  Use behavioral analytics to build expectations of ‘normal’

•  Flag entities as potentially ‘abnormal’ and sniff them out


Visualizing the Threat



Thanks!


Joe Travaglini Director of Products, Sqrrl Data, Inc.

@joe_travaglini [email protected]

http://www.sqrrl.com

hortonworks sqrrl webinar v5.pptx

Technology