august 2014 hug : comprehensive security for hadoop

© Hortonworks Inc. 2014

Apache Argus Comprehensive Security for Enterprise Hadoop

Don Bosco Durai Balaji Ganesan


Agenda

•  Security Landscape •  Apache Argus Vision •  Apache Argus Features •  Apache Argus Architecture •  Demo


Security needs are changing

Administration Centrally management & consistent security

Authentication Authenticate users and systems

Authorization Provision access to data

Audit Maintain a record of data access

Data Protection Protect data at rest and in motion

Security needs are changing •  YARN unlocks the data lake •  Multi-tenant: Multiple applications for data access

•  Changing and complex compliance environment

•  ETL of non-sensitive data can yield sensitive data

Summer 2014 65% of clusters host multiple workloads

Fall 2013 Largely silo’d deployments with single workload clusters

5 areas of security focus


Apache Argus Vision

•  Provide a centralized framework for managing security within Hadoop ecosystem

•  Single point of administration for authentication, authorization, auditing and data protection

•  Integrated approach for entire Hadoop ecosystem, i.e., platform level security

•  Tight integration with components •  Integrate with standard APIs, hooks for access control

•  Maintain single version of truth, for consistency in access policies

•  Extensible Infrastructure •  REST APIs for managing policies •  Pluggable enforcement


Security in Hadoop w/ Apache Argus

Authorization Restrict access to explicit data

Audit Understand who did what

Data Protection Encrypt data at rest & in motion

•  Kerberos in native Apache Hadoop

•  HTTP/REST API Secured with Apache Knox Gateway

•  HDFS Permissions, HDFS Extended ACLs

•  Audit logs in with HDFS & MR •  Hive ATZ-NG

Authentication Who am I/prove it?

•  Wire encryption in Hadoop

•  Orchestrated encryption with 3rd party tools

•  HDFS, Hive , Hbase, Storm, Knox, etc.

•  Fine grain access control

•  RBAC

•  Centralized audit reporting

•  Policy and access history

•  Fine grain encryption*

•  Enabled partners*

Had

oop

Apa

che

Arg

us

Centralized Security Administration

•  SSO* •  Provisioning* •  Brokering*


Apache Argus Security Features - Current

Apache Argus Role Based Authorizations

Fine grained access control HDFS – Folder, File, Hive – Database, Table, Column, UDFs HBase – Table, Column Family, Column

Wildcard Resource Names Yes Permission Support HDFS – Read, Write, Execute

Hive – Select, Update, Create, Drop, Alter, Index, Lock Hbase – Read, Write, Create

Auditing Configurable audit Auditing can be controlled through policy Resource access auditing User id, request type, repository, access resource, IP

address, timestamp, access granted/denied Admin auditing Changes to policies, login sessions and agent

monitoring,


Apache Argus

Reporting

Basic reporting in portal Global view of policies, across HDFS, Hbase and Hive

Manage

User/ Group mapping Local, Sync with LDAP/AD, Sync with Unix Delegated administration Delegate policy administration to groups or users

Apache Argus Security Features


Central Security Administration Argus - Portal •  Delivers a ‘single pane of glass’ for the security administrator •  Centralizes administration of security policy


Setup Authorization Policies - HDFS

file level access control, flexible definition

Control permissions


Setup Authorization Policies - Hive


Monitor through Auditing

11


Authorization and Auditing w/ Argus

Hadoop distributed file system (HDFS)

Argus Administration Portal

HBase

Hive Server2

Argus Policy Server

Argus Audit Server

Argus Plugin

Had

oop

Com

pone

nts

Ent

erpr

ise

Use

rs

Argus Plugin

Argus Plugin

Legacy Tools

Integration API

RDBMS

HDFS

Knox

[components]

Argus Plugin*

Argus Plugin*

Argus Plugin*

Storm

YARN : Data Opera.ng System

* - Future Integration


Resources

Page 13


Argus Page


Argus GitHub


Roadmap

Page 16


Hortonworks Security Investment Plans HDP + Apache Argus

Comprehensive Security for Enterprise Hadoop

…all IN Hadoop

Goals:

Investment themes

Central Administration Provide one location for administering security policies and audit reporting for entire platform

Comprehensive Security Meet all security requirements across Authentication, Authorization, Audit & Data Protection for all HDP components

Consistent Integration Integrate with other security & identity management systems, for compliance with IT policies

Current Phase • Centralized Security Admin for HDFS, Hive & Hbase, Storm, Knox

• Centralized Audit Repor@ng • Delegated Policy Administra@on

Previous Phases ü  Kerberos Authen@ca@on ü  HDFS, Hive & Hbase authoriza@on ü  Wire Encryp@on for data in mo@on ü  Knox for perimeter security ü  SQL Style Hive Authoriza@on ü  ACLs for HDFS

Delivered

Future Phases • Encryp@on in HDFS, Hive & Hbase • Centralized security administra@on of en@re Hadoop plaHorm

• Centralized audi@ng of en@re plaHorm • Expand Authen@ca@on & SSO integra@on choices • Tag based global policies (e.g. Policy for PII)

Delivered XA Secure


Q&A

Page 18

august 2014 hug : comprehensive security for hadoop

Education

security needs

security requirements

security administrator

apache argusrole

hadoop apache arguspage

access resource

argus pagepage

argus githubpage