nyc hug - application architectures with apache hadoop

Headline Goes Here Speaker Name or Subhead Goes Here

DO NOT USE PUBLICLY PRIOR TO 10/23/12

ApplicaAon Architectures with Apache Hadoop Mark Grover | @mark_grover NYC HUG slideshare.com/markgrover October 14th, 2014

About Me •  CommiPer on Apache Bigtop, commiPer and PPMC member on Apache Sentry (incubaAng).

•  Contributor to Hadoop, Hive, Spark, Sqoop, Flume. •  SoWware developer at Cloudera • @mark_grover

Co-‐authoring O’Reilly book

• @hadooparchbook •  hadooparchitecturebook.com •  Strata Hadoop World Tutorial

•  at 9 AM tomorrow

Click Stream Analysis

Case Study

AnalyAcs

Web Logs – Combined Log Format

244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/1.0" 200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; HTC Vision Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1”

Clickstream AnalyAcs

244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36”

Similar use-‐cases

•  Sensors – heart, agriculture, etc. •  Casinos – session of a person at a table

Challenges of Hadoop ImplementaAon

Other challenges -‐ Architectural ConsideraAons

•  Storage managers? •  HDFS? HBase?

•  Data storage and modeling: •  File formats? Compression? Schema design?

•  Data movement •  How do we actually get the data into Hadoop? How do we get it out?

•  Metadata •  How do we manage data about the data?

•  Processing •  How can we transform it? How do we query it?

•  OrchestraAon •  How do we manage the workflow for all of this?

Since that’s all what the Ame allows todayJ

2. Processing

Processing

• De-‐duplicaAon •  Filtering •  SessionizaAon

DeduplicaAon – remove duplicate records

244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36”

Filtering – filter out invalid records

244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/1.0" 200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U…

SessionizaAon

Website visit

Visitor 1 Session 1

Visitor 1 Session 2

Visitor 2 Session 1

> 30 minutes

Why sessionize?

Helps answers quesAons like: • What is my website’s bounce rate?

•  i.e. how many % of visitors don’t go past the landing page? • Which markeAng channels (e.g. organic search, display ad, etc.) are leading to most sessions?

• Which ones of those lead to most conversions (e.g. people buying things, signing up, etc.)

• Do aPribuAon analysis – which channels are responsible for most conversions?

SessionizaAon

244.157.45.12 - - [17/Oct/2014:21:08:30 ] "GET /seatposts HTTP/1.0" 200 4463 "http://bestcyclingreviews.com/top_online_shops" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36” 165 244.157.45.12 - - [17/Oct/2014:21:59:59 ] "GET /Store/cart.jsp?productID=1023 HTTP/1.0" 200 3757 "http://www.casualcyclist.com" "Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; HTC Vision Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1” 166

How to Sessionize?

1.  Given a list of clicks, determine which clicks came from the same user

2.  Given a parAcular user's clicks, determine if a given click is a part of a new session or a conAnuaAon of the previous session

#1 – Which clicks are from same user?

• We can use: •  IP address (244.157.45.12) •  Cookies (A9A3BECE0563982D) •  IP address (244.157.45.12)and user agent string ((KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36")

#2 – Which clicks part of the same session?

> 30 mins apart = different sessions

github.com/hadooparchitecturebook/clickstream-‐tutorial

SessionizaAon in MapReduce

Reduce

Log line IP1, log lines

IP1, log line

Log line, session ID

Log line

Log line IP2, log lines

IP2, log lines Log line, session ID

Mapper for SessionizaAon

public static class SessionizeMapper extends Mapper<Object, Text, IpTimestampKey, Text> { private Matcher logRecordMatcher; public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { logRecordMatcher = logRecordPattern.matcher(value.toString()); // We only emit something out if the record matches with our regex. Otherwise, we assume the record is busted and simply ignore it if (logRecordMatcher.matches()) { String ip = logRecordMatcher.group(1); DateTime timestamp = DateTime.parse(logRecordMatcher.group(2), TIMESTAMP_FORMATTER); Long unixTimestamp = timestamp.getMillis(); IpTimestampKey outputKey = new IpTimestampKey(ip, unixTimestamp); context.write(outputKey, value); } } }

Reducer for SessionizaAon

public static class SessionizeReducer extends Reducer<IpTimestampKey, Text, IpTimestampKey, Text> {

private Text result = new Text();

public void reduce(IpTimestampKey key, Iterable<Text> values, Context context

) throws IOException, InterruptedException { // The sessionId generated here is per day, per IP. So, any queries // that will be done as if this session ID were global, would require // a combination of the day in question and IP as well. String sessionId = null; Long lastTimeStamp = null; for (Text value : values) { String logRecord = value.toString();

Reducer for SessionizaAon

// If this is the first record for this user or it's been more than the timeout since // the last click from this user, let's increment the session ID. if (lastTimeStamp == null || (key.getUnixTimestamp() -‐ lastTimeStamp > SESSION_TIMEOUT_IN_MS)) { sessionId = key.getIp() + "+" + key.getUnixTimestamp(); } lastTimeStamp = key.getUnixTimestamp(); result.set(logRecord + " " + sessionId); // Since we only care about printing out the entire record in the result, with session ID appended // at the end, we just emit out "null" for the key context.write(null, result); } } }

Secondary sorAng – by Amestamp

• Need records to reducer to be grouped by IP address and sorted by Amestamp – a concept called secondary sor/ng

•  Instead of using just IP address as map output key and reduce input key

• We use a composite key (IP, Amestamp) as map output key and reduce input key

Secondary sorAng – vocabulary

•  Composite key – IP address, Amestamp • Natural key – IP address •  Secondary sort key -‐ Amestamp

Secondary sorAng

•  Custom Grouping Comparator – on Natural Key (IP) •  Custom Sort Comparator – on Composite Key (IP, address) •  Custom ParAAoner – on Natural Key (IP) job.setGroupingComparatorClass(NaturalKeyComparator.class); job.setSortComparatorClass(CompositeKeyComparator.class); job.setPartitionerClass(NaturalKeyPartitioner.class);

Final Architecture

Hadoop Cluster

BI/VisualizaAon tool (e.g.

microstrategy)

BI Analysts

Spark For machine learning and graph processing

R/Python StaAsAcal Analysis

Custom Apps

3. Accessing

2. Processing

4. OrchestraAon via Oozie 1. IngesAon

OperaAonal Data Store

CRM System Via Sqoop

Web servers

Website users

Final Architecture – High Level Overview

Data Sources IngesAon

Data Storage/Processing

Data ReporAng/Analysis

Final Architecture – IngesAon

Web App Avro Agent Web App Avro Agent

Flume Agent

Fan-‐in PaPern

MulA Agents for Failover and rolling restarts

Final Architecture – Storage and Processing

/etl/weblogs/20140331/ /etl/weblogs/20140401/ …

Data Processing /data/markeAng/clickstream/bouncerate/ /data/markeAng/clickstream/aPribuAon/ …

Final Architecture – Data Access

Hive/Impala

BI/AnalyAcs Tools

DWH Sqoop

Local Disk

R, etc.

DB import tool

JDBC/ODBC

Contact info • Mark Grover

• @mark_grover •  www.linkedin.com/in/grovermark

•  Slides at slideshare.net/markgrover

nyc hug - application architectures with apache hadoop

Technology

jan 2013 hug: impala - real-time queries for apache hadoop

july 2012 hug: building data pipelines on hadoop

hadoop, pig, and python (pydata nyc 2012)

dc hug hadoop for windows

introduction to hbase -...

cassandra hadoop integration at hug france by piotr...

hue: the hadoop ui - stockholm hug

kafka & hadoop - for nyc kafka meetup

galvanise nyc - scaling r with hadoop & spark. v1.0

november 2013 hug: cyber security with hadoop

nyc hadoop meetup - mapr, architecture, philosophy and...

feb 2013 hug: a visual workbench for big data analytics on...

aug 2012 hug: hug bigtop

hadoop user group ireland (hug) ireland - eddie baggot...

couchdoop: connecting hadoop with couchbase (london hug)

august 2013 hug: hue: the ui for apache hadoop

hug france - administration hadoop et retour d’expérience...

august 2013 hug: compression options in hadoop - a tale of...

july 2012 hug: using standard file-based applications and...

may 2012 hug: oozie: towards a scalable workflow management...