ramunas balukonis. research dwh

VISIT OUR BLOG: adform.comTWITTER: adforminsider

Research of technologies for Big Data Analytics

(2013-2014)

1

Ramūnas Balukonis, Adform

Our impressions growth

3

Now 2 blns transaction or 1,4 TB per day

(RAW)

2012 we started to research for technology to

process, load and provide data for analytics

0

50

100

150

200

250

300

350

400

450

500

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Impressions Per Year, BLNS of ROWS

Where we are now

4

DWH – our needs for Big Data Analytics

5

Query performance up to moments

No downtime window

Short time to market

Near real time latency

No backups

Unattended scaling

Inessential data loss and data discrepancies

6

How we tested

7

Testing takes up 3 month for each technology to

finish test

Testing env: 3X (24 Cores + 96 GB RAM + 800

GB RAID10)

Loaded 5 TB of data (non compressed data)

Candidates for BIG Data Analytics

8

IBM Netezza

9

Appliance: no commodity HW

No elastic scale out

Global presence, sales, delivery and support.

HP Vertica

10

Elastic scale out

Brilliant performance (Load/Select)

No stored procedures

No UI

Price per TB

SAP Sybase IQ

11

Scaling using shared disk

Similar to MS SQL (tools, logic, stored procs,

system views and SP, BOL similar)

Concerns about easy of implementation and

use

Price per core

Amazon Redshift

12

Price – the only player we tested that provides

prices online

Filters impact on query performance badly

Cluster resize/scaling

Unstable connection

Calpont InfiniDB

13

Shared nothing

MySQL as front end – tools, connectors,

procedures etc.

Community (offers prebuild solutions) or EE

Super fast load

Relatively slow query perf

Slow insert/update/delete

Where we are now

15

What we learned

Number of suitables technologies drops whenTBs increses

Adopt technology to your requirements and notvice versa

No Silver Bullet: Queries vs row store – 10X

Load speed vs row store – 4X

Compression vs row store – 4X

... And we‘ll learn much more after we‘ll run ourfirst report

16

Thank you

Ramūnas Balukonis

17