Weathering the Data Storm: How SnapLogic and Amazon Web Services

Deliver Analytics in the Cloud for Earth Networks

Today’s Agenda

Eddie Dingels

Architect, Earth NetworksErin Curtis

Product Marketing, SnapLogic

•  Amazon Web Services

•  SnapLogic and Redshift

•  Earth Networks: Moving Data Analytics Into The Cloud

•  Discussion

Kyle Lichtenburg

Solutions Architect, Amazon Web Services

2011

82

159

2012

280

2013

516

2014

AWS’  Rapid  Pace  of  Innova3on  AWS  has  launched  a  total  of  1,515  new  features  and/or  services  since  incep:on  in  2006.    

2015

+342*

* As of July 9, 2015

More  Func:onality  Than  Any  Other  Infrastructure  Provider  

It’s  never  been  easier  and  less  expensive  to  collect,  store,  analyze  &  share  data  

Companies  will  use  data  more  expansively  than  at  any  other  point  in  history  

Fully  Loaded  for  Big  Data  

•  Sources  of  Truth   •  High  Performance  Databases  

•  Analysis  PlaKorms  

Amazon S3 Amazon Glacier

Amazon EFS

Amazon DynamoDB Amazon Aurora

Amazon Redshift Amazon Kinesis Amazon EMR

Amazon  Simple  Storage  Service  (S3)  

• Storage  for  the  Internet    • Store  and  retrieve  any  amount  of  data,  at  any  :me,  from  anywhere  on  the  web  

• Highly  scalable,  reliable,  and  secure  • Supports  encryp:on  • Pay  only  for  what  you  use  

Amazon  DynamoDB  

• Fast,  fully-­‐managed  NoSQL  Database  Service  • Capable  of  handling  any  amount  of  data  • Durable  and  Highly  Available  • All  SSD  storage  • Simple  and  Cost  Effec:ve  

Amazon  RedshiX  

• Fast,  simple,  fully-­‐managed  petabyte-­‐scale  data  warehousing  

• Online  and  func:onal  in  minutes  • SQL  based  • Con:nuous  backup  • Less  than  $1,000/TB/Year  • ODBC/JDBC  Compliant  

Connect Faster

Unified Platform for Data, Apps, Things

Our unified platform significantly speeds up enterprise data access everywhere.

– Gaurav Dhillon, co-founder and CEO, SnapLogic

Why SnapLogic Elastic Integration?

Unified Platform Productive User Experience

Modern Architecture Connected: 300+ Snaps

Productive: UX for Citizen and Advanced Users

We can do more in two hours with SnapLogic than we could in two days

with traditional solutions.

•  Integration Cloud: Design, Admin, Monitoring• Drag, Drop, Connect: HTML5 interface built for speed

Modern Architecture: Hybrid and Elastic

Streams: No data is stored/cachedSecure: 100% standards-basedElastic: Scales out & handles data and app integration use cases

Metadata

Data

Databases Enterprise Systems Hadoop

Modern Architecture: Real-Time and Batch

Ultra Pipelines SnapReduce and the Hadooplex

Map Reduce

Certified YARN Execution

Connected: 300+ Snaps

We look at SnapLogic as an opportunity to think differently about integration.

SnapLogic Integration for Amazon Redshift

Customers:

Free, hosted trial of SnapLogic + Redshift: www.snaplogic.com/redshift-trial

The Redshift Snap helps customers rapidly transfer data into and out of Amazon Redshift from multiple sources•  Rapidly connect Redshift to database services•  Quickly load data into an Amazon S3 bucket and kick off the Redshift

import process in a single step•  Easily replicate source tables into their Amazon Redshift clusters and

detect daily changes to keep data synchronized•  Take advantage of core REST and SOAP connectivity

Edward Dingels

7/22/2015

AWS & SnapLogic

Company

•  Weather Networks

•  Schools/Education

•  Consumer

•  Alerting

•  Environmental Network

•  Energy

7/22/2015

Operational Environment

•  Weather •  Dynamic

•  Local

•  Users •  Engaged

•  Proximity

•  Spikes/Peak Periods

7/22/2015 22

Data Center

•  Scale •  Weather intersecting users •  Increase rapidly

•  Capacity •  Hardware •  Software

7/22/2015 23

Data Center

•  Mobile pushed us to the limit •  Demanding performance •  Feature releases delays due to capacity planning

The data center became a limiting factor in a space where technology should enable

7/22/2015 24

AWS EC2

•  EC2 •  Dynamic capacity •  Automatic capacity

•  SQL •  Data tier •  Horizontal scale

7/22/2015 25

SQL Data Store

API Tier

Ingest

AWS Storage

•  Blob •  S3

•  Transactional •  Dynamo •  RDS

•  Warehouse •  Redshift

7/22/2015 26

Cloud Storage

API Tier

Ingest

AWS ETL

•  Storage was great

•  ETL limiting •  SQS

•  Kinesis •  EMR

•  Data Pipeline

•  How do we move data between different cloud data stores effectively?

7/22/2015 27

Cloud Integration

•  Needed a new tool set

•  Criteria •  Data stayed in our VPC

•  Repeatable building blocks

•  Cloud data stores are 1st tier citizens

•  Horizontal scale

•  Performance

•  Price

7/22/2015 28

Project – Data Ingest

•  Network of Networks

•  Challenge – Providers •  Formats

•  Delivery •  Standardization

•  Solution – Pipeline Per Provider

7/22/2015 29

Partner A

Partner B

Pipeline A

Pipeline B

Cloud Data Stores

Project – Data Analysis

•  Deriving KPIs for BI

•  Challenge – Domains •  Unique domain

•  Different storage technologies •  Varied timeliness requirements

•  Solution – Pipeline Per Domain

7/22/2015 30

Redshift

BI Toolset

Pipeline

Domain

Operational – Automated Database Tasks

•  Storage Limitation

•  Challenge – Automation •  Redshift

•  MSSQL RDS

•  Solution – Scheduled pipeline

7/22/2015 31

Scheduled Pipeline

Cloud Data Stores

AWS + snapLogic

•  AWS is the platform

•  SnapLogic is the glue

Drives faster implementations with repeatable patterns for more business value

7/22/2015

Thank you Questions?

See SnapLogic in action:

Contact us: info@snaplogic.com

http://video.snaplogic.com/

@SnapLogic @awscloud @EarthNetworks