data driven culture & infrastructure from the ground up

Data-driven culture & infrastructure from the ground up

January 2017

Data-driven culture and infrastructure from the ground up

Agenda

● R&D and IT consultancy firm● Clients - Silicon Valley Startups from Round A to Round D● Hardcore Big Data Analytics team in Kazan

Data-driven culture and infrastructure from the ground up

January 2017

People driven

Data-driven culture and infrastructure from the ground up

Excel driven

January 2017

Data-driven culture and infrastructure from the ground up

BI driven

January 2017

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product Manager

Owners

Engineer UI Data Scientist

Sales/Marketing DevOps Customer

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Engineer

Alerts, Errors

Metrics

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Engineer

Alerts, Errors

Metrics

UX

Usage Patterns

A/B tests

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Engineer

Alerts, Errors

Metrics

Data Scientist

Scripts

Models

UX

Usage Patterns

A/B tests

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Owners

Revenue trends

Prescriptions

Insights

Engineer

Alerts, Errors

Metrics

Data Scientist

Scripts

Models

UX

Usage Patterns

A/B tests

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Owners

Revenue trends

Prescriptions

Insights

Engineer

Alerts, Errors

Metrics

Data Scientist

Scripts

Models

Marketing

BI tools

A/B test

UX

Usage Patterns

A/B tests

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Owners

Revenue trends

Prescriptions

Insights

Engineer

Alerts, Errors

Metrics

DevOps

Alerts, Errors

Metrics

Monitoring

Data Scientist

Scripts

Models

Marketing

BI tools

A/B test

UX

Usage Patterns

A/B tests

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Customer

Dashboards

Personalization

Insights

Reports

Owners

Revenue trends

Prescriptions

Insights

Engineer

Alerts, Errors

Metrics

DevOps

Alerts, Errors

Metrics

Monitoring

Data Scientist

Scripts

Models

Marketing

BI tools

A/B test

UX

Usage Patterns

A/B tests

Data-driven culture and infrastructure from the ground up

Data Driven Team

January 2017

Product

Metrics

Ad-hoc reports

Reports

Customer

Dashboards

Personalization

Insights

Reports

Owners

Revenue trends

Prescriptions

Insights

Engineer

Alerts, Errors

Metrics

DevOps

Alerts, Errors

Metrics

Monitoring

Data Scientist

Scripts

Models

Marketing

BI tools

A/B test

UX

Usage Patterns

A/B tests

Data-driven culture and infrastructure from the ground up

● Product Managers - SQL

● Engineers - Production, Product, Metrics, SOFT SKILLS

● UX - Marketing, Analytics

● Sales Engineer

● Business - BI, Reports, Metrics

January 2017

New Cross Skills - Evangelism

Data-driven culture and infrastructure from the ground up

Infrastructure

Data-driven culture and infrastructure from the ground up

● You could not just BUY it

● It’s not a Data Warehouse

● It’s not easy at all

January 2017

Myths

Data-driven culture and infrastructure from the ground up

High Level

January 2017

App Servers

Production database

Logs

Storage

BI

Prescriptions

Application Data Infrastructure Visualisation

Data-driven culture and infrastructure from the ground up

Micro

January 2017

App Servers

Production database Logs agg/

Papertrail

Keen.ioClicdata

DatadogPrometheus

Application Data Infrastructure Visualisation

GA

Data-driven culture and infrastructure from the ground up

Small

January 2017

App Servers

Production database

Log agg/ Papertrail

S3

ClicdataInteranaLooker

DatadogPrometheus

Application Data Infrastructure Visualisation

Slave replica

Data-driven culture and infrastructure from the ground up

Medium

January 2017

App Servers

Prod

Log stream S3ClicdataInteranaLooker

DatadogPrometheus

Application Data Infrastructure Visualisation

Slave replica

ETL RedShift

Ad-hoc SQL

Data-driven culture and infrastructure from the ground up

Medium

January 2017

App Servers

Prod

AWS Kinesis S3ClicdataInteranaLooker

Datadog

Application Data Infrastructure Visualisation

Slave replica

AWS Lambda RedShift

Ad-hoc SQL

Data-driven culture and infrastructure from the ground up

Medium

January 2017

App Servers

Prod

Kafka S3ClicdataInteranaLooker

DatadogPrometheus

Application Data Infrastructure Visualisation

Slave replica

Spark Streaming RedShift

Ad-hoc SQL

Data-driven culture and infrastructure from the ground up

Big

January 2017

App Servers

Prod

TableauClicdataInteranaLooker

Application Data Infrastructure Visualisation

Slave replica

ETL/Spark/Streaming

RedShift

Ad-hoc SQL

Hadoop/Parquet Impala

Spark SQLData Science

www.squadex.com

125 University Avenue,

Suite 290, Palo Alto,

California, 94301

Questions, details?We would be happy to answer!