gerard toonstra | migrating to the cloud: our …...what we learned lessons learned from this...

Gerard Toonstra | Migrating to the Cloud: Our Journey | 28-11-2019

Data Center Architecture

Data Source 1

External Systems

(...)

Data Source 2

Data Source 3


Data Source 1

External Systems

(...)

Staging Data Warehouse

Data Mart 1

Data Source 2

Data Source 3

Data Mart 2


Data Source 1

External Systems

(...)


Data Mart 1 OLAP 1

Data Source 2

Data Source 3

Data Mart 2 OLAP 2


Data Source 1

External Systems

(...)

Azkaban


Data Mart 1 OLAP 1

Data Source 2

Data Source 3

Data Mart 2 OLAP 2


Data Source 1

External Systems

(...)


Data Mart 1 OLAP 1

Data Source 2

Data Source 3

Data Mart 2 OLAP 2

Azkaban

Azkaban


Data Source 1

External Systems

(...)


Data Mart 1 OLAP 1

Data Source 2

Data Source 3

Data Mart 2 OLAP 2

Migration Steps

Move OLAP Server from Data Center to

Cloud

Move SQL Server from Data Center to

Cloud

Moving away from Azkaban and adopt

Airflow

Create data validation mechanisms

● Templated arguments, like SQL

queries

● A lot of building blocks designed for

data engineering work

● Interface is not great to

browse historical runs

● Hard to rerun individual tasks

Azkaban vs Airflow

Azkaban● Task configuration separate

from code

● Dates as first class citizen; easily

trigger historical runs

Airflow

What do we need to migrate?

Big daily process

Identifying and splitting up

Introducing checkpoints

Organizing it in Airflow● Multiple pipelines instead of one

● Interdependent communication

● Multiple “checkpoints”

Our new daily process(es!)

Time

Load to staging area325 tasks

Heavy calculations50 tasks

Loading data warehouse tables140 tasks

Process semantic layer20 tasks

● Easier to read and reason

● Maintainable

● Separate logical units

● Less dependency management

● Interdependency checks can fail,

blocking the next step

● More code because of the

interdependency checks

Checkpoints approach

Disadvantages● Easier to test● Generally slower

Advantages

Ignoring the black box

Observing behaviour

?Source Staging

Old code

Most code looked like this

def main():

result = run_query('some_query.sql')

filename = create_csv(result)

upload_to_gcs(filename, GCS_FILENAME)

load_to_mssql(GCS_FILENAME, MSSQL_TABLE)

New code

Now we have this

OracleToMsSqlOperator(

sql='some_query.sql',

source_conn_id=’oracle’,

target_conn_id=’mssql’)

Advantages

Now we have this● Configuration as code

○ Easier to read

○ Very easy to test● Less code to maintain

○ Written and maintained by Airflow contributors

○ Custom code is rare instead of the default

● Quicker to create new pipelines

Summary

Airflow● All configuration now in code

● Building blocks for faster pipeline development

● Lot less code

● Manageable daily process

Migration Steps


Cloud


Cloud


Airflow


SQL Server

SQL Server from Data Center to Cloud

Data CenterPhysical Server Cloud

SQL Server

Amazon Relational Database Service (RDS)● Simple to setup and configure

● Supports multiple databases providers

● Patching the database software, backing up databases and

some other DBA tasks are managed by AWS itself

SQL Server

Step by Step - SQL Server Migration to Cloud

New SQL Server Instance on RDS

Deploy DW onto new Instance

Populate historical tables

Configure daily ETL in Airflow

Data Validation tools

SQL Server


MyDB: Properties: AllocatedStorage: "100" DBInstanceClass: db.m1.small Engine: sqlserver-se EngineVersion: "14.00.3015.40.v1" Type: "AWS::RDS::DBInstance"

SQL Server







Team City Deployment

SQL Server







SQL Server







Data Source 1

Data Source 2

Data Source 3ETL

SQL Server







SQL Server






Apache Beam


NBi

Data Validation

Summary

SQL Server on RDS● We can easily scale our instance

● No server maintenance

● All configurations in code (Cloudformation) facilitates maintenance

● Backup mechanism offered by AWS needs good understanding

+

Migration Steps


Cloud


Cloud


Airflow


OLAP Server

What is an OLAP database?● OLAP stands for OnLine Analytical Processing

● An OLAP database is a multi-dimensional array of data,

commonly referred as “cube”

● This technology used to facilitate query processing on

data warehouse.

OLAP Server

OLAP on top of Data warehouse

Data warehouse

Report 1

Report 2

Report N

OLAP Server

(SSAS)

OLAP Server

How to migrate our OLAP Server?

?

OLAP Server

Main Challenges

OLAP Server

Main Challenges

No support for our OLAP technology

OLAP Server

Main Challenges

No support for our OLAP technology

● Owning and support our VM

(EC2)

● Configure VM using “code” (no UI on Windows Server Core)

OLAP Server

Main Challenges

Weekly Recycling (wipe)

OLAP Server

Main Challenges

Weekly Recycling (wipe)

● Keep same machine

configurations after recycling

● Keep data in OLAP Server after recycling

OLAP Server

1st step - AMI (basebox)

431 2

OLAP Server

2nd step - Cloudformation (AWS Architecture)

21 43

OLAP Server

3rd step - Configurations and Backups

21 2 3 4

OLAP Server

4th step - Integration with our ETL pipeline

1 322 4

OLAP Server

Integrate OLAP Server with Airflow

Partition 2019W04

Partition 2019W03

Partition 2019W02

...

OLAP Server

Integrate OLAP Server with Airflow

Process Partition

Create Partition Partition 2019W04

Partition 2019W03

Partition 2019W02

...

OLAP Server

Integrate OLAP Server with Airflow and... USERS

Partition 2019W04

Partition 2019W03

Partition 2019W02

...

Summary

OLAP Server on EC2● We can easily scale our instance

● Infrastructure as Code facilitates maintenance

● Easy to rebuild machine if gets corrupted

● A lot of effort on understanding AWS & CI tools

● Load balancers don’t like SSAS

+

Migration Steps


Cloud


Cloud


Airflow


Automated validation


Same result set is important

PK value

1 A

2 B

3 C

PK value

4 D

3 C

2 B

SELECTTOP 3 *FROM FooORDER BY PK


Getting the hashes

PK hash

1 c4ca4238

2 c81e728d

3 eccbc87e

PK hash

1 c4ca4238

2 c81e728d

3 a87ff679


Comparing hashes

Sourcehashes

Targethashes Apache Beam

Not in source

Not in target

Different


Grouping the outputTable Type Count

A not_in_target 0

not_in_source 5

different 1000

B not_in_target 20

not_in_source 0

different 500


Daily report

Table Difference Difference yesterday

A 5000 0

B 300 300

C 20 10,000


What’s different?

Table Primary Key Type

A 1 not_in_target

A 2 not_in_source

A 3 different


Automated validation steps1. Get result set from source and target

2. Calculate hashes

3. Compare hashes, track differences

4. Store counts of differences in tracking tables

5. Talk through differences every day

Custom validation

Custom validation

NBi

● Unit testing for Business Intelligence, based on NUnit

● For tables where the logic changed, so needs custom

validation

● For validating the OLAP Server output

Summary

Validation● Automated validation for most of our data

● Custom validation for tables that changed

● Custom validation for important parts of

the OLAP Server

Apache Beam NBi

What we gained

What we learned

Lessons learned from this migration (1 / 2)● Not everything you have on data center will be supported by AWS as it is

● Less sqlserver DB monitoring capabilities in comparison to data center.

No server admin on RDS.

● Doing two migrations in parallel (Azkaban → Airflow, data center → AWS)

causes a much longer migration process, but the target environment shadows

your still active production environment.

What we learned

Lessons learned from this migration (2 / 2) ● You should arrange training on AWS/DevOps upfront

● Think about infrastructure as code, both for Airflow pipelines as well as weekly

OLAP recycling: all is in code now, less in documentation and manual routines

● AWS flexibility allows you to scale your infrastructure with ease

Timeline

Gerard Toonstra| Data Discovery with Amundsen | 28-11-2019 | https://www.careersatcoolblue.com

Join at slido.com with #bigdata2019

gerard toonstra | migrating to the cloud: our …...what we learned lessons learned from this...

Documents