TURKCELL DBA Ferhat ŞENGÖNÜL http://ferhatsengonul.wordpress.com http://twitter.com/ferhatsengonul

TURKCELL’s EXADATA

Journey Part1 & Part 2

May 2011

Who am I?

• 11 years in IT in finance sector.

• Works with (nearly) all db’s from hierarchical to relation

• Found peace in Exadata

• Likes APEX (as an amateur)

• 1 year in Turkcell

• http://ferhatsengonul.wordpress.com

• http://twitter.com/ferhatsengonul

Headlines

• Turkcell in numbers

• BI domain in numbers

• First project • Migration to V2 (8 nodes) ( Total uncompressed size 250 TB)

• Second project • Migration to 2 x2-2’s (16 nodes)

• Consolidation of 4 databases ( Total uncompressed size 600 TB)

• Geographical (continental) change of data center

• Future plans

Turkcell in Numbers

• Leading Telco in Turkey

• 34 million subscribers in Turkey as of Feb 28, 2011

• Third biggest GSM operator in Europe in terms of subscribers

• Turkcell started its operations in February 1994

• Turkcell co-operates in 8 other countries and has over 60 million subscribers total

• First and only Turkish company listed on NYSE

• Over 500 Oracle Databases, 150+ Production

• DB Machine hosts our biggest Database from DWH Domain

Turkcell’s BI Environment

Reporting MSTR

ETL AB Initio

Oracle ODI Source

DBs

Exadata

Other DWH DBs

Amount Of Data • 3 Billion CDR per Day

• 600 -1000 GB raw data extracted from 20+ source databases

• 5 TB Data on file system processed

• 2-3 TB loaded into databases, all into Exadata

• Approximately 600 TB Customer data stored in multiple Databases • 600 TB (60 TB compressed) on Exadata

Turkcell’s BI Environment – Application Footprint

ETL Environment • AB Initio – runs on 8 Node Linux Cluster

• Data loaded daily between 19:00 - 08:00 (SLA)

• Sql*Loader used to load tables with 16 parallel threads

• Works to implement external tables continue

Reporting Environment • MSTR (Microstrategy) is used mostly.

• 4 nodes before migration. 3 nodes after migration.

• 300+ Users

• 3000 distinct reports with 50k run per month

• Runs between 08:00 – 20:00 (SLA)

First Project ( completed in July 2010)

• Turkcell’s largest 100 TB (~250 TB uncompressed) DB was migrated to DBM v2, now only 25 TB with the help of HCC on Full SAS Rack

• Over 50K reports run every month on this DB, Performance improvement up-to 400x, average 10x

• 1 RACK instead of 11 RACKS

Business Drivers - Why Exadata ?

EMC DMX-4

70 TB

HITACHI USP-V

50 TB

Sun M9000 Sparc 7

176 Threads

ORACLE Exadata

V2

OLD SYSTEM NEW SYSTEM

Server Model Sun M9000 Oracle Exadata V2

Cpu Type Sun Sparc 7 2.52 GHz Xeon® E5540 Processors 2.53 GHz

Number Of Cpu Threads 176 128

Total Main Memory 512 GB 576 GB

Total Storage Capacity 120 TB 30 TB

Storage Connection Technology Fiber Channel (32 x 4 Gtbit/s) Infiniband ( 8 x 40 Gbit

Storage Maximum IO Throughput Capacity 5 GB/s 21 GB/s

Server + Storage units Total Power 57 KVA 20 KVA

Server + Storage units Total Form Factor 11 Rack 1 Rack Approximate Data Backup Duration 44 Hours 14 Hours

Number of Backup Tape Cartridges Per Backup 159 57

Business Drivers - Why Exadata ?

Simplified Architecture • Full Rack replaces Sun M9000 & ten(10) Storage Cabinets

• Single Vendor Strategy

• It took Turkcell a few years to perfect current environment – Exadata up & running in 5 days

• We need to address/deal with one party if a problem occurs

Effortless Scalability: one to Multi-Rack • Data volume explodes – data size doubles every year (45TB to 100TB)

• Old storage environment was maxed out – no chance to scale out

• Old env 600+ (EMC DMX4 & Hitatchi USP-V) disks with max I/O throughput of 5 GB/sec. vs 168 SAS disks with a max I/O throughput of 21 GB/sec.

Business Drivers - Why Exadata ?

2486

648 1161

314 193 177 139 40 49 33 29 87

2944

665

1403

489 271 260 112 52 45 35 17 75

2227

541 1023

361 179 192 107

57 34 28 25 88

3284

454 346 88 33 28 9 5 2 4 0 6

4827

376 199 34 22 14 3 2 1 3

4750

419 252 35 19 4 2 1 1 1 0

1000

2000

3000

4000

5000

6000

0-5

dk

5-1

0 d

k1

0-3

0 d

k3

0-4

5 d

k4

5-6

0 d

k1

-1.5

saa

t1

.5-2

saa

t2

-2.5

saa

t2

.5-3

saa

t3

-3.5

saa

t3

.5-

4 s

aat

4 s

aat…

0-5

dk

5-1

0 d

k1

0-3

0 d

k3

0-4

5 d

k4

5-6

0 d

k1

-1.5

saa

t1

.5-2

saa

t2

-2.5

saa

t2

.5-3

saa

t3

-3.5

saa

t3

.5-

4 s

aat

4 s

aat…

0-5

dk

5-1

0 d

k1

0-3

0 d

k3

0-4

5 d

k4

5-6

0 d

k1

-1.5

saa

t1

.5-2

saa

t2

-2.5

saa

t2

.5-3

saa

t3

-3.5

saa

t3

.5-

4 s

aat

4 s

aat…

0-5

dk

5-1

0 d

k1

0-3

0 d

k3

0-4

5 d

k4

5-6

0 d

k1

-1.5

saa

t1

.5-2

saa

t2

-2.5

saa

t2

.5-3

saa

t3

-3.5

saa

t3

.5-

4 s

aat

4 s

aat…

0-5

dk

5-1

0 d

k1

0-3

0 d

k3

0-4

5 d

k4

5-6

0 d

k1

-1.5

saa

t1

.5-2

saa

t2

-2.5

saa

t2

.5-3

saa

t3

-3.5

saa

t0

-5 d

k5

-10

dk

10

-30

dk

30

-45

dk

45

-60

dk

1-1

.5 s

aat

1.5

-2 s

aat

2-2

.5 s

aat

2.5

-3 s

aat

3.5

- 4

saa

t

Temmuz (05-11) Temmuz (12-18) Temmuz (19-25) Temmuz (26-01) Agustos(02-08) Agustos(09-15)

Weekly Report Count & Run-time

25 min 27 min 7 min 3 min 3 min 25 min

Avg Time

Exadata Migration

Performance Needs • Over 50K reports run every month on this DB

• Average report run time is reduced from 27 minutes to 3 minutes !!!

• Reports completed less than 5 mins rose from %45 to %90

• Reports running more than 4 hours down from 87 to 1

The Project Overview

Planned as simple migration

• No Application version or interface changes

• Migrate to new Exadata V2 machine

• Upgrade to new Oracle Version (from 10.2.0.4 to 11.2.0.1)

• Move from Single Instance to RAC

4 Months (Apr-July) testing of all the components

• Crash Tests (working closely with support)

• Network Transfer Speed Tests (10G vs IB)

• Backup/Restore Tests

• Performance Tests (RAT, Report run times, data load times)

Project Challenges

• Will we fit into 30 TB ? (~100 TB 10g compressed)

• How to move that much data in 2 days ? • 100TB 10g compressed, how much of it can be moved before/after

• How much data needs to be moved during mig window

• What kind of Network infrastructure is needed to support such xfr rate

• 8-node RAC ? (earlier attempts to use RAC did not go through) • Training needs for RAC and exadata storage architecture

• Rollback plan: Parallel run (load data to both DB) Environment

Migration Facts

• Insert/append over DB Links • Platform and version change forced us to use insert over db-link

• None of the other methods like TTS, ASM Rebalance was applicable

• Used in-house pl/sql utility to perform migration • Metadata (Tablespaces were re-organized)

• Insert/Append with 128 parallel sessions partition-wise

• 40TB data was transferred during the migration window • Transfer rate 300mB/sec, around 1 TB per hour approximately

• Completed in 36 hours.

• High CPU load on comp nodes because of HCC (expected)

• SQL*Net compressing the data 2-3x (this was a nice finding)

• 20 TB readonly data was transferred before, ~30TB after

• Network Infrastructure • 10Gb-to-IB (Voltaire 4036E) gateway used to connect old system

Migration Facts

• After the test migration we ended-up staying on Exadata • End users were so happy, they did not let us go back

• 3 Weeks before the go-live date we were live

• Parallel runs continued for few weeks till we completely feel comfortable on Exadata • Stability of the system under real load was proved for various load

patterns

• Backup/Restore tests were completed

Compression in Action

Old System 10gR2 Compression • ~2-3 times ~250TB raw data to 100TB

Exadata V2 with EHCC • Raw Data 250TB to 25TB (Data)+ 5TB (Temp) = 30TB

• EHCC - Compress ratio ~7-10x

• Archive compression is efficient but high CPU consumption

http://ferhatsengonul.wordpress.com/2010/08/09/getting-the-most-from-hybrid-columnar-compression/

SORT COMPRESS SIZE (GB) RATIO

NOSORT NOCOMP 137,59 1

NOSORT Q_HIGH 21,21 6,48

SORT_A Q_HIGH 12,18 11,29

SORT_B Q_HIGH 15,37 8,95

SORT_A_B Q_HIGH 11,64 11,80

Performance Gains

Report Name Old System Exadata by…X

CRC Control Report 0:15:48.73 0:05:06.07 X2

prepaid bireysel toplam harcanan kontor tl 8:02:10.59 1:51:33.20 X4.3

eom_equipment_utilization report 0:38:17.77 0:00:23.34 X163

Eom equipment arpu report 0:09:46.25 0:02:08.00 X4.5

Son 10 gun icinde yapilan gonderimler 0:17:57.95 0:00:37.61 X45.7

Rapor 01. Satış Kanalı Kırılımında Blackberry 0:03:22.24 0:00:00.66 X487

İnternet Paketleri Postpaid paketiçi ve paket 0:05:41.34 0:01:00.34 X4.3

Connectcard Fatura tutarları 0:31:32.38 0:00:46.51 X66.3

Connectcard Aktivasyon 0:25:21.00 0:00:54.88 X44.9

• Over 50K reports run every month

• Performance improvement is up-to 400x for some reports and on average it is 10x

User Feedbacks

• We heard before that infrastructure changes would give us performance gains but this time we were surprised with it, it was well over our expectations. Now we can take faster actions in this competitive environment. Director of Marketing Department

• XDDS is fantastic in a single word, none of the reports take more than 10 minutes, It was taking 3-4 hours before now it completes in 3 minutes. It sounds like un-real but it is real. Power end-user from Finance Department

• It was a never ending race to match the business' performance and capacity needs. With the Database Machine V2, we have outperformed our user's expectations and we are prepared for the future growth. Veteran System Admin

• “You started to scare me MSTR” updated her status on facebook. End-user from Marketing Department

I had a great holiday

Second Project

• Monthly 1 TB increase in size.

• Need a second RACK.

• Management was satisfied and bought 2 RACKS in stead of one.

• Migration of Data Center from Europe to Asia

• Consolidation on Exadata.

SMARTCUBE - MicroStrategy

Operational Sources

Extract

S

..

..

RDS

ODS CDRDM

Feed

DDS

BIS INFRASTRUCTURE ROADMAP

EXADATA x2-2

SINGLE DWH ENVIRONMENT WITHOUT

DUPLICATION OF DATA

CDRDM 15TB

DDS 35 TB + 18 Ay

DEV DOMAIN 5 TB RDS 5 TB

SAS 60TB HOT SAS 20TB COLD

100TB COLD

Moving the data!

ODS 5 TB

TEST DOMAIN 5 TB

HIGH AVAILABILTY

SOLUTION FOR DWH 50TB

OTHER DBS 25TB

ZDDS 5TB

SINGLE DATABASE

Project Status (Feb 2011)

• RDS was migrated and became XRDS

• Size : 15 TB decreased to 3 TB

• A single tables compression rate 70x (Full of numbers)

• Performance increase 3x even the ETL Server and the DB are on different continents

• And runs in only one database node. (server pool which has only one node.)

Project Status (March 2011)

• CDRDM was migrated and became XCDRDM

• Size : 45 TB decreased to 15 TB

• Our aim is to use this space for ETL server migration.

Project Status (April 2011)

• Existing XDDS has migrated in April.

• ETL servers and reporting servers have migrated simultaneously.

Project Status (End of May 2011)

• ODS will be migrated in May

• The existing v2 system will turn into a high capacity disk system and will be the home of XCDRDM/dev/test and high availability system.

Migration method

• From Sun Solaris to Exadata

• Insert/append over dblink method.

• We still love our inhouse code.

• From Exadata v2 to Exadata x2

• Incr0 backup in Europe / restore in Asia

• Incr1 backup in Europe / restore in Asia

• R/O the source , last incr1 backup and final restore at the target.

• Upgrade and open in Asia.

GAINS and PROJECTION

DBNAME BEFORE AFTER END of 2011 END of 2012

XDDS 35TB 35TB 40TB 50TB

XCDRDM 60TB 13TB 15TB 20TB

XRDS 15TB 3TB 5TB 7TB

ZDDS 15TB 3TB 5TB 7TB

NODS 6TB 2TB 5TB 10TB

ARA TOPLAM

96TB 21TB 30TB 44TB

TOTAL 131TB 56TB 70TB 94TB

• 120 TB (net space) of disk is given back with the first project.

• 100 TB (net space) of disk is given back with the second project.

GAINS on reporting

6,42 min 7,1 min 3,28 min

Avg Time

• Even though we’re using only 8 nodes on X2-2

cluster, we had performance increase.

A little talk about server pools.

• Quality of Service

• 1 sec limit , unaware of parallel DML

• Server Pools

• Still can be used for dividing the nodes between servers.

• We do not want to run 2 different instances on the same node.

• But want to increase or decrease the number of nodes between systems.

• Still want to have the change to get all 16 nodes for 1 database.

What is waiting for the future

• Combining DDS,RDS and ODS together in a single database. (There will be only one database)

• At this moment everybody wants their DBs in exadata.

• All the test and development environments will be consolidated in high capacity SAS system.

Thank YOU

Questions

www.turkcell.com.tr

Ferhat ŞENGÖNÜL http://ferhatsengonul.wordpress.com http://twitter.com/ferhatsengonul