progress database administration

8/18/2019 Progress database administration

1/17

DB-13: OpenEdge VLDB

Dan Foreman

Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

1

OpenEdgeOpenEdge®®

VLDBVLDB(Very Large(Very Large DataBasesDataBases))Dan ForemanDan Foreman

BravePointBravePoint

[email protected]@prodb.com



2

IntroductionIntroduction -- Dan ForemanDan Foreman

Progress User since 1984Progress User since 1984

Author of: Author of:

Progress Database Administration Guide

Progress Performance Tuning Guide

V10 Database Administration Jumpstart

Virtual System Tables

Pro Dump & Load

ProMonitor


3

Audience Survey Audience Survey

Progress VersionProgress Version

Progress V8

Progress V9

OpenEdge R10.0*

OpenEdge R10.1A

OpenEdge R10.1B


2/17


Dan Foreman



2


4

Audience Survey Audience Survey

LargestLargest SingleSingle Database SizeDatabase Size

> 1tb

> 500gb> 250gb

> 100gb

Everyone else can leave the room


5

Agenda Agenda

Definit ion of VLDBDefinit ion of VLDB

Common Characteristics of VLDBCommon Characteristics of VLDB

Growth Rates and Capacity PlanningGrowth Rates and Capacity Planning

Top Challenges for VLDB CustomersTop Challenges for VLDB Customers

Wish ListWish List

QuestionsQuestions

ConclusionConclusion


6

Definition of VLDBDefinition of VLDB

Minimum of 100gbMinimum of 100gb

Single Database (not a set)Single Database (not a set)

For this presentation I didnFor this presentation I didn’’ ttdifferentiate between allocated spacedifferentiate between allocated spaceand High Water Markand High Water Mark

OpenEdge DB only (no Oracle allowed)OpenEdge DB only (no Oracle allowed)

If the site was not a BravePointIf the site was not a BravePointcustomer, they needed to be willing tocustomer, they needed to be willing toshare various statisticsshare various statistics


3/17


Dan Foreman



3


7

Note about Single DB RequirementNote about Single DB Requirement

Two V9 sites previously had muchTwo V9 sites previously had much

largerlarger DBsDBs but split their VLDB intobut split their VLDB into

multiplemultiple DBsDBs for performance reasonsfor performance reasonsReads per Second beyond a certainReads per Second beyond a certain

point would not improve for a singlepoint would not improve for a single

DB regardless ofDB regardless of – –spin orspin or – –B valuesB values

The “ speed limit” is because certain

actions need to be serialized - for

example, adjusting the LRU chain when

reading a block


8

Units of MeasureUnits of Measure

kbkb == KilobytesKilobytes

mbmb == Megabytes (1000kb)Megabytes (1000kb)

gbgb == Gigabytes (1000mb)Gigabytes (1000mb)

tbtb == Terabytes (1000gb)Terabytes (1000gb)

pbpb == PetabytesPetabytes (1000tb)(1000tb)

ebeb == ExabytesExabytes (1000pb)(1000pb)


9

Progress HistoryProgress History – – DB SizeDB Size LImitsLImits

V8V8

64gb 1k DB Block Size

256gb 8k DB Block Size

V9V9

Maximum Areas: 1,000 (some are reserved)

Area Size: 1k Blk Size & 256 RPB = 8gb

Area Size: 8k Blk Size & 1 RPB = 16tb

995 Areas * 16tb = 15,920tb = 16pb


4/17


Dan Foreman



4


10

Progress HistoryProgress History – – DB Size LimitsDB Size Limi ts

OpenEdge 10OpenEdge 10

32,000 Areas (starting in OE 10.1B)

Max File Size of 1tb * 1024 Extents = 1pb1pb * 32000 Areas approx 32eb


11

Some Progress LimitsSome Progress L imits -- FragmentsFragments

A record can be spl it into two or more A record can be spl it into two or morefragmentsfragments

Each fragment has a ROWID addressEach fragment has a ROWID address

V9 and V10.0* to V10.1AV9 and V10.0* to V10.1A2 billion fragments per Area

OpenEdge 10.1BOpenEdge 10.1B9,223,372,036,854,775,807 (9 quintillionfor 1k DB block & 256 RPB)

A mere 1 or 2.5 quintillion for 4k/8k DBblock sizes


12

Database SizesDatabase Sizes

SiteSite VersionVersion HWMHWM Allocated Al located

Anonymous Anonymous 9.1E029.1E02 1.7 TB1.7 TB 2.0 TB2.0 TB

AHM AHM 9.1E04129.1E0412 743GB743GB 941GB941GB

WachoviaWachovia 9.1E04099.1E0409 359GB359GB 400GB400GB

Broder Broder 10.1A020510.1A0205 290GB290GB 339GB339GB

Toll (OZ)Toll (OZ) 10.0B052010.0B0520 123GB123GB 183GB183GB


5/17


Dan Foreman



5


13

Largest TableLargest Table

SiteSite RecordsRecords SizeSize SameSame

Table?Table?

Anonymous Anonymous 825 million825 million 139gb139gb yesyes

Toll (OZ)Toll (OZ) 147m147m 18gb18gb nono

WachoviaWachovia 202 million202 million 39gb39gb yesyes

AHM AHM 431 million431 million 67gb67gb nono

Broder Broder 81 million81 million 31gb31gb nono


14

Progress VersionProgress Version

Three were on V9.1E*Three were on V9.1E*

One was R10.1A SP02 05One was R10.1A SP02 05

One was R10.0B SP05 20One was R10.0B SP05 20

Al l s ites were 64 Al l s ites were 64--bit Progress exceptbit Progress except

the Anonymous site due to athe Anonymous site due to a

‘‘disagreementdisagreement ’’ about licensingabout licensing


15

Server DemographicsServer Demographics

SiteSite Server Server OSOS RAMRAM CPUCPU

Anon Anon IBM P590IBM P590 AIX 5.2 AIX 5.2 64gb64gb 3232

AHM AHM FujitsuFujitsu

M850M850

Solaris 10Solaris 10 64gb64gb 1616

WachoviaWachovia SunSun

V1280V1280

SolarisSolaris

2.82.832gb32gb 88

Broder Broder IBM P590IBM P590 AIX 5.3 AIX 5.3 32gb32gb 1616

TollToll Sun V890Sun V890 Solaris 10Solaris 10 32gb32gb 88


6/17


Dan Foreman



6


16

Server DemographicsServer Demographics

Where is Windows?Where is Windows?

One site was migrating from Tru64 toOne site was migrating from Tru64 to

Linux the day this presentation wasLinux the day this presentation wasduedue

One of our HP/UX customersOne of our HP/UX customers

submit ted but they left out too manysubmit ted but they left out too many

details to be included in thisdetails to be included in this

presentationpresentation


17

Disk Array DemographicsDisk Array Demographics

IBM DS8300; 8tbIBM DS8300; 8tb

IBM DS4800; 8tb ; 100 disksIBM DS4800; 8tb; 100 disks

EMC DMX; 1tb; 128 disksEMC DMX; 1tb; 128 disks

EMC CX700EMC CX700 – – unknown (under control ofunknown (under control of

corporate administrators)corporate administrators)

HP XP1024 (Hitachi)HP XP1024 (Hitachi )

SunSun StorEdgeStorEdge 9990 (aka HDS 9990)9990 (aka HDS 9990)

12 GB Cache

Disks are 146GB/15K FC

Total about 3TB


18

Disk Array DemographicsDisk Array Demographics

Bottom line: No JBOD is being usedBottom line: No JBOD is being used

in the VLDB worldin the VLDB world


7/17


Dan Foreman



7


19

Concurrent Database ConnectionsConcurrent Database Connections

Anon: Anon: 472472 (355(355 AppSrv AppSrv; 115 W/S); 115 W/S)

4203 actual users4203 actual users

AHM: AHM: 2999 (.2999 (.liclic))Toll:Toll: 1864 (.1864 (.liclic))

Broder Broder :: 1163 (.1163 (.liclic))

Wachovia:Wachovia: 389 (.389 (.liclic))


20

Monitoring ToolsMonitoring Tools

ProMonitor ProMonitor

Fathom ManagementFathom Management

HomegrownHomegrown

ProTopProTop


21

Daily Growth RatesDaily Growth Rates

SiteSite Approx imate Growth per Day Approx imate Growth per Day

AHM AHM 600mb600mb

TollToll 280mb280mb

WachoviaWachovia 250mb250mb

Broder Broder TBATBA


8/17


Dan Foreman



8


22

Capacity PlanningCapacity Planning

No site had a formal capacity planningNo site had a formal capacity planning

processprocess


23

Capacity Planning ToolsCapacity Planning Tools -- CPUCPU

sar sar

nmonnmon (AIX)(AIX)

Adrian Performance Monitor (Solaris) Adrian Performance Monitor (Solaris)

““ User User ’’s Screams Scream””


24

Capacity Planning ToolsCapacity Planning Tools -- RAMRAM

vmstatvmstat

nmonnmon (AIX)(AIX)


9/17


Dan Foreman



9


25

Capacity Planning ToolsCapacity Planning Tools -- DiskDisk

iostatiostat

SAN Vendor SAN Vendor ’’s toolss tools


26

Capacity Planning ToolsCapacity Planning Tools – – DB GrowthDB Growth

Area Status (_ Area Status (_ areastatusareastatus) Reports) Reports

dbanalys + Exceldbanalys + Excel

Fathom ManagementFathom Management

ProMonitor ProMonitor


27

Number of DedicatedNumber of Dedicated DBAsDBAs

Lowest:Lowest: .3 (i.e. 30% of 1 person).3 (i.e. 30% of 1 person)

Highest:Highest: 22


10/17


Dan Foreman



10


28

Backup MethodsBackup Methods

probkupprobkup onlineonline

probkupprobkup online to diskonline to disk

Full each SundayIncremental for the rest of t he week

proquietproquiet + Snap Copy+ Snap Copy

Shutdown, Snap Copy, RestartShutdown, Snap Copy, Restart

Customer was uncomfortable with ‘hot’

backup


29

BackupsBackups -- WachoviaWachovia

Full B/U once a week (5hrs)Full B/U once a week (5hrs)

TwoTwo IncrementalsIncrementals Daily (3hrs each)Daily (3hrs each)

These are a backstop to the occasional

AI gli tch

Warm Spare restarted weeklyWarm Spare restarted weekly

AI f iles are sh ipped hour ly


30

Database ReplicationDatabase Replication

Hot:Hot: Fathom Replication (2)Fathom Replication (2)

Warm:Warm: Af ter Imaging (2) Af ter Imaging (2)

Cool:Cool: Restore from Snap Copy (2)Restore from Snap Copy (2)


11/17


Dan Foreman



11


31

Replication IssuesReplication Issues

Getting Fathom Replication toGetting Fathom Replication to

integrate smoothly wi th Veritasintegrate smoothly wi th Veritas

Cluster Server Cluster Server Long Redo Phase BugLong Redo Phase Bug


32

Maintenance WindowsMaintenance Windows

Anonymous Anonymous

Every Sunday 0300-0400

Everything else is negotiated

TollToll

Weekly DB restart at 0400 on Sunday

morning for 15 minutes

Every 6-8 weeks fo r 6 hours (only if there

is a valid reason – which is normally DBor Appl ication Software Upgrade)


33

Maintenance WindowsMaintenance Windows

AHM AHM

Once a week 2200-0200

Quarterly for Dump/Load

WachoviaWachovia

Weekdays 1700-2300

Weekends by negotiation

Broder Broder

5 minutes every night


12/17


Dan Foreman



12


34

TuningTuning

The recommendation of 10000 * (# ofThe recommendation of 10000 * (# of

CPUs) for settingCPUs) for setting – –spin does not scalespin does not scale

wellwellGenerallyGenerally – –spin should be under 50000spin should be under 50000


35

TuningTuning Anonymous Anonymous

-n 5000

-bibufs 1000 -aibufs 1500

-aistall

-Mn 70 -Mi 3 -Ma 10 -Mpb 50

-L 1500000

-B 160000 (* 8k = 1.28gb)

-t -T /s430/temp/batch

-spin 4000

-tablerangesize 800 -indexrangesize 2000

-semsets 32

-directio

-pinshm-napmax 500

-Bpmax 40000


36

TuningTuning AHM Startup AHM Startup

-B 2000000 (* 8k blocks = 16gb)-spin 25000

-bibufs 130 -aibufs 130

-L 100000

-N tcp -S ntunifipc2

-Mn 1000 -Mi 1 -Ma 50 -Mpb 2 -n 3000

-Mxs 70

-aistall –bistall


-semsets 128

-ServerType Both

-DBService replserv -pica 8192


13/17


Dan Foreman



13


37

TuningTuningBroder Broder

-S liv1

-L 400000

-B 200000

-c 100

-spin 30000

-n 2000

-bibu fs 25 -aibu fs 25

-directio

-semsets 5


-Mn 21 -Mi 2 -Mpb 10


38

TuningTuningWachoviaWachovia

-B 2883584

-bi bufs 30 -aibufs 45-bithold 2048 -aistall

-n 500-N TCP

-L 30000

-Ma 6 -Mn 300 -Mpb 50

-spin 160000

-semsets 30-Bpmax 2048

-Mxs 65536-indexrangesize 2500 -tablerangesize 1000

-SQLTempDisk 4000000 -SQLTempPgSize 20


39

TuningTuningTollToll

-B 600000

-L 1000000

-n 2200

-spin 160000-aibufs 200 -bibufs 120

-aistall -bistall-bithold 8000

-tab lerangesize 700 -indexrangesize 700

-Mn 200 -Mi 5 -Ma 10

-semsets 10-pinshm

-napmax 500

-directio


14/17


Dan Foreman



14


40

Storage AreasStorage Areas

Anonymous Anonymous

560 Storage Areas (Table/Index)

No fixed extentsLarge files enabled (of course)

8k DB Block Size

RPB

256 for smaller Ar eas

64 for large Areas

1 for index Areas


41

Dump/LoadDump/Load

Broder Broder :: Annual Pro D&L Annual Pro D&L

AHM: AHM: Quarterly B inary D&LQuarterly B inary D&L

Wachovia:Wachovia: ““ Are you k idding? Are you k idding?””

Anonymous: Anonymous: D&L by Area or NoneD&L by Area or None


42

Dump/LoadDump/Load -- AHM AHM

Typical outage duration is 24Typical outage duration is 24--36 hours36 hours

Do not dump the entire database. FindDo not dump the entire database. Findthe worst of fenders by tablethe worst of fenders by table(fragmentation or scatter, or a(fragmentation or scatter, or acombination), and then full dump andcombination), and then full dump andload the storage areas that containload the storage areas that containthe majority of these worst offendersthe majority of these worst offenders

The intention is to select a subset ofThe intention is to select a subset ofthe DB that can be completed withinthe DB that can be completed withinthe outage windowthe outage window


15/17


Dan Foreman



15


43


Fathom Replication needs to beFathom Replication needs to be

resyncresync’’dd which means a full backupwhich means a full backup

and restore to a separate machineand restore to a separate machine

With a 743gb DB, that's a 12 hourWith a 743gb DB, that's a 12 hour

backup, and a 14backup, and a 14--16 hour restore16 hour restore


44


While loading should be faster thanWhile loading should be faster thandumping, the limit o f only one loaddumping, the limit o f only one loadper storage area to do an index buildper storage area to do an index buildinlineinline (build indexes),(build indexes), or do multipleor do multipleloads with a full index rebuild after theloads with a full index rebuild after theloads complete, makes loading theloads complete, makes loading thecritical factor in the outage windowcritical factor in the outage window

Only an area that can be fully dumpedOnly an area that can be fully dumped

and loaded in the time frame can beand loaded in the time frame can beprocessedprocessed


45

Top ChallengesTop Challenges

““ 24 hours is not enough24 hours is not enough””

““ Physical Redo Phase in AI rollPhysical Redo Phase in AI roll

forward. It could take seconds,forward. It could take seconds,

minutes or several hours.minutes or several hours.””

Who else has this problem?

Client Performance visibility/trackingClient Performance visibility/tracking

““ It's impossible to add indexes toIt's impossible to add indexes to

some tablessome tables””


16/17


Dan Foreman



16


46

Top ChallengesTop Challenges

““ Knowing more about what a programKnowing more about what a program

is doing would be HUGE help. Theis doing would be HUGE help. The

last X number of DB (access)last X number of DB (access)statements executed would be nice.statements executed would be nice.

Like the SQL query plan, but hopefullyLike the SQL query plan, but hopefully

more comprehensible!more comprehensible!””


47

Wish ListWish List

What program is a Client running (#1)What program is a Client running (#1)

Backup by AreaBackup by Area

Anonymous probkup is 14-16 hours

Table Partitioning (AKA HorizontalTable Partitioning (AKA Horizontal

Partitions)Partitions)

Online SQL Permissions ChangesOnline SQL Permissions Changes

Online dump/loadOnline dump/load

Change DB parameters on lineChange DB parameters on line


48

SummarySummary

DonDon ’’t be afraid of OpenEdget be afraid of OpenEdge VLDBsVLDBs

especially wi th OpenEdge 10.1Bespecially wi th OpenEdge 10.1B

VLDB on Windows might be a lonelyVLDB on Windows might be a lonely

placeplace


17/17


Dan Foreman



49

QuestionsQuestions


50

ConclusionConclusion

Thank you for coming!Thank you for coming!

DonDon ’’t forget your evaluations (botht forget your evaluations (both

good and bad)good and bad)

progress database administration

Documents