database storage considerations

Database Storage Considerations

Database Storage Considerations

Adam BackmanAdam Backman

White Star SoftwareWhite Star Software

[email protected]@wss.com

DB-05:DB-05: DB-05:DB-05:

© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations

OutlineOutline

Why is DB storage so important?Why is DB storage so important? Hardware optionsHardware options

RAID vs. JBODRAID vs. JBOD Network Storage (SAN, NAS, …)Network Storage (SAN, NAS, …)

Database setup optionsDatabase setup options Block sizeBlock size Cluster size (DB and BI)Cluster size (DB and BI) Splitting Data into areasSplitting Data into areas


Why is DB Storage So Important?Why is DB Storage So Important?

Everything starts from the diskEverything starts from the disk Disks need to be reliable and fastDisks need to be reliable and fast All physical disks are only capable of doing All physical disks are only capable of doing

approximately 100 operations per secondapproximately 100 operations per second Salespeople are weaselsSalespeople are weasels


Who cares about 100 I/O operations per second?Who cares about 100 I/O operations per second?

Example:Example:

10000 read operations10000 read operations

90% buffer hit rate90% buffer hit rate

1 disk would take 10 seconds to complete 1 disk would take 10 seconds to complete

10 disks would take 1 second to complete10 disks would take 1 second to complete

Which of these would your users choose?Which of these would your users choose?


Stop I/O before it gets to the disksStop I/O before it gets to the disks

Application code Application code Use indexesUse indexes Watch out for can-findWatch out for can-find Bulk updates vs. many small updatesBulk updates vs. many small updates

Use memory not diskUse memory not disk Add system memory (no swapping or paging)Add system memory (no swapping or paging) Increase buffers (-B, -Bt, -Bp)Increase buffers (-B, -Bt, -Bp) Add cache to disk controllersAdd cache to disk controllers


A little more about buffer hit rateA little more about buffer hit rate

Using same example: 10000 Logical readsUsing same example: 10000 Logical reads

90% buffer hit rate = 1000 Physical reads90% buffer hit rate = 1000 Physical reads



See the effect buffer efficiency has on disks! See the effect buffer efficiency has on disks!


What causes disk I/O?What causes disk I/O?

Operating system (swapping and paging)Operating system (swapping and paging) ProgressProgress

Database (DB and BI)Database (DB and BI) Application (code and temp files)Application (code and temp files)

Other applicationsOther applications


Hardware OptionsHardware Options

RAID vs. JBODRAID vs. JBOD Buying disksBuying disks What is varianceWhat is variance Simplicity and cost vs. control and budget Simplicity and cost vs. control and budget

Network vs. Locally Attached StorageNetwork vs. Locally Attached Storage DefinitionsDefinitions ConfigurationConfiguration Pros and ConsPros and Cons


Buying DisksBuying Disks

Buy small disksBuy small disksEach disk regardless of it’s size is capable of Each disk regardless of it’s size is capable of

doing the same number of I/Os per seconddoing the same number of I/Os per second Buy fast disksBuy fast disks

Slow disk = slow performanceSlow disk = slow performance Buy reliable disksBuy reliable disks Buy many disksBuy many disks

The inner portion of the disk is 20% faster than The inner portion of the disk is 20% faster than the outer portion of the diskthe outer portion of the disk


Balancing Disk I/OBalancing Disk I/O

Balancing disk I/O is the process of making Balancing disk I/O is the process of making sure you are using all of the available disk sure you are using all of the available disk resources (file systems, disks and resources (file systems, disks and controllers) are working equally as hard at controllers) are working equally as hard at load. This is also called eliminating load. This is also called eliminating variance. variance. A well tuned system will have less A well tuned system will have less than a 15% variance across all disks.than a 15% variance across all disks.


How can I eliminate variance?How can I eliminate variance?

Warning: This is a religious argument. Warning: This is a religious argument. Everyone has an opinion and everyone is Everyone has an opinion and everyone is right.right.

There are two schools of thought.There are two schools of thought. The system should take care of this.The system should take care of this. I will take care of this.I will take care of this.


The system will take care of this.The system will take care of this.

Disk I/O can be effectively distributed by the Disk I/O can be effectively distributed by the operating system with the use of smart I/O operating system with the use of smart I/O systems. Most notably RAID. systems. Most notably RAID.

Basic Definition of RAID: Basic Definition of RAID: RRedundant edundant AArray rray of of IIndependent ndependent DDisksisks


What RAID really meansWhat RAID really means

RAID has many levels. I will only cover a RAID has many levels. I will only cover a few.few.

RAID 0: This level is also called RAID 0: This level is also called striping.striping. RAID 1: This is referred to as RAID 1: This is referred to as mirroringmirroring.. RAID 5: Most popular RAID levelRAID 5: Most popular RAID level RAID 10: This is mirroring and striping. RAID 10: This is mirroring and striping.

Also known as RAID 0 + 1Also known as RAID 0 + 1


Network vs. Local StorageNetwork vs. Local Storage

Network storage optionsNetwork storage options Benefits of network storageBenefits of network storage Drawbacks of network storageDrawbacks of network storage


Network Storage OptionsNetwork Storage Options

SAN (Storage Area Network)SAN (Storage Area Network)A dedicated storage network supporting multiple A dedicated storage network supporting multiple

computerscomputers NAS (Network Attached Storage)NAS (Network Attached Storage)

A dedicated storage device on an existing network A dedicated storage device on an existing network including a SANincluding a SAN

iSCSIiSCSIA storage networking protocol allows storage traffic to A storage networking protocol allows storage traffic to

travel through general-purpose networks instead of the travel through general-purpose networks instead of the specialized Fibre Channel SANspecialized Fibre Channel SAN


Benefits of Network StorageBenefits of Network Storage

Centralized data store - buy a SAN, hook up Centralized data store - buy a SAN, hook up many serversmany servers

Easier to maintain (backup one place) Easier to maintain (backup one place)


Drawbacks to Network StorageDrawbacks to Network Storage

More complex performance monitoringMore complex performance monitoringMany servers all making requests at the same time make it Many servers all making requests at the same time make it

difficult to resolve conflictsdifficult to resolve conflicts Overall more complexOverall more complex

More things to breakMore things to break Single point of failure Single point of failure

Many vendors support array to array mirroring to resolve Many vendors support array to array mirroring to resolve this issuethis issue

Generally more expensiveGenerally more expensiveUnless you have a large number of servers using the Unless you have a large number of servers using the

network storagenetwork storage


Creative Hardware Solution: Solid State DisksCreative Hardware Solution: Solid State Disks

PROsPROs ReliableReliable FastFast

CONsCONs ExpensiveExpensive Really expensiveReally expensive

Conclusion: Good for logs (AI and BI) and Conclusion: Good for logs (AI and BI) and limited portions (hot spots) of the databaselimited portions (hot spots) of the database


I will take care of thisI will take care of this

Manual spread of data across disksManual spread of data across disks Better control as you can see where the I/O Better control as you can see where the I/O

is goingis going More attention by system administrator is More attention by system administrator is

neededneeded Administrator must think about reliability Administrator must think about reliability

as wellas well


Database Setup OptionsDatabase Setup Options

Database StuffDatabase Stuff Storage areasStorage areas Blocksize and Records per blockBlocksize and Records per block Type II areasType II areas

BI cluster sizeBI cluster size Effect of BI cluster sizeEffect of BI cluster size Why are APWs affected by BI Cluster sizeWhy are APWs affected by BI Cluster size


Storage AreasStorage Areas

BenefitsBenefits Greater control of location of dataGreater control of location of data Minimize downtime for utilitiesMinimize downtime for utilities Stripe some, leave some on straight disksStripe some, leave some on straight disks

DrawbacksDrawbacks More things to breakMore things to break More complex to monitorMore complex to monitor


Storage Areas - ControlStorage Areas - Control

A Storage Area can hold 1 or more data A Storage Area can hold 1 or more data objects (index, table, schema, …)objects (index, table, schema, …)

Separate schema from data if possibleSeparate schema from data if possible Isolate the logs (AI and BI)Isolate the logs (AI and BI) Try to keep the number of areas Try to keep the number of areas

manageable, only add more areas for valid manageable, only add more areas for valid business reasonsbusiness reasons


Database Block SizeDatabase Block Size

8k for most (4k for NT)8k for most (4k for NT) More in synch with the operating systemMore in synch with the operating system More stuff per readMore stuff per read Especially good for index readsEspecially good for index reads Watch the number of records per block so Watch the number of records per block so

you don’t waste block spaceyou don’t waste block space


Records per BlockRecords per Block

Can be set to a binary number between 1 Can be set to a binary number between 1 and 256and 256

If it is set too low you will waste space at If it is set too low you will waste space at the end of each blockthe end of each block

If you set it too high you run the risk of If you set it too high you run the risk of fragmenting records (most true for records fragmenting records (most true for records that grow over time)that grow over time)

More important to get right for type II areasMore important to get right for type II areas


Setting Records per BlockSetting Records per Block

First do a database analysis.First do a database analysis.proutil <dbname> -C dbanalys > filenameproutil <dbname> -C dbanalys > filename

Then take the mean record size of the table(s) Then take the mean record size of the table(s) you are working with and add 20 bytes for you are working with and add 20 bytes for record and block overhead.record and block overhead.

Divide the result above into your block size.Divide the result above into your block size.Last take a binary number just above the Last take a binary number just above the

result above.result above.


Example: Setting Records/BlockExample: Setting Records/Block

Table 1 has a mean record size of 120 bytesTable 1 has a mean record size of 120 bytes Add 20 bytes of overhead (140)Add 20 bytes of overhead (140) Divide block size (8192) by result (140)Divide block size (8192) by result (140)

This (58.51) is how many mean-sized records This (58.51) is how many mean-sized records will fit in a single database block will fit in a single database block

Take the next higher binary number (64)Take the next higher binary number (64)

Are there exceptions, you bet!Are there exceptions, you bet!


Type II AreasType II Areas

The table and index blocks from individual The table and index blocks from individual objects are clustered togetherobjects are clustered together

The cluster size is “tunable” to 8, 64 or 512 The cluster size is “tunable” to 8, 64 or 512 block clustersblock clusters

Benefits: Potential performance benefitsBenefits: Potential performance benefits Drawbacks: Yet another thing to try to get Drawbacks: Yet another thing to try to get

right, potential wasted space right, potential wasted space


How Type II Areas WorkHow Type II Areas Work

The data blocks are clustered together and in turn The data blocks are clustered together and in turn these clusters are chained togetherthese clusters are chained together

Scan type operations need only find the first block Scan type operations need only find the first block and the rest of the blocks will followand the rest of the blocks will follow

Example: When you do an index rebuild, the first Example: When you do an index rebuild, the first step is to delete all of the index blocks. In a type 1 step is to delete all of the index blocks. In a type 1 area all of the blocks in the area are scanned and area all of the blocks in the area are scanned and the index blocks are deleted. In a type II area the the index blocks are deleted. In a type II area the first cluster is put on the free chain and no other first cluster is put on the free chain and no other updates are needed.updates are needed.


Why are APWs Affected by BI Cluster SizeWhy are APWs Affected by BI Cluster Size

The BI cluster size determines the length of The BI cluster size determines the length of the checkpointthe checkpoint

The APW efficiency determines the The APW efficiency determines the efficiency of the checkpointefficiency of the checkpoint

Generally, I start with 1 apw and the Generally, I start with 1 apw and the “correct” BI cluster size and then add 1 apw “correct” BI cluster size and then add 1 apw if I see buffers flushed at checkpoint during if I see buffers flushed at checkpoint during my “important” processing times.my “important” processing times.


BI Cluster SizeBI Cluster Size

The default value of 512 KB is wrong for The default value of 512 KB is wrong for most peoplemost people

Monitor the DB during your high update Monitor the DB during your high update portion of the dayportion of the day

Set the cluster size high enough to keep Set the cluster size high enough to keep checkpoint lengths greater than 2 minutescheckpoint lengths greater than 2 minutes

Generally, 1 MB to 8 MB for most sitesGenerally, 1 MB to 8 MB for most sites


ConclusionConclusion

Look at your storageLook at your storage No RAID 5No RAID 5 RAID or JBODRAID or JBOD To network or not to networkTo network or not to network

Get your DB setup rightGet your DB setup right Block size (remember records per block)Block size (remember records per block) Rational split of your data into storage areasRational split of your data into storage areas Type II areas (where needed)Type II areas (where needed) Get your BI cluster size rightGet your BI cluster size right


Use Both MethodsUse Both Methods

Use mirroring to protect your dataUse mirroring to protect your data Use stripping to get your data onto more Use stripping to get your data onto more

physical drivesphysical drives Use storage areas and extents eliminate Use storage areas and extents eliminate

variancevariance Use storage areas and extents to further Use storage areas and extents to further

spread your dataspread your data


QuestionsQuestions

database storage considerations

Documents