dm109 a practical guide to ase 11.9.2 optimizer statistics and optdiag eric miner development...

DM109A Practical Guide To ASE 11.9.2 Optimizer Statistics and Optdiag

DM109A Practical Guide To ASE 11.9.2 Optimizer Statistics and OptdiagEric MinerDevelopment EngineerOptimizer [email protected]

11.9.2 Optimizer Statistics

»Statistics now stored in two system tables

» Column level statistics (distribution, density values and date and time of last modification) are stored in sysstatistics.

» Table and Index level statistics are stored in systabstats.

» Space for statistics is limited only by the size of the database.

The New System Tables

»Sysstatistics - Column level statistics

» The date and time of last modification to column statistics. This will tell you when the statistics were changed via create index, update statistics or optdiag.

» The histogram cell boundary values and associated weights.

» The range cell and total density, among others

» Most values are stored in varbinary datatype.

The New System Tables cont.

»Systabstats - Table and Index level statistics

» Number of rows and pages in a table.

» Number of pages in the leaf level of an index, height of an index.

» The ‘raw’ number used to derive the Datapage, Index page and Data row cluster ratios.

» The number of deleted and forwarded rows

» The number of empty datapages and index leaf pages, among others.

Update Statistics - What it is and isn’t

»Update statistics does not guarantee good performance.

»Update statistics does guarantee up to date statistics.

»There are no rules about when to run update statistics.

Changes to Update Statistics

»New syntax and functionality - update statistics table_name(column_name)

» This will create or update column statistics on the column listed. If a column list is given column statistics will be built or updated on the first column listed.

» Multi-column density values will be created or updated for all combinations of columns in the list.

update statistics table_name(colA,colB)

Changes to Update Statistics cont.

update statistics table_name using 100 steps

» This will create column statistics on the table using the specified number of steps (cells) for each column effected.

» Step count can be specified in all update statistics modes.

» Cells consume procedure cache. The larger (width) of the column’s datatype the more cache that is consumed by a cell.


update index statistics table_name [index_name]

» This will create or update column statistics on all columns of all indexes in the table.

» WARNING - This option can take a long time to run. Depending on the number of indexes and the number of columns in each index.

» Statistics on minor index attributes supplies the optimizer with a very accurate picture of the index.

» Add the index name to create or update column statistics for all columns of a single index.


update all statistics table_name

» This will create or update column level statistics of all columns of the table. It will also run update partition statistics.

» It is very helpful to the optimizer to have statistics on minor attributes of indexes, and in some cases on non-indexed columns.

» WARNING - This option can take a LONG time to run depending on the size of the table and the number of columns.


»Update statistics and locking

» No table lock taken by update statistics

» On DOL tables dirty reads (level 0) will be used.

» On APL tables level I will be used.

»Parallel update statistics» Update statistics … with consumers = <value>

» This will indicate the number of consumer tasks to use when sorting data in parallel. This will only apply when specifying a column or set of columns.

Optdiag

»What is optdiag?

» Optdiag is an OS level utility that can be used to read, write and simulate optimizer statistics.

» Technical Support may ask for Optdiag outputs when analyzing your case instead of asking for your dataset.

» Optdiag was designed to make accessing the optimizer statistics easy.

Optdiag cont.

»When will optdiag be most useful?

» When you want to examine column, table and index level statistics.

» When you want to determine if reorg should be run.

» When you need to write statistics directly rather than running update statistics.

» When you want to simulate what effects changes in the statistics will have on the optimization of queries.

Optdiag Formats

»Statistics format optdiag statistics dbname..tablename

» This format outputs optdiag in easy to read and write ASCII format. ‘statistics’ must be in the command line for all formats.

Range cell density: 0.0000083294442894

» This is the format most commonly used in optdiag work.

» Use this format when precision with some datatypes (real, floats, doubles) are not an issue.

Optdiag Formats cont.

»Binary formatoptdiag binary statistics dbname..tablename

» This format outputs column level statistics in both ASCII and binary formats. Table/index level statistics are output in ASCII format.

Range cell density: 0x07c32b1bd677e13e # Range cell density: 0.0000083294442894

» Use this format for maximum precision with some datatypes (float,real, double).

» Use binary format files only on machines with compatible byte ordering of data.

Optdiag Formats cont.

»Simulate format

Data row count:600572.000000000000 (simulated) # Data row count:600572.000000000000 (actual)

» This format can be used to simulate to optimizer statistics.

» The optimizer can use simulated statistics when estimating costs of queries.

» You can use this format to perform ‘what if’ analysis of queries.

Optdiag Files

»Optdiag output files» Use the -o option to get optdiag output files

optdiag statistics db..table -Usa -P -o filename

» The files can be edited using a text editor.

» Use the -i option to rad in the optdiag file optdiag statistics db..table -Usa -P -i filename

» Only column level statistics that do not begin with a # can be edited. Except for optdiag simulate files.

» Keep an unedited copy of the file so that you can go back to the original statistics.

Reading Statistics with Optdiag - Table and Index Level Statistics

»The table and index level statistics

» These statistics are used to describe aspects of tables and indexes.

» Row count, page count, index leaf page count, empty page, deleted and forwarded row counts and the clustering ratios among others.

» These statistics are stored in systabstats.

» Because these statistics are maintained dynamically optdiag cannot overwrite them.

Reading Statistics with Optdiag - Table and Index Level Statistics cont.

»The cluster ratios

» Cluster ratio values are used to estimate the cost of using large I/O and non-clustered index accesses.

» The date page and index page cluster ratios are used in costing queries that can use large I/O.

» The data row cluster ratio is used to estimate the cost of using a non-clustered index for a query.

» The values used in costing are found in the ‘Derived statistics’ section of the optdiag output.


»The cluster ratios cont.» The ‘CR count’ values are used to establish

the ‘Derived statistics’. Derived statistics Data page cluster ratio: 0.0144195618619513 Index page cluster ratio: 0.9975369458128078 Data row cluster ratio: 0.0007887456170921

» The closer to 1, the better the cluster ratio.


»Other table & index level statistics of interestEmpty data page count:

This is the number of empty pages in the table.

Forwarded row count: This is the number of rows that have been moved off of their original page by an update.

Deleted row count: This is the number of rows that have been deleted but whose space has not been reclaimed by reorg or another row.

Empty leaf page count: This is the number of empty pages in the leaf level of the index.

Reading Statistics with Optdiag - Column Level Statistics cont.

»The column level statistics

» These statistics primarily describe the distribution of values in the column. They begin with: Statistics for column: column_name

» Most were previously stored in the single distribution page of each index.

Last update of column statistics: This is a new value now being stored. It is the last date and time the column statistics were modified.


»The density values

» These values describe the average number of duplicates of values in the column.

» Range Cell Density - Range cell density: .0010065599111111

» This is the average number of duplicate values which fall into cells that represent more than one value.

» This value, along with the cell weight, is used in estimating the cost of a SARG value that falls within a range cell.


»The density values cont.

» Total density - Total density: 0.0909101578591566

» This is the average number of duplicates for all values in all cells of the column.

» This value is used when estimating the cost of joins.

» This value will be effected by data skew in the column. You may want to modify it in such cases.


» Total density cont. -

» In most cases a simple arithmetic average for the total density value will eliminate costing problems due to data skew.

» The number of distinct values divided by the total number of rows in the column will give you the arithmetic average.

» Use optdiag to write a new total density value.

» WARNING: Changing the total density value will effect the costing of all joins using the column.


»The default selectivity values

» These values replace the ‘magic numbers’ that were hard coded into previous versions of ASE for open range and in between SARGs.

» These values are used by the optimizer when the value of a SARG is not known at run time.

» For example when local variables are used in a query or a stored procedure where column => @var

» Default values will only exist if the column has statistics.


» The range selectivity Range selectivity: default used (0.33)

» Used to estimate the selectivity of open range SARGs (<,>,<= or >=) when the SARG value is not known at run time.

» The default value is 0.33 or 33% of the column.

» Change this value if your queries contain open range SARGs whose values aren’t known at run time.

» Range selectivity: 0.003300 Make sure to replace the text ‘default used’ with a value.


» The in between selectivity In between selectivity: default used (0.25)

» Used to estimate the selectivity of closed range SARGs (between) when the SARG value is not known at run time.

» The default value is 0.25 or 25% of the column.

» Change this value if your queries contain closed range SARGs whose values aren’t known at run time.

» In between selectivity: 0.002500 Make sure to replace the text ‘default used’ with a value.


»Additional information» Histogram for column: column_name This is the name of the column that the following histogram is on.

» Column datatype: integer This is the datatype of the column, in this case an integer.


» Requested step count: 20 This is the number of steps (histogram cells) specified in create index or update statistics. If you don’t specify a number of steps the default of 20 will be used.

» Actual step count: 18 This is the number of steps that were actually used. This will vary from the requested number at times when the column’s histogram can be built using fewer steps. Often in the case of highly duplicated values.


»The histogram - column distribution statistics.

» Contains the step numbers, the cell weights, the cell boundary value and the operator (indicating the type of cell).

» The steps are the number of boundary values that are read from the column. The cells are the values represented by the boundary values and their corresponding weights.

» The boundary value of a cell includes the upper boundary (in the Value column). The lower boundary of the cell is the previous boundary value plus 1 (or the next most significant bit)


»The histogram cont.

» The first cell of a histogram (step 1) represents NULL values in the column.

» If the weight of the first cell is greater than 0, there are NULL values in the table and the weight is the percentage of NULLs.

Step Weight Value 1 0.00159007 <= 0 2 0.05265480 <= 1057 3 0.05268644 <= 2121 4 0.05265480 <= 3181


Statistics for column: ”l_orderkey” Last update of column statistics:Jan 29 1999 3:09:29:980PM

Range cell density: 0.0000083294442894 Total density: 0.0000083294442894 Range selectivity: default used (0.33) In between selectivity: default used (0.25)

Histogram for column: "l_orderkey” Column datatype: integer Requested step count: 20 Actual step count: 20


»The histogram weights

» Each cells has a weight.

» The weight represents the fraction of the column that is occupied by the values within the cell

» A weight of 1 represents 100% of the rows are occupied by the value.

» A weight of 0 marks the lower boundary of a Frequency Count cell.


»Cell types

» There are two types of cells, Range Cells and Frequency Count Cells.

» A histogram can contain all of one type or a combination of both.

» The type of cell can be determined by its weight and/or it neighboring cells.


»Range cells

» Range Cells represent more than one value in the column.

» The operator of a range cell is ‘<=‘.

» Its weight is generally low.

» The cell weight and the range cell density will be used when SARG values falls into a range cell.

» The distribution of values within a range cell is assumed to be uniform.


»Range cells cont.» Step Weight Value 1 0.00000000 <= 0 2 0.05263982 <= 31425 3 0.05263316 <= 63042 4 0.05263482 <= 94497 5 0.05263316 <= 125959

» Weights are fairly uniform.

» Value is the cell’s upper boundary, lower boundary is previous value +1 (or next most significant bit for some datatypes)

» In this case cell 3 represents all values between 31426 and 63042


»Frequency count cells

» Frequency count cells represent a single highly duplicated value in the column.

» They are the most accurate type of cell since the weight of the cell represents only one value.

» There is no need to assume any distribution within the cell.

» You may want to increase the number of cells to get frequency count cells.


»Frequency count cells cont. Step Weight Value 1 0.00000000 <= 0 2 0.05263500 <= 10564 3 0.05264000 <= 20998 4 0.04577000 < 50000 5 0.09991500 = 50000 6 0.05263000 <= 60551

» In this case the weight of the frequency count cell is high in comparison to the surrounding cells.

» Cell 5 represents the value 50000. Cell 4 represents values between 20999 and 50000.


»Frequency count cells cont. Step Weight Value

1 0.00000000 < 1 2 0.06250000 = 1 3 0.06250000 <= 2 4 0.57500000 <= 3 5 0.06250000 <= 4

» When there is a dense frequency count the cells look like range cells, but the high weight indicates the cell is a frequency count.

» Use the weight to identify highly duplicated values in the column.

Writing Statistics With Optdiag

»When would you want to write statistics with optdiag?

» When the time to run update statistics is prohibitive. Or, when you don’t want update statistics to effect the statistics of a column or set of columns of the table.

» When you want to change the total density value to deal with data skew.

» When you want to change the default selectivity values.

» When you want to add frequency count cells without increasing the number of cells.

Writing Statistics With Optdiag cont.

»Writing statistics using optdiag

» Only the column level statistics can be overwritten using optdiag.

» Table and Index level statistics are dynamic and will be overwritten quickly by ASE.

» WARNING - Keep a copy of your original optdiag output file. If you need to go back to your original statistics you’ll need it.

Writing Statistics With Optdiag cont.

»General tips for writing statistics.

» Know how your dataset changes and if queries are being adversely effected by the statistics.

» Do your query analysis. Use showplan and traceons 302 and 310.

» WARNING - Make changes to statistics only after running tests. Do you actually need to write these statistics with optdiag?

» The step numbers must increase monotonically.

» The weight of a cell must be between 0 and 1.0.

» The sum of the cell weights must be close to 1.0 (0.99 to 1.01)

Optdiag Simulate

»Optdiag can simulate optimizer statistics.

» Optimizer will use simulated statistics rather than the actual statistics to estimate the cost of a query.

» Simulates table, index, and some configuration values.

» Test queries against differing dataset sizes and column statistics during testing.

» Test queries using various cache, parallel and partition configurations.

Optdiag Simulate cont.

»Simulated statistics

» Simulated statistics are stored in a special column of sysstatistics.

» They can be shared and/or table and index level statistic.

» Shared statistics are server wide - cache and parallel configurations.

» Simulated statistics will be used by the optimizer only if set statistics simulate on is run prior to executing a query.


»What can be simulated?

» Cache pool sizes.

» Configured Max parallel degree and Max scan degree.

» Size of largest partition.

» All table and index level statistics.

» Caution - Column level statistics will be written directly to sysstatistics, they are not in simulate format. Keep a copy of original optdiag simulate output files.


»Preparing a simulation

» Recommend that optdiag simulate files are obtained from an existing dataset.

» Run update statistics on the existing dataset before getting optdiag simulate output files.

» If an empty dataset is used -

» Manually build column level statistics.

» Make sure simulated values for table and index statistics are reasonable.

» If an existing dataset is used get showplan and traceons 302 and 310 outputs to use as a baseline comparing to simulation outputs.


»Preparing a simulation cont.

» Have a copy of ASE configuration values from the original server.

» Make sure all tables and indexes are present and identical in the simulation and original databases.

» Make sure all caches in the original database are present in the simulation database.

» Make sure all tables in the simulated database are partitioned the same as in the original.

» Keep copies of original optdiag output files.


»Simulating cache sizes

Size of 2K pool in Kb: 10240 (simulated) #Size of 2K pool in Kb: 10240 (actual) ………………………………………………………………………………………………………………………… Size of 16K pool in Kb: 25600 (simulated) #Size of 16K pool in Kb: 0 (actual)

» Here the cache has a total of 20 Mb of 2K cache. The simulated amount is equal to the actual.

» There is no actual 16K cache available on the original server, but we are simulating 50Mb of 16K cache here.


»Simulating Parallel Strategy Max parallel degree: 10 (simulated)

# Max parallel degree: 1 (actual)

Max scan parallel degree: 10 (simulated) # Max scan parallel degree: 1 (actual)

» Make sure to set these to values that are reasonable for your production server.


»Removing or changing Simulated Statistics

» The shared statistics can be removed by running delete shared statistics

» Other simulated statistics can be changed or reset by inputting an optdiag simulate file.

» Remember that the column level statistics are written directly to sysstatistics and must be overwritten with update statistics or by inputting and optdiag file.


»Simulating Table and Index Statistics

» Table and index statistics are not shared.

» Simulated statistics are written to sysstatistics and must be overwritten to be replaced. Simulate changes in the size of tables and/or indexes.

» Row and page/leaf counts.

» Simulate table fragmentation -

» Empty data/leaf pages

» Forwarded rows

» Deleted rows


»The Cluster Ratios -

» The values used by the optimizer are derived from the CR count values. They are printed by optdiag as Derived Statistics.

» The Derived Statistics cannot be edited.

» Change the CR count values carefully. Test carefully.


»Simulating the Column Level Statistics

» The column level statistics cannot be simulated.

» They are written directly to sysstatistics.

» If changed during a simulation you will need to input an optdiag file change them.

» Run update statistics to reset them to actual values, if a dataset with rows is being used.


»To insure that simulated statistics are used by the optimizer run: set statistics simulate on.

»To verify that simulated statistics were used:

» Showplan output QUERY PLAN FOR STATEMENT 1(at line 1). Optimized using simulated statistics.

» Trace on 302 output. Statistics for this column have been edited. This message appears whenever statistics are changed (edited) via optdiag.

Conclusion

»Optimizer Statistics have changed considerably in 11.9.2

»Optdiag allows you read, write and simulate the statistics.

»This is an advantage to you, but takes some practice and testing to perfect.

»Changing statistics can effect queries that use the statistics - BE CAREFUL.

dm109 a practical guide to ase 11.9.2 optimizer statistics and optdiag eric miner development...

Documents