1. Mastering InnoDB Diagnostics Harrison Fisk Senior Principal Technical Support Engineer

2. Overview

Lots of information

• Hard to understand

3. Easy to get overwhelmed

Hard to figure out what is useful

4. Two methods for troubleshooting

• Symptom based

• What we will be doing today

Data based 5. Information Sources 6. Sources

MySQL info

• SHOW ENGINE INNODB STATUS

7. InnoDB monitors

8. SHOW GLOBAL STATUS 9. INFORMATION_SCHEMAtables

• SHOW PROCESSLIST

10. SHOW MUTEX STATUS

OS info

• vmstat / iostator Performance Monitor

11. Gdb ( Poor mans profiler (PMP))

12. OS specific tools 13. SHOW INNODB STATUS

Be careful of truncation

innodb_status_file

14. Table monitors Ensure enough time has passed 15. Exact format varies

InnoDB built-in

16. InnoDB plugin 17. XtraDB 18. Data Dictionary 19. Problems

InnoDB: Error: table test/parent already exists in InnoDB internal InnoDB: data dictionary. Have you deleted the .frm file InnoDB: and not used DROP TABLE? InnoDB: Cannot find table test/child2 from the internal data dictionary InnoDB: of InnoDB though the .frm file for the table exists. Maybe you InnoDB: have deleted and recreated InnoDB data files but have forgotten InnoDB: to delete the corresponding .frm files of InnoDB tables?

Various errors about not being in sync

20. Can't drop and/or create tables or databases 21. Example errors: 22. Diagnostics

InnoDB table monitor

• Dumps InnoDB data dictionary

23. Output to error log

24. CREATE TABLE innodb_table_monitor (id int) ENGINE=innodb; 25. Interpreting

Ignore system tables

NamedSYS_

Ignore hidden fields from InnoDB

• DB_ROW_ID

26. DB_TRX_ID

27. DB_ROLL_PTR

Ignore hidden indexes

GEN_CLUST_INDEX

28. Solutions

Ensure.frmmatches InnoDB

• Create.frmas needed

• Create another table with same structure

29. Copy.frmto the name of the table

Drop orphaned.frmfiles

If not in InnoDB

30. Remove them 31. Crashing 32. Problem

mysqld Crash

• Signal 11

• Segfault

Signal 6

Abort

33. InnoDB loves this one

May not actually notice immediately

• mysqld_safe restarts

34. Diagnostics

Error log

• Hopefully it has info in it

Core file

• Not enabled by default

35. OS often restricts setuid cores

/proc/sys/fs/suid_dumpable

36. coreadm

General log

• When all else fails

37. Interpreting Error log

Error log

• Look for signal type

• Signal 11

38. Signal 6

Backtrace

May or may not be present

OS

39. Version In 5.0 and before you need to resolve it

resolve_stack_dump

Search bug system for function names 40. Interpreting Cores

Must be enabled

41. Memory image produced 42. Requires exactly the same binary and libraries 43. With rpm ensure debuginfo is installed 44. Non-stripped binaries 45. Load with gdb

• thread apply all bt

46. Use PMP to summarize if needed

Solaris:pstack

47. Windows: Minidump/DrWatson/Windbg 48. Solutions

Find cause of the crash

• InnoDB doing on purpose

49. Real crash

Find query causing problem

• Stop doing it

50. Report bug to us

Upgrade

• Lots of bugs fixed each release

51. Locking 52. Problems

Deadlocks

• Real deadlocks

• ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Lock wait timeouts

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

General locking slowness 53. Diagnostics

SHOW INNODB STATUS

• TRANSACTIONSsection

innodb_lock_monitortable

• Changes output ofSHOW INNODB STATUS

INFORMATION_SCHEMA

• INNODB_LOCKS

54. INNODB_LOCK_WAITS

SHOW GLOBAL STATUS LIKE 'innodb_row_lock%'

55. Interpreting

Find what lock is needed

• Who is holding conflicting lock

Look for type of lock

• X (exclusive)

56. S (shared)

Relatively rare

Others

Look for what the lock is on

• Table (primary key)

57. Index

58. Solutions

Fix queries to reduce locks

59. Shared locks

• Find source

60. Eliminate

Upgrade to 5.1 run in read committed

61. Remove source of lock

AUTO-INC

• Upgrade to 5.1

• innodb_autoinc_lock =2?

Long running transactions

62. Performance 63. Overview

Lots of potential performance issues

• Disk I/O

• Log files

64. Table space

CPU

Too many rows being read

65. Thread thrashing

Gather lots of information up front

• Big picture, not details first

Ensure it is during problem time

66. Disk: Log files

Many small serialized synchronous writes

67. InnoDB redo log writes bottleneck

• fdatasync () is expensive in many setups

innodb_flush_log_at_trx_commit

• 1 Safest, sync every commit

68. 2 Write every commit

69. 0 Write only every second 70. Diagnostics

SHOW INNODB STATUS

• State of transactions

• COMMITTED IN MEMORY

71. PREPARED

FILE I/O

Pending flushes (fsync) log

72. fsyncs/s

iostat

• Number of write operations ( w/s )

• awaitand/orsvctm

73. Solutions

InnoDB plugin group commit

74. Changeinnodb_flush_log_at_trx_commit 75. Better disks

• BBU write cache

76. Faster writes

77. Disk: Tablespace

Reading and writing your actual tables and indexes

78. More common with very large tables

• Compare to buffer pool size

Sometimes caused by read ahead in InnoDB

79. Diagnostics

SHOW INNODB STATUS

• Buffer pool hit rate

• Watch absolute values too

FILE I/O

Reads/s

Avg bytes/read shows reada head

Pending reads Read ahead

Pages read ahead/evicted without access

iostat

avgqu-sz

80. await 81. Solutions

Optimize queries

82. Increaseinnodb_buffer_pool_size 83. New LRU algorithm in InnoDB plugin 84. Tune/disable readahead

In plugin, can be tuned

85. In non-plugin

Use gdb hack

86. set srv_startup_is_before_trx_rollback_phase=1 Buy better disks 87. CPU: Lots of Rows

Bad queries can read lots of rows

• Or lots and lots of smaller queries

Data fully cached

• Lots of buffer pool reading

88. Often smaller dataset

Eventually will bottleneck InnoDB on CPU

89. Diagnostics

SHOW INNODB STATUS

• ROW OPERATIONS

• Reads/s

SHOW GLOBAL STATUS LIKE 'Handler%'

• Type of row operations going on

Slow query log

• log-queries-not-using-indexes

90. Solutions

Fix your queries

• Ensure proper index usage

Do less queries

• External caching

Scale reads

• Replication

91. CPU: Thread Thrashing

InnoDB can have issues at high concurrency

92. Later releases help reduce this dramatically

• InnoDB plugin

93. Further versions

XtraDB

94. MySQL 5.5 95. Diagnostics

SHOW INNODB STATUS

• Lots of queued threads

• TRANSACTIONS

96. Queries in queue

• ROW OPERATIONS

• • reads/s is low

vmstat

• Very high CPU usage

97. Lots of cs context switches

SHOW MUTEX STATUS

98. Solutions

innodb_thread_concurrency

• Very high or 0 works best until it doesn't

• Sometimes very low works best

99. A bit of a black art in tuning

innodb_concurrency_tickets

• Size based on number of rows being touched

Use newer version

Lots of work to help this in InnoDB Plugin

100. Questions? 101. The presentation is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle. 102.