addressing vendor weaknesses in user space (robert treat)
DESCRIPTION
TRANSCRIPT
![Page 1: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/1.jpg)
Addressing Vendor Weaknesses in User-Space
ROBERT TREAT,OmniTI
Highload++ 2011
@robtreat2xzilla.net
+Robert Treat1
Monday, October 3, 11
![Page 2: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/2.jpg)
Who Am I?
OMNTI - Internet Scalability ConsultantsLead Database Operations
2
Monday, October 3, 11
![Page 3: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/3.jpg)
Who Am I?
OMNTI - Internet Scalability ConsultantsLead Database Operations
“Large Scale”
3
Monday, October 3, 11
![Page 4: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/4.jpg)
Who Am I?
OMNTI - Internet Scalability ConsultantsLead Database Operations
“Large Scale”
High TransactionsTB+ Data
4
Monday, October 3, 11
![Page 5: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/5.jpg)
Who Am I?
OMNTI - Internet Scalability ConsultantsLead Database Operations
“Large Scale”
High TransactionsTB+ Data
Mission Critical
5
Monday, October 3, 11
![Page 6: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/6.jpg)
Who Am I?
Database Operations @OMNTI
PostgresMySQLOracle& More
6
Monday, October 3, 11
![Page 7: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/7.jpg)
Postgres for Scalability
Traditional RDBMSHighly ExtensibleRuns Everywhere
Talks To Everything“BSD” Licensed
15+ Years DevelopmentOpen Development Community
7
Monday, October 3, 11
![Page 8: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/8.jpg)
The Bloat Problem
Data Footprint Can Be Critical To Performance
8
Monday, October 3, 11
![Page 9: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/9.jpg)
The Bloat Problem
Data Footprint Can Be Critical To Performance
Size On Disk Affects The Needs OfRAM, Disk Speed, Storage
9
Monday, October 3, 11
![Page 10: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/10.jpg)
The Bloat Problem
Data Footprint Can Be Critical To Performance
Size On Disk Affects The Needs OfRAM, Disk Speed, Storage
“Bloat” is unused, wasted disk space, used by the database,
but not needed for actual data storage
10
Monday, October 3, 11
![Page 11: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/11.jpg)
The Bloat Problem
Data Footprint Can Be Critical To Performance
Size On Disk Affects The Needs OfRAM, Disk Speed, Storage
“Bloat” is unused, wasted disk space, taken up by the database,
but not needed for actual data storage
Why?
11
Monday, October 3, 11
![Page 12: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/12.jpg)
MVCC Architecture
Multiversion Concurrency Control (MVCC) allows Postgres to offer high concurrency even during significant database read/write activity. MVCC specifically offers behavior where "readers never block writers, and writers never block readers".
12
Monday, October 3, 11
![Page 13: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/13.jpg)
MVCC Architecture
• Oracle
• MySQL (InnoDB)
• Informix
• Firebird
• MSSQL (optional)
13
Monday, October 3, 11
![Page 14: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/14.jpg)
MVCC Architecture
• Oracle
• MySQL (InnoDB)
• Informix
• Firebird
• MSSQL (optional)
• CouchDB
14
Monday, October 3, 11
![Page 15: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/15.jpg)
“Bloat” Manifests Differently, But Is Common
• MongoDB (deletes, some updates)
• dump/restore
• mongod --repair
• db.runCommand( { compact : 'mycollectionname' } )
• Lucene (updates)
• Hadoop / HDFS (small files)
15
Monday, October 3, 11
![Page 16: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/16.jpg)
Postgres MVCC Architecture
• Implemented Postgres 6.5• 1999, Vadim Mikheev
• MVCC Unmasked • http://momjian.us/main/writings/pgsql/mvcc.pdf
16
Monday, October 3, 11
![Page 17: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/17.jpg)
Postgres MVCC Architecture
• Postgres maintains global transaction counters
• Keeps track of transaction counter per row for• creating transaction• removing transaction
• Using these counters, Postgres allows different transactions to see different rows, based on visibility rules.
17
Monday, October 3, 11
![Page 18: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/18.jpg)
Postgres MVCC Architecture
• Postgres maintains global transaction counters
• Keeps track of transaction counter per row for• creating transaction• removing transaction
• Using these counters, Postgres allows different transactions to see different rows, based on visibility rules.
18
Transaction Reading An Old RowDoesn’t Block Transaction Writing A Row
Monday, October 3, 11
![Page 19: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/19.jpg)
MVCC Architecture
19
user_id X42Create 32 Expire
INSERT
Monday, October 3, 11
![Page 20: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/20.jpg)
MVCC Architecture
20
user_id X42Create 32 Expire
INSERT
DELETEuser_id X42Create 32 Expire 38
Monday, October 3, 11
![Page 21: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/21.jpg)
MVCC Architecture
21
user_id X69Create 43 Expire
user_id X69Create 43 Expire 56
OLD(delete)
NEW(insert)
UPDATE
Monday, October 3, 11
![Page 22: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/22.jpg)
MVCC Architecture
22
user_id X69Create 43 Expire
user_id X69Create 43 Expire 56
<~~ DEAD ROW
<~~ VISIBLE ROW
Clean Up / Bloat
Monday, October 3, 11
![Page 23: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/23.jpg)
MVCC Architecture
23
user_id X69Create 43 Expire
user_id X69Create 43 Expire 56
<~~ DEAD ROW
<~~ VISIBLE ROW
Clean Up / Bloat
Speed Up SQL Commands ByDealing With Clean Up Later
Monday, October 3, 11
![Page 24: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/24.jpg)
How Postgres Deals With Bloat
• Heap-Only-Tuples (HOT)• On-The-Fly, Per Page Cleanup• Marks Given Row’s Space Reusable• Update Only
24
Monday, October 3, 11
![Page 25: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/25.jpg)
How Postgres Deals With Bloat
• Heap-Only-Tuples (HOT)• On-The-Fly, Per Page Cleanup• Marks Given Row’s Space Reusable• Update Only
• VACUUM• Non-Blocking Bulk Cleanup• Removes End-Of-File Pages• “autovacuum” Process Monitors Tables
25
Monday, October 3, 11
![Page 26: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/26.jpg)
Problems With Automatic Cleanup
• HOT• Update Only• Doesn’t Work With Changing Index Data
26
Monday, October 3, 11
![Page 27: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/27.jpg)
Problems With Automatic Cleanup
• HOT• Update Only• Doesn’t Work When Changing Index Data
• VACUUM• Must Wait For Long Transactions To Complete• Costs I/O, Can Only Work So Fast• Can’t Remove Non End-Of-File Pages • Leaves A “High Water Mark”
27
Monday, October 3, 11
![Page 28: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/28.jpg)
Dealing With Bloat - The Hard Way
• VACUUM FULL / CLUSTER• The Good
• Reclaims All “Dead Rows”
28
Monday, October 3, 11
![Page 29: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/29.jpg)
Dealing With Bloat - The Hard Way
• VACUUM FULL / CLUSTER• The Good
• Reclaims All “Dead Rows”
• The Bad• Exclusive Lock• Rewrite All Data In Tables• Needs Working Space• Heavy I/O
29
Monday, October 3, 11
![Page 30: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/30.jpg)
Monitoring Your Bloat
• check_postgres.pl• Nagios plugin• Compares physical size to row size estimates• http://bucardo.org/wiki/Check_postgres
• “bloat report”• Script to measure table/index bloat• Compares physical size to row size estimates• http://labs.omniti.com/labs/pgtreats/browser/trunk/tools/
30
Monday, October 3, 11
![Page 31: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/31.jpg)
Dealing With Bloat In Userspace
• Solving MVCC Bloat Is A “Hard Problem”• Even a good solution would be hard to implement in core
31
Monday, October 3, 11
![Page 32: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/32.jpg)
Dealing With Bloat In Userspace
• Solving MVCC Bloat Is A “Hard Problem”• Even a good solution would be hard to implement in core
• Can we build a tool in user space?• Develop solution quicker• Easier to deploy and maintain • Provide a prototype for future development
32
Monday, October 3, 11
![Page 33: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/33.jpg)
Dealing With Bloat Redux
• Updating A Row Rewrites Data To New Location
33
Monday, October 3, 11
![Page 34: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/34.jpg)
Dealing With Bloat Redux
• Updating A Row Rewrites Data To New Location
• Use Vacuum To Mark Old Rows “Reusable”
34
Monday, October 3, 11
![Page 35: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/35.jpg)
Dealing With Bloat Redux
• Updating A Row Rewrites Data To New Location
• Use Vacuum To Mark Old Rows “Reusable”• Update Row To Rewrite Data At “Front” Of Page
35
Monday, October 3, 11
![Page 36: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/36.jpg)
Dealing With Bloat Redux
• Updating A Row Rewrites Data To New Location
• Use Vacuum To Mark Old Rows “Reusable”• Update Row To Rewrite Data At “Front” Of Page• Use Vacuum To Reclaim Space From End Of File
36
Monday, October 3, 11
![Page 37: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/37.jpg)
Dealing With Bloat Redux
• Updating A Row Rewrites Data To New Location
• Use Vacuum To Mark Old Rows “Reusable”• Update Row To Rewrite Data At “Front” Of Page• Use Vacuum To Reclaim Space From End Of File
• Put A Script On It• https://labs.omniti.com/pgtreats/trunk/tools/compact_table
37
Monday, October 3, 11
![Page 38: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/38.jpg)
Dealing With Bloat Redux
• “Compact Table”• Requires Lots of Time, I/O• Often Causes Heavy Index Bloat• Heavy Concurrency Bloats Faster Than We Can Recover It
38
Monday, October 3, 11
![Page 39: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/39.jpg)
Dealing With Bloat For Real!
• Enter “pg_reorg”
39
Monday, October 3, 11
![Page 40: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/40.jpg)
Dealing With Bloat For Real!
• Enter “pg_reorg”• Vacuum / Cluster Replacement
40
Monday, October 3, 11
![Page 41: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/41.jpg)
Dealing With Bloat For Real!
• Enter “pg_reorg”• Vacuum / Cluster Replacement• Command Line Tool
41
Monday, October 3, 11
![Page 42: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/42.jpg)
Dealing With Bloat For Real!
• Enter “pg_reorg”• Vacuum / Cluster Replacement• Command Line Tool• Online Table Rewrite
• Uses Minimal Locking
42
Monday, October 3, 11
![Page 43: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/43.jpg)
Dealing With Bloat For Real!
• Enter “pg_reorg”• Vacuum / Cluster Replacement• Command Line Tool• Online Table Rewrite
• Uses Minimal Locking• Developed By NTT
43
Monday, October 3, 11
![Page 44: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/44.jpg)
Dealing With Bloat For Real!
• Enter “pg_reorg”• Vacuum / Cluster Replacement• Command Line Tool• Online Table Rewrite
• Uses Minimal Locking• Developed By NTT• BSD Licensed• C Code• http://pgfoundry.org/projects/reorg/
44
Monday, October 3, 11
![Page 45: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/45.jpg)
How pg_reorg Works
45
• Create a log table for changes• Create triggers on the old table to log changes (I/U/D)• Create a new table with a copy of all data in old table• Create all indexes on the new table• Apply all changes from the log table to the new table• Modify the system catalogs information about table files• Drop old table, leaving new table in it’s place
Monday, October 3, 11
![Page 46: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/46.jpg)
How pg_reorg Works
46
• Create a log table for changes• Create triggers on the old table to log changes• Create a new table with a copy of all data in old table• Create all indexes on the new table• Apply all changes from the log table to the new table• MODIFY THE SYSTEM CATALOGS INFORMATION ABOUT THE TABLE FILES (!!!)• Drop old table, leaving the new table in it’s place
Monday, October 3, 11
![Page 47: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/47.jpg)
Dealing With Bloat For Real!
Open Source Code
The Power Is In Your Hands
Look At CodeExamine the SQL
(User Space Is Really Visible)
TEST!
47
Monday, October 3, 11
![Page 48: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/48.jpg)
Dealing With Bloat For Real!
What Does Testing Look Like?
Create Some Tables, Create Artificial Bloat,
run pg_reorg
48
Monday, October 3, 11
![Page 49: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/49.jpg)
Dealing With Bloat For Real!
What Does Testing Look Like?
Create Some Tables, Create Artificial Bloat,
run pg_reorg
WIN!
49
Monday, October 3, 11
![Page 50: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/50.jpg)
Dealing With Bloat For Real!
Test In “Prod”
50
Monday, October 3, 11
![Page 51: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/51.jpg)
Dealing With Bloat For Real!
Test In “Prod”
Find Some Bloated Tables,Make Backup Of Tables,
Cross Fingers, pg_reorg
51
Monday, October 3, 11
![Page 52: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/52.jpg)
Dealing With Bloat For Real!
Test In “Prod”
Find Some Bloated Tables,Make Backup Of Tables,
Cross Fingers, pg_reorg
WIN!
52
Monday, October 3, 11
![Page 53: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/53.jpg)
Dealing With Bloat For Real!
Eventually You Have To Use ItOn Something That Matters
53
Monday, October 3, 11
![Page 54: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/54.jpg)
pg_reorg In The Real World
• Production Database (OLTP) • 540GB Size• 2000 TPS (off-peak time, multiple statements)• Largest Table (pre-reorg) 127GB
54
Monday, October 3, 11
![Page 55: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/55.jpg)
pg_reorg In The Real World
• Production Database (OLTP) • 540GB Size• 2000 TPS (off-peak time, multiple statements)• Largest Table (pre-reorg) 127GB
• Rebuild Stats• 5.75 Hours To Rebuild• Reclaimed 52GB Disk Space • No outages reported for Website/API’s
55
Monday, October 3, 11
![Page 56: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/56.jpg)
pg_reorg In The Real World
56
Monday, October 3, 11
![Page 57: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/57.jpg)
pg_reorg In The Real World
56
Monday, October 3, 11
![Page 58: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/58.jpg)
pg_reorg In The Real World
57
Monday, October 3, 11
![Page 59: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/59.jpg)
pg_reorg In The Real World
57
Monday, October 3, 11
![Page 60: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/60.jpg)
pg_reorg In The Real World
57
Monday, October 3, 11
![Page 61: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/61.jpg)
pg_reorg In The Real World
YAY!
58
Monday, October 3, 11
![Page 62: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/62.jpg)
Return Of The Jedi
59
Monday, October 3, 11
![Page 63: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/63.jpg)
“your overconfidence is your weakness.”
-Luke Skywalker
60
Monday, October 3, 11
![Page 64: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/64.jpg)
“your faith in your friends is yours.”
-Emperor Palpatine
61
Monday, October 3, 11
![Page 65: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/65.jpg)
Sometimes You Can Have Both
Trust in NTT’s Code == faith in friends
Success in production == overconfidence
62
Monday, October 3, 11
![Page 66: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/66.jpg)
When Good pg_reorgs Go Bad!
WARNING: unexpected attrdef record found for attr 61 of rel orders
WARNING: 1 attrdef record(s) missing for rel orders
63
Monday, October 3, 11
![Page 67: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/67.jpg)
When Good pg_reorgs Go Bad!
WARNING: unexpected attrdef record found for attr 61 of rel orders
WARNING: 1 attrdef record(s) missing for rel orders
64
Yes, On A Production SystemYes, Trying To Take 1000’s of Orders Per Second
Monday, October 3, 11
![Page 68: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/68.jpg)
When Good pg_reorgs Go Bad!
create table test ( a int4, b int4 default 2112, c bool);
65
Monday, October 3, 11
![Page 69: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/69.jpg)
When Good pg_reorgs Go Bad!
create table test ( a int4, b int4 default 2112, c bool);
Postgres internals track defaults / constraints based on column position “2”, not column name “b”
66
Monday, October 3, 11
![Page 70: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/70.jpg)
When Good pg_reorgs Go Bad!
create table test ( a int4, b int4 default 2112, c bool);
Postgres internals track defaults / constraints based on column position “2”, not column name “b”
If you drop column “a” and then do pg_reorg, column “c” is now column “2”, and default 2112 is on boolean
67
Monday, October 3, 11
![Page 71: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/71.jpg)
When Good pg_reorgs Go Bad!
create table test ( a int4, b int4 default 2112, c bool);
Postgres internals track defaults / constraints based on column position “2”, not column name “b”
If you drop column “a” and then do pg_reorg, column “c” is now column “2”, and default 2112 is on boolean
This Is Fair - pg_reorg hacks the system tables68
Monday, October 3, 11
![Page 72: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/72.jpg)
When Good pg_reorgs Go Bad!
69
Basic Fix: Drop All Defaults And Recreate
Monday, October 3, 11
![Page 73: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/73.jpg)
When Good pg_reorgs Go Bad!
70
Basic Fix: Drop All Defaults And Recreate
Alternative Fix: Hack System Catalogs Some More
Monday, October 3, 11
![Page 74: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/74.jpg)
When Good pg_reorgs Go Bad!
71
Basic Fix: Drop All Defaults And Recreate
Alternative Fix: Hack System Catalogs Some More
Haven’t we had enough system catalog hacking
for now?
Monday, October 3, 11
![Page 75: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/75.jpg)
When Good pg_reorgs Go Bad!
72
“now, if you'll excuse me, I'll go away and have a heart attack.”
Monday, October 3, 11
![Page 76: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/76.jpg)
What Next?
Report Problem To Mailing ListSubmit A Patch
Ultimately The Problem Is FixedEveryone’s Happy?
73
Monday, October 3, 11
![Page 77: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/77.jpg)
Hackers Discussion
Postgres Development Community Is Funny
Sometimes Hard To Get Them To Recognize Problems
Not Everyone See Online Rebuild As A Big Problem
74
Monday, October 3, 11
![Page 78: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/78.jpg)
Hackers Discussion
Postgres Development Community Is Funny
Sometimes Hard To Get Them To Recognize Problems
Not Everyone See Online Rebuild As A Big Problem
In All The Fairness, Not Everyone Has This Problem
75
Monday, October 3, 11
![Page 79: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/79.jpg)
Hackers Discussion
Hackers Meeting 2011, Discussion On Internal Queuing System
Could Be Used As Underlying Basis For On-Line Rebuilding
Until Then...
76
Monday, October 3, 11
![Page 80: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/80.jpg)
pg_reorg Is A Great Tool!Best Option For Difficult Situation
Just Be Careful!
77
Monday, October 3, 11
![Page 81: Addressing vendor weaknesses in user space (Robert Treat)](https://reader033.vdocuments.site/reader033/viewer/2022042613/547e0196b4af9faf158b5532/html5/thumbnails/81.jpg)
Highload++NTT
OmniTIPostgres Community
Momjian, Depesz, Patel, Kocoloski
xzilla.net@robtreat2
+ Robert Treat
78
THANKS!
Monday, October 3, 11