1. PostgreSQL 8.4 features

• Andrew Dunstan [email_address] [email_address]

2. Topics

General info and history

3. PostgreSQL general features (briefly) 4. PostgreSQL 8.4 features

Not comprehensive

5. See the release notes A few looks at what's in the pipeline 6. Untopics

Why PostgreSQL is better or worse than

7. Should the name be PostgreSQL or Postgres? 8. GPL vs BSD 9. Four legsUnix good,two legsWindows bad 10. emacs vs. vi 11. GUI vs command line 12. ... 13. Play along

wgetftp://ftp10.us.postgresql.org/pub/postgresql /source/v8.4.1/postgresql-8.4.1.tar.bz2

14. tar -j -xf postgresql-8.4.1.tar.bz2 15. cd postgresql-8.4.1 16. ./configure prefix=`pwd`/../pg84--with-pgport=5678 17. make && make install && cd contrib && make && make install 18. Who uses PostgreSQL?

Yahoo

19. myyearbook 20. Skype 21. Etsy 22. New York Post 23. Afilias 24. Whitepages.com

JourneyX

25. IMDB 26. Rockport 27. Apple 28. NTT 29. Cisco 30. National Weather Service 31. What uses PostgreSQL?

Bugzilla

32. Wikipedia 33. Drupal 34. Bricolage 35. OpenACS 36. Gforge 37. xTuple/OpenRPT

OpenBravo

38. Serendipity 39. PostGIS 40. OpenStreetMap 41. Reddit 42. Trac 43. LedgerSMB 44. PostgreSQL history Original Postgres project started at Berkeley 1986 by Michael Stonebraker SQL added in 1995 Current community project dates from 1996 Several developers from then still active, e.g.Bruce Momjian 45. PostgreSQL License

BSD

46. Nobody owns the code, anyone can use the code 47. No monopoly 48. PostgreSQL philosophy

Stability

49. Safety 50. Correctness 51. Robustness 52. Standards compliance 53. Performance 54. PostgreSQL Features

Multi Version Concurrency Control (MVCC)

Readers don't block writers

55. Writers don't block readers Extensive and extensible type system 56. Joins and Subqueries 57. Foreign keys 58. Namespaces (schemas) 59. Triggers 60. More PostgreSQL features

Stored functions

C

61. Plpgsql

partial clone of plsql

plPerl, plTcl, plPython, plRuby, plJava, plR, ..... Standard modules

pgcrypto, dblink, uuid-ossp, ltree, ....

Transactional DDL 62. Point In Time Recovery 63. PostgreSQL 8.4

Released 1 July 2009

64. 17 months in development 65. Over 200 new features and improvements 66. PostgreSQL 8.4 killer features

Common Table Expressions

67. Window Functions 68. Parallel Restore 69. PostgreSQl 8.4 GBH features

Column permissions

70. Variadic functions 71. Per database locales 72. Significant performance improvements 73. Version aware psql command 74. Common Table Expressions

In SQL standard

Put a query in a CTE and later treat it as a table

with t as ( select a,b,c from foo ) select * from t; 75. with t as ( select a,b,c from foo ), s as (select a,d,e from bar) select b,d from t,s where t.a=s.a; 76. Recursive CTEs

with recursive f as(select 0::numeric as a, 1::numeric as b, 1::int as r union select b, a+b, r+1 from f where r < 100 ) select b from f where r = 100;

77. 354224848179261915075 78. Transitive closure with CTEs

with recursive ancestors as (select 1::int as gen, parent as anc, child as des from childrenunionselect gen+1, anc, childfrom ancestors join childrenon des = parent ) select gen, anc from ancestors where des = 'fred';

79. Forget nested sets similar monstrosities 80. Interesting effects with CTEs create or replace function hanoi(discs integer, move out integer, a out int[], b out int[], c out int[]) returns setof record language sql as $$ with recursive han as ( select 1::int as move, $1 as ndiscs, '{99}'::int[] || array_agg(discs)as a, '{99}'::int[] as b, '{99}'::int[] as c from generate_series($1,1,-1) as discs union all select move + 1 , ndiscs, hnext(move, ndiscs, a,b,c), hnext(move, ndiscs, b,c,a), hnext(move, ndiscs, c,a,b) from han where array_length(b,1) < ndiscs + 1 ) select move, a[2:$1+1] as a, b[2:$1+1] as b, c[2:$1+1] as c from han order by move $$; Select * from hanoi(4); 81. Results: move |a|b|c------+-----------+-----------+--------- 1 | {4,3,2,1} | {}| {} 2 | {4,3,2}| {}| {1} 3 | {4,3}| {2}| {1} 4 | {4,3}| {2,1}| {} 5 | {4}| {2,1}| {3} 6 | {4,1}| {2}| {3} 7 | {4,1}| {}| {3,2} 8 | {4}| {}| {3,2,1} 9 | {}| {4}| {3,2,1} 10 | {}| {4,1}| {3,2} 11 | {2}| {4,1}| {3} 12 | {2,1}| {4}| {3} 13 | {2,1}| {4,3}| {} 14 | {2}| {4,3}| {1} 15 | {}| {4,3,2}| {1} 16 | {}| {4,3,2,1} | {} (16 rows) 82. Sneak Peak: CTE's in 8.5

with t as(delete from foo where bar returning *) select * from t;

83. And similar for insert and update queries 84. Window functions

In SQL standard

85. Similar to aggregates, but does not collapse rows 86. All aggregate functions can be used as window functions 87. Window function examples

select *,row_number() over (order by foo) as rownum from bar order by foo

88. select *, row_number() over t as rownum from bar window t as (order by foo) order by foo 89. (you can omit the outer order by, but it's best not to). 90. More window function examples

select salary,salary / avg(salary) over t, rank() over t from employees window t as( partition by deptorder by salaryrows between unbounded preceding and unbounded following )

91. Major missing SQL features

grouping sets

92. merge 93. Parallel pg_restore

My humble contribution :-)

94. Only works with custom format dumps 95. Uses specified number of connections to the database 96. Especially useful for partitioned databases 97. Typical speedup is around number_of_processors/2 98. Sweet spot is between number_of_processors and 2 *number_of_processors 99. Parallel pg_restore continued

Uses a separate connection for each step

100. Individual steps are not parallelized 101. Uses dependency information to make sure steps are done in right order 102. Parallel clients are forked processes on *nix, threads on Windows 103. pg_restore -d dbname -j 4 dumpfile 104. Parallel pg_restore TODO

Support tar format dumps

105. Parallel pg_dump

Needs snapshot cloning

106. Needs new archive format (directory) Parallel COPY, index build 107. A small digression

Three monkeys on our back

108. All being addressed 109. First monkey on our back

No Upgrade in place

Attempted for 8.4, but many caveats

Parallel pg_restore some help 110. Two efforts underway ... 111. Core team committed to providing less painful upgrade mechanisms

Makes it harder to make improvements in certain areas can't just change on-disk format when we want

112. Replication

Often a box to be checked

113. Three forms:

Statement replay

114. Data replay via triggers

Needs unique key

115. Can cause foreign key constraint problems Log shipping

Uses Write Ahead Log, for a whole cluster

116. Transparent to applications 117. The second monkey on our back

No built-in management for log shipping

Need third party tools or custom scripts to manage

8.5 feature (nearly): Streaming Replication

Developed by NTT in Japan

118. New special built in WALsender daemon 119. After initial backup (rsync) all managed via config file settings 120. Streams log records rather than waiting for complete WAL files much lower latency 121. Third monkey on our back

Can't read from WAL based replicas

Useful for failover, useless for load balancing

122. Many users thus have a combination of log shipping and trigger based replicas 8.5 feature (nearly): Hot Standby

Makes WAL based replicas available for read-only queries

123. Ideal for a report server, decision support etc. 124. Take load off transaction processing system 125. Significant features

Standard module citext: compares text case insensitively

126. Column level privileges

grant select(colname), update(colname) on tabname to rolename;

127. Previously only way to do this was via a view 128. In SQL Standard 129. Locales

Locale is now per-database instead of per-cluster

130. Can be set at time ofcreatedbinstead ofinitdb 131. Ultimate goal is to be able to set locale and charset / encoding for each column

Can involve performance issues

132. Performance improvements

suppress_redundant_updates_trigger

Update normally writes a new record whether or not there is a change

133. This trigger inhibits the update if the new and old records are identical 134. Very fast (uses memcmp() on whole data area) 135. Break even point is at worst around 30%, i.e. you're better off using it if more than 30% of the updates are in fact redundant. 136. Also save vacuuming etc. 137. Performance improvements continued

Relation forks

Information about a relation stored out of line

visibility map fork

Reduces vacuum cost on tables that don't change a lot

free space map fork

Now stored per table and not in shared memory

138. No more config settings required Infrastructure exists for other such forks if required. Hash indexes now much faster

Still not crash safe

139. SQL Features - Arrays

array_agg()

Turn rows into array entries

unnest()

Turn array entries into rows

140. Opposite of array_agg() 141. SQL Features - Views

CREATE OR REPLACE VIEW can now add columns at the end

142. SQL Features limit/offset

No longer require a static expression, can now take a volatile expression or subquery instead

143. SQL Standard syntax now supported:

OFFSET start [ ROW | ROWS ]

144. FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY 145. Function features

Variadic functions

CREATE FUNCTION mleast(VARIADIC numeric[]) RETURNS numeric AS $$

146. SELECT min($1[i]) FROM generate_subscripts($1, 1) g(i); 147. $$ LANGUAGE SQL; select mleast(1,4,9,-6,2,5); 148. Result: -6 149. More Function features

Default argument values

CREATE FUNCTION foo(a int, b int DEFAULT 2, c int DEFAULT 3) RETURNS int LANGUAGE SQL AS$$SELECT $1 + $2 + $3;$$; select foo(10,20,30); -- 60 select foo(10,20); -- 33 select foo(10); -- 15

150. Previously required overloading to get same effect. 151. PL/PGSQL features

Pass parameters to dynamic query

EXECUTE ... USING ...

CASE statement 152. psql features

version-aware queries

Now makes the right queries to suit the server it's connected to

ef function_name 153. Questions?