the case (or not) for...

The Case (Or Not) For PostgreSQL

mradcliffe Matthew Radcliffe

Copyright Jeff MacDonald. 2008. Image. http://en.wikipedia.org/wiki/File:Postgresql_elephant.svg

Background• First introduced to PostgreSQL 8 with an ERP

migration from Oracle 8 in 2007.

• Influenced my open source development to try to support PostgreSQL in module development.

• Hobbyist contributor to pgsql driver in Drupal.

• First time to speak at a large conference.

Session Summary

• Briefly review the past.

• Detail present situation for do-ers.

• Review option for removing from core.

• Look at other options.

Here we are again…• PostgreSQL driver has bugs in Drupal 7 and

Drupal 6.

• PostgreSQL driver is completely broken in Drupal 8.

• Should we continue to support PostgreSQL in core?

Edison Wong. DB Support in contribute: is it a good idea? 2008.

Database Abstraction

• Database abstraction is a challenge

• Database abstraction is a challenge.

• The common denominator problem

Douglas Adams. Restaurant at the End of the Universe. 1980. Chapter 1. Kristian Köhntopp. 57 Annotations to Five Common PHP Database Problems. 2006.

4 Types of DBAL

1. Provide access to a database

2. Provide common interface to different software

3. Write portable code

4. Object-relational Mapping (ORM)

Baron Schwartz. Four types of database abstraction layers. 2006. Larry Garfield. Abstracting databases. 2007.



• Framework provide DBAL for different reasons.



• Framework provide DBAL for different reasons.

• Business value

Steve Atkins. What value does ORM add? 2006

Remove PostgreSQL driver support from core if it

• provides small value to Drupal community

• is given low priority to fix critical and architectural issues with the storage interface

Why does this thing exist?

• PostgreSQL is an ancient, strict, free, open source RDBMS.

Past glory and infamy

• Drupal 4

• Drupal 6

• Schema API

• Drupal 7

• DBTNG: A real abstraction layer


• The pgsql driver as a litmus test.



• And that test has consistently failed.

Matthew Radcliffe. Issue #2001350. Drupal cannot be installed on PostgreSQL. 2013



• And that test has consistently failed.

• But we lack the tools to discover and fix it.

State of Drupal DBAL

• Almost a straight port from Drupal 7.

• Exception: Views

• Exception: Storage Abstraction

State of pgsql driver• Issue that killed PostgreSQL: #1167144

• Other bugs: #2279395 (sqlsrv: #2342699)

• No testing infrastructure

• Missing functionality

• schema support

• JSON/PostGIS support

Testing infrastructure

• 2013: Goal to provide testing infrastructure for platform, language, and database systems.

• Good progress, but ran into difficulties.

• 2014: Re-architect testing infrastructure.

• Work lost, had to start over on pgsql test bot.

Yes, YOU can help!

Yes, YOU can help!

• Install PostgreSQL and SQLite on your development environment!

Yes, YOU can help!

• Install PostgreSQL and SQLite on your development environment!

• vagrant: github.com/mradcliffe/drupal8.dev

http://github.com/mradcliffe/drupal8.dev

#2157455: Support PostgreSQL• #998898: Index key length is too long

• #2181291: Transaction handling to mimic implicit commit.

• #1600670: Case sensitivity on column names

• #697220: Test bots for everyone

• #1060476: Support non-public schema.

• #2061879: Implement copyTable or remove it.

#218129: Fake Implicit Commits

• Wrap all queries in a transaction in a save point to emulate behavior in other DBMS.

• Current status: RTBC!

• This issue has masked errors from being discovered by a PostgreSQL test bot (Example: #1837118).

• build_2014_09_30_063525 Exceptions: 208

Shameless PlugModernizing Testbot: The Future of Drupal.org

Automated Testing

Jeremy Thorson Ricardo Amaro !

Tomorrow, 10:45 !

Amsterdam Suite

#2157455: Support PostgreSQL

• #2192877: File list view broken because of ambiguous column “fid”.

Not so distant future

• Drupal 8.0.0

• Drupal 8.x.0 and beyond (9.0.0)


• Drupal 8.0.0

• #2160433: Remove support for PostgreSQL in core


Remove from coreBenefit Risk

Optimize core architecture for MySQL/MySQL-like

Driver is playing catch up, removing from core won’t help

Contrib does not have constraint of core i.e. rewrite all the things.

More difficult to introduce new Schema types such as json.

Don’t need to test all of Drupal - focus on queries

Marginalizing users discourages participation

Optimize Core• Don’t worry about PHP or SQL quirks:

• MySQL-specific features

• Remove hacks that increased code complexity.

• Analyze all queries made in core and optimize for MySQL/MySQL-like exclusively.




Contrib does not have constraint of core




Extend all the things• Just rewrite Core storage classes, cache storage,

entity storage, field storage, all the storage to get rid of SQL assumptions.

• All contrib drivers must do this to some extent.

• Increased security burden on maintainers as low-level systems are replaced.

• Hacking Drupal installer may be necessary for improved PostgreSQL experience.

Károly Négyesi, Daniel Wehner, Larry Garfield. Issue #2302617. Define a standard mechaism [sic] for backend-aware service overrides David Garcia, Matthew Radcliffe. Issue #2342699. ContentEntityDatabaseStorage tries to update identity/serial values by default

Storage only tests• A driver maintainer can focus on tests to make

sure queries and SQL functionality are supported.

• doctrine/dbal has unit testing on MySQL, SQLite, and PostgreSQL (but not the other drivers it supports).

• Runtime: ~30 minutes

Difficult to change core• Issues similar to #1167144: Support querying

against tables that don’t exist is weird.

• We have to work around this and decrease performance for PostgreSQL - #2181291.

• Issue #2279395: NULL != DEFAULT

• Will we run into the same issue with MySQL and MySQL-like drivers as those forks mature?

Playing catch up• Every time core changes, PostgreSQL has to

catch up.

• This won’t change in contrib.

• More work arounds of core behavior decreases performance.

• This may lead to complete non-use of Drupal by PostgreSQL users.






Don’t need to test all of Drupal - focus on unit tests similar to

doctrine/dbalMarginalizing users discourages

participation

Support Nice Features• JSON / PostGIS: Contrib modules would need

to depend on PostgreSQL contrib driver or alter Schema types to take advantage of driver-specific features.

• If we provide core with fallback, like we do now, then it becomes easier to adopt driver features.

Marginalization• We don’t see many PostgreSQL users because the

experience of using PostgreSQL is not great.

• I asked an old Drupal and PostgreSQL user to help review issues, he helps, gets discouraged by the core process, and made some offensive remarks directed at core developers. :-(

• It’s difficult to regain trust and rebuild confidence once it has been lost.

• All that’s left are altruists and dabblers.

How do we get there?

• Patch to remove pgsql driver.

• Take responsibility for community failure.

• Support pgsql driver in contrib by providing resources.

• Drupal 9: Keep the dialogue open.


• Drupal 8.0.0


• Simple/Enhanced drivers

• Other Alternatives

Basic/Enhanced driver• Create a database-agnostic core MySQL driver

for qa.drupal.org and core development.

• Strict typing, SQL standards, Full Group By

• Should there be a non-sql driver as well?

• Create follow-up to #2302617 to create optimizations for SQLite, PostgreSQL, and MySQL/MySQL-like drivers

http://qa.drupal.org

Basic/Enhanced driver

• This approach is really aimed at our current functional Q/A approach.

• It’s more code to maintain.

• But this approach does meet my criteria.

Shameless Plug #2

The Future of Drupal Functional Testing

Lee Rowlan Nick Schuch Konstantin Kuoryashov Cameron Zemeck

!

Tomorrow, 13:00 !

G107 Pantheon

Basic/Enhanced driver

• This approach is really aimed at our current functional Q/A approach.

• It’s more code to maintain.

• But this approach does meet my criteria.

doctrine/dbal• Switch to DBAL - all the drivers

• DBAL has MySQL, PostgreSQL, SQLite tests.

• DBAL has code that is discouraged (regex, string replaces).

• Drupal DBAL is more advanced in some regards (transaction class).

• Or just split our drivers out of drupal/core and into drupal/dbal with their own test bot.

The Future of PostgreSQL• Must be prioritized - History will repeat itself.

• Testing infrastructure is required.

• Be less selfish, take other systems for a spin.

• Make the database abstraction layer and storage abstraction layer maintainable.

• If not, then by all means, let’s get this out of core.

Conversation

• Take a moment and fill out the session evaluations.

• https://amsterdam2014.drupal.org/node/1763

https://amsterdam2014.drupal.org/node/1763

Jeff MacDonald. 2008. Image. http://en.wikipedia.org/wiki/File:Postgresql_elephant.svg !Edison Wong. DB Support in contribute: is it a good idea? 2008. http://hswong3i.net/blog/hswong3i/db-support-contribute-it-good-idea !Douglas Adams. Restaurant at the End of the Universe. 1980. Chapter 1. !Kristian Köhntopp. 57 Annotations to Five Common PHP Database Problems. 2006. http://mysqldump.azundris.com/archives/57-Annotations-to-Five-Common-PHP-database-problems.html !Ralph Schindler. Database abstraction layers must live! 2009. http://ralphschindler.com/2009/07/15/database-abstraction-layers-must-live !Baron Schwartz. Four types of database abstraction layers. 2006. http://www.xaprb.com/blog/2006/08/13/four-types-of-database-abstraction-layers/ !Larry Garfield. Abstracting databases. 2007. http://www.garfieldtech.com/blog/abstracting-databases !Steve Atkins. What value does ORM add? 2006. http://lists.scsys.co.uk/pipermail/catalyst/2006-June/008068.html !Matthew Radcliffe. Issue #2001350. Drupal cannot be installed on PostgreSQL. 2013. https://drupal.org/node/2001350 !Károly Négyesi, Daniel Wehner, Larry Garfield. Issue #2302617. Define a standard mechaism [sic] for backend-aware service overrides. https://drupal.org/node/2302617. !David Garcia, Matthew Radcliffe. Issue #2342699. ContentEntityDatabaseStorage tries to update identity/serial values by default. https://www.drupal.org/node/2342699.

References

Note: There were few primary sources to use for research available to me, but further research into database abstraction layers on an academic level would be interesting.

http://hswong3i.net/blog/hswong3i/db-support-contribute-it-good-idea

http://mysqldump.azundris.com/archives/57-Annotations-to-Five-Common-PHP-database-problems.html

http://ralphschindler.com/2009/07/15/database-abstraction-layers-must-live

http://www.xaprb.com/blog/2006/08/13/four-types-of-database-abstraction-layers/

http://www.garfieldtech.com/blog/abstracting-databases

http://lists.scsys.co.uk/pipermail/catalyst/2006-June/008068.html

https://drupal.org/node/2001350

https://drupal.org/node/2302617

https://www.drupal.org/node/2342699

the case (or not) for...

Documents