gpdb.docs.pivotal.io · contents release notes 3 contents chapter 2: pivotal greenplum 6.9 release...

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 6.9

Pivotal Greenplum DatabaseDocumentationRev: A01

© 2020 VMware, Inc.

Copyright Release Notes

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2020 VMware, Inc. or its affiliates. All Rights Reserved.

Revised June 2020 (6.9.0)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents Release Notes

3

Contents

Chapter 2: Pivotal Greenplum 6.9 Release Notes.................................. 14Release 6.9.0.................................................................................................................................... 15

Features.................................................................................................................................. 15Resolved Issues..................................................................................................................... 15Upgrading from Greenplum 6.x to Greenplum 6.9.................................................................16

Deprecated Features.........................................................................................................................17Migrating Data to Greenplum 6........................................................................................................ 18Known Issues and Limitations.......................................................................................................... 19Differences Compared to Open Source Greenplum Database.........................................................21

Chapter 3: Installing and Upgrading Greenplum................................... 22Platform Requirements......................................................................................................................23

Operating Systems................................................................................................................. 23Hardware and Network...........................................................................................................25Storage....................................................................................................................................25Tools and Extensions Compatibility....................................................................................... 26Hadoop Distributions.............................................................................................................. 28

Introduction to Greenplum.................................................................................................................29The Greenplum Master...........................................................................................................30The Segments........................................................................................................................ 30The Interconnect.....................................................................................................................34ETL Hosts for Data Loading.................................................................................................. 36Greenplum Performance Monitoring...................................................................................... 37

Estimating Storage Capacity............................................................................................................. 39Calculating Usable Disk Capacity.......................................................................................... 39Calculating User Data Size.................................................................................................... 40Calculating Space Requirements for Metadata and Logs......................................................40

Configuring Your Systems.................................................................................................................41Disabling SELinux and Firewall Software.............................................................................. 41Recommended OS Parameters Settings............................................................................... 42Synchronizing System Clocks................................................................................................ 49Creating the Greenplum Administrative User.........................................................................50Next Steps.............................................................................................................................. 51

Installing the Greenplum Database Software................................................................................... 52Installing Greenplum Database.............................................................................................. 52(Optional) Installing to a Non-Default Directory..................................................................... 53Enabling Passwordless SSH.................................................................................................. 53Confirming Your Installation................................................................................................... 54About Your Greenplum Database Installation........................................................................55Next Steps.............................................................................................................................. 55

Creating the Data Storage Areas......................................................................................................56Creating Data Storage Areas on the Master and Standby Master Hosts...............................56Creating Data Storage Areas on Segment Hosts.................................................................. 56Next Steps.............................................................................................................................. 57

Validating Your Systems................................................................................................................... 58Validating Network Performance............................................................................................ 58Validating Disk I/O and Memory Bandwidth...........................................................................59

Initializing a Greenplum Database System.......................................................................................60


4

Overview................................................................................................................................. 60Initializing Greenplum Database.............................................................................................60Setting Greenplum Environment Variables............................................................................ 64Next Steps.............................................................................................................................. 65

Installing Optional Extensions........................................................................................................... 66Procedural Language, Machine Learning, and Geospatial Extensions..................................66Python Data Science Module Package..................................................................................66R Data Science Library Package........................................................................................... 70Greenplum Platform Extension Framework (PXF)................................................................. 73

Installing Additional Supplied Modules..............................................................................................74Configuring Timezone and Localization Settings..............................................................................75

Configuring the Timezone...................................................................................................... 75About Locale Support in Greenplum Database..................................................................... 75Character Set Support............................................................................................................77Setting the Character Set.......................................................................................................79Character Set Conversion Between Server and Client..........................................................80

Upgrading to Greenplum 6................................................................................................................83Upgrading from an Earlier Greenplum 6 Release..................................................................83Migrating Data from Greenplum 4.3 or 5 to Greenplum 6..................................................... 86

Enabling iptables (Optional).............................................................................................................. 93Example iptables Rules..........................................................................................................93

Installation Management Utilities.......................................................................................................96Greenplum Environment Variables................................................................................................... 97

Required Environment Variables............................................................................................97Optional Environment Variables............................................................................................. 97

Example Ansible Playbook................................................................................................................99

Chapter 4: Greenplum Database Administrator Guide........................ 101Greenplum Database Concepts...................................................................................................... 102

About the Greenplum Architecture....................................................................................... 102About Management and Monitoring Utilities........................................................................ 104About Concurrency Control in Greenplum Database...........................................................105About Parallel Data Loading................................................................................................ 113About Redundancy and Failover in Greenplum Database...................................................114About Database Statistics in Greenplum Database............................................................. 116

Managing a Greenplum System..................................................................................................... 124About the Greenplum Database Release Version Number................................................. 124Starting and Stopping Greenplum Database....................................................................... 124Accessing the Database.......................................................................................................127Configuring the Greenplum Database System.....................................................................135Enabling Compression..........................................................................................................137Enabling High Availability and Data Consistency Features................................................. 138Backing Up and Restoring Databases................................................................................. 155Expanding a Greenplum System..........................................................................................195Migrating Data with gpcopy..................................................................................................210Monitoring a Greenplum System..........................................................................................210Routine System Maintenance Tasks....................................................................................226Recommended Monitoring and Maintenance Tasks............................................................ 230

Managing Greenplum Database Access.........................................................................................237Configuring Client Authentication......................................................................................... 237Managing Roles and Privileges............................................................................................256

Defining Database Objects..............................................................................................................263Creating and Managing Databases......................................................................................263Creating and Managing Tablespaces...................................................................................265Creating and Managing Schemas........................................................................................267


5

Creating and Managing Tables............................................................................................ 269Choosing the Table Storage Model..................................................................................... 274Partitioning Large Tables......................................................................................................285Creating and Using Sequences........................................................................................... 298Using Indexes in Greenplum Database............................................................................... 301Creating and Managing Views............................................................................................. 305Creating and Managing Materialized Views.........................................................................312

Distribution and Skew..................................................................................................................... 314Local (Co-located) Joins.......................................................................................................314Data Skew............................................................................................................................ 314Processing Skew.................................................................................................................. 315

Inserting, Updating, and Deleting Data...........................................................................................318About Concurrency Control in Greenplum Database...........................................................318Inserting Rows...................................................................................................................... 319Updating Existing Rows........................................................................................................320Deleting Rows.......................................................................................................................320Working With Transactions...................................................................................................320Global Deadlock Detector.....................................................................................................322Vacuuming the Database..................................................................................................... 324Running Out of Locks...........................................................................................................324

Querying Data................................................................................................................................. 326About Greenplum Query Processing....................................................................................326About GPORCA....................................................................................................................329Defining Queries................................................................................................................... 342WITH Queries (Common Table Expressions)......................................................................354Using Functions and Operators............................................................................................358Working with JSON Data..................................................................................................... 369Working with XML Data........................................................................................................382Using Full Text Search.........................................................................................................394Using Greenplum MapReduce............................................................................................. 429Query Performance.............................................................................................................. 437Managing Spill Files Generated by Queries........................................................................ 437Query Profiling...................................................................................................................... 437

Working with External Data.............................................................................................................443Accessing External Data with PXF...................................................................................... 443Defining External Tables...................................................................................................... 443Accessing External Data with Foreign Tables..................................................................... 461Using the Greenplum Parallel File Server (gpfdist)..............................................................470

Loading and Unloading Data.......................................................................................................... 474Loading Data Using an External Table................................................................................ 475Loading and Writing Non-HDFS Custom Data.................................................................... 475Handling Load Errors............................................................................................................478Loading Data with gpload.....................................................................................................480Accessing External Data with PXF...................................................................................... 481Transforming External Data with gpfdist and gpload........................................................... 482Loading Data with COPY..................................................................................................... 492Running COPY in Single Row Error Isolation Mode............................................................493Optimizing Data Load and Query Performance................................................................... 493Unloading Data from Greenplum Database......................................................................... 493Formatting Data Files........................................................................................................... 496Example Custom Data Access Protocol.............................................................................. 499

Managing Performance................................................................................................................... 506Defining Database Performance.......................................................................................... 506Common Causes of Performance Issues............................................................................ 507Greenplum Database Memory Overview............................................................................. 510Managing Resources............................................................................................................514


6

Investigating a Performance Problem.................................................................................. 544

Chapter 5: Greenplum Database Security Configuration Guide......... 547Securing the Database....................................................................................................................548Greenplum Database Ports and Protocols..................................................................................... 549Configuring Client Authentication.................................................................................................... 553

Allowing Connections to Greenplum Database....................................................................553Editing the pg_hba.conf File.................................................................................................555Authentication Methods........................................................................................................ 556SSL Client Authentication.....................................................................................................559PAM-Based Authentication...................................................................................................561Radius Authentication...........................................................................................................562Limiting Concurrent Connections......................................................................................... 563Encrypting Client/Server Connections..................................................................................564

Configuring Database Authorization................................................................................................565Access Permissions and Roles............................................................................................565Managing Object Privileges..................................................................................................565Using SSH-256 Encryption...................................................................................................566Restricting Access by Time..................................................................................................568Dropping a Time-based Restriction.................................................................................... 570

Greenplum Command Center Security........................................................................................... 571Auditing............................................................................................................................................ 574Encrypting Data and Database Connections.................................................................................. 579

Encrypting gpfdist Connections............................................................................................ 579Encrypting Data at Rest with pgcrypto.................................................................................580

Security Best Practices................................................................................................................... 588

Chapter 6: Greenplum Database Best Practices..................................592Best Practices Summary.................................................................................................................593System Configuration...................................................................................................................... 599Schema Design............................................................................................................................... 604

Data Types........................................................................................................................... 604Storage Model...................................................................................................................... 604Compression......................................................................................................................... 605Distributions.......................................................................................................................... 606Partitioning............................................................................................................................ 609Indexes..................................................................................................................................611Column Sequence and Byte Alignment............................................................................... 611

Memory and Resource Management with Resource Groups......................................................... 613Memory and Resource Management with Resource Queues........................................................ 616System Monitoring and Maintenance..............................................................................................620

Monitoring............................................................................................................................. 620Updating Statistics with ANALYZE.......................................................................................621Managing Bloat in a Database.............................................................................................622Monitoring Greenplum Database Log Files..........................................................................626

Loading Data................................................................................................................................... 628INSERT Statement with Column Values..............................................................................628COPY Statement.................................................................................................................. 628External Tables.....................................................................................................................628External Tables with Gpfdist................................................................................................ 628Gpload...................................................................................................................................629Best Practices.......................................................................................................................630

Security............................................................................................................................................ 631Encrypting Data and Database Connections.................................................................................. 634


7

Tuning SQL Queries....................................................................................................................... 643How to Generate Explain Plans........................................................................................... 643How to Read Explain Plans................................................................................................. 643Optimizing Greenplum Queries............................................................................................ 645

High Availability............................................................................................................................... 647Disk Storage......................................................................................................................... 647Master Mirroring....................................................................................................................647Segment Mirroring................................................................................................................ 648Dual Clusters........................................................................................................................ 649Backup and Restore.............................................................................................................649Detecting Failed Master and Segment Instances................................................................ 650Segment Mirroring Configurations........................................................................................651

Chapter 7: Greenplum Database Utility Guide..................................... 656About the Greenplum Database Utilities.........................................................................................657

Referencing IP Addresses....................................................................................................657Running Backend Server Programs.....................................................................................657

Utility Reference.............................................................................................................................. 659analyzedb..............................................................................................................................659clusterdb................................................................................................................................663createdb................................................................................................................................ 665createlang............................................................................................................................. 667createuser............................................................................................................................. 668dropdb................................................................................................................................... 671droplang................................................................................................................................ 673dropuser................................................................................................................................ 674gpactivatestandby................................................................................................................. 675gpaddmirrors......................................................................................................................... 677gpbackup_manager.............................................................................................................. 680gpbackup...............................................................................................................................684gpcheckcat............................................................................................................................ 690gpcheckperf...........................................................................................................................693gpconfig.................................................................................................................................696gpcopy...................................................................................................................................699gpdeletesystem..................................................................................................................... 699gpexpand.............................................................................................................................. 700gpfdist....................................................................................................................................704gpinitstandby......................................................................................................................... 707gpinitsystem.......................................................................................................................... 709gpload................................................................................................................................... 717gplogfilter...............................................................................................................................728gpmapreduce........................................................................................................................ 730gpmapreduce.yaml................................................................................................................732gpmovemirrors...................................................................................................................... 739gppkg.................................................................................................................................... 740gprecoverseg........................................................................................................................ 742gpreload................................................................................................................................ 746gprestore............................................................................................................................... 748gpscp.....................................................................................................................................754gpssh.....................................................................................................................................756gpssh-exkeys........................................................................................................................ 759gpstart................................................................................................................................... 761gpstate.................................................................................................................................. 764gpstop................................................................................................................................... 768pg_config...............................................................................................................................770


8

pg_dump............................................................................................................................... 772pg_dumpall............................................................................................................................781pg_restore............................................................................................................................. 785pgbouncer............................................................................................................................. 790pgbouncer.ini.........................................................................................................................792pgbouncer-admin.................................................................................................................. 804plcontainer.............................................................................................................................812plcontainer Configuration File...............................................................................................817psql........................................................................................................................................821reindexdb.............................................................................................................................. 846vacuumdb..............................................................................................................................847

Additional Supplied Programs......................................................................................................... 850

Chapter 8: Greenplum Database Reference Guide.............................. 851SQL Commands.............................................................................................................................. 852

SQL Syntax Summary..........................................................................................................855ABORT..................................................................................................................................893ALTER AGGREGATE...........................................................................................................894ALTER COLLATION.............................................................................................................895ALTER CONVERSION......................................................................................................... 896ALTER DATABASE.............................................................................................................. 897ALTER DEFAULT PRIVILEGES.......................................................................................... 899ALTER DOMAIN...................................................................................................................901ALTER EXTENSION.............................................................................................................903ALTER EXTERNAL TABLE..................................................................................................906ALTER FOREIGN DATA WRAPPER...................................................................................907ALTER FOREIGN TABLE.................................................................................................... 909ALTER FUNCTION...............................................................................................................912ALTER GROUP.................................................................................................................... 915ALTER INDEX...................................................................................................................... 916ALTER LANGUAGE............................................................................................................. 917ALTER MATERIALIZED VIEW.............................................................................................918ALTER OPERATOR............................................................................................................. 919ALTER OPERATOR CLASS................................................................................................ 920ALTER OPERATOR FAMILY...............................................................................................921ALTER PROTOCOL............................................................................................................. 923ALTER RESOURCE GROUP.............................................................................................. 924ALTER RESOURCE QUEUE...............................................................................................927ALTER ROLE....................................................................................................................... 929ALTER SCHEMA..................................................................................................................934ALTER SEQUENCE............................................................................................................. 934ALTER SERVER.................................................................................................................. 937ALTER TABLE......................................................................................................................938ALTER TABLESPACE..........................................................................................................951ALTER TEXT SEARCH CONFIGURATION.........................................................................952ALTER TEXT SEARCH DICTIONARY................................................................................ 953ALTER TEXT SEARCH PARSER........................................................................................955ALTER TEXT SEARCH TEMPLATE....................................................................................955ALTER TYPE........................................................................................................................956ALTER USER....................................................................................................................... 959ALTER USER MAPPING..................................................................................................... 960ALTER VIEW........................................................................................................................ 960ANALYZE..............................................................................................................................962BEGIN................................................................................................................................... 966CHECKPOINT.......................................................................................................................968


9

CLOSE.................................................................................................................................. 968CLUSTER............................................................................................................................. 969COMMENT............................................................................................................................971COMMIT................................................................................................................................974COPY.................................................................................................................................... 974CREATE AGGREGATE........................................................................................................986CREATE CAST.....................................................................................................................993CREATE COLLATION..........................................................................................................996CREATE CONVERSION...................................................................................................... 997CREATE DATABASE........................................................................................................... 999CREATE DOMAIN..............................................................................................................1001CREATE EXTENSION........................................................................................................1002CREATE EXTERNAL TABLE.............................................................................................1004CREATE FOREIGN DATA WRAPPER..............................................................................1013CREATE FOREIGN TABLE............................................................................................... 1015CREATE FUNCTION..........................................................................................................1017CREATE GROUP............................................................................................................... 1027CREATE INDEX................................................................................................................. 1027CREATE LANGUAGE........................................................................................................ 1032CREATE MATERIALIZED VIEW........................................................................................1034CREATE OPERATOR........................................................................................................ 1036CREATE OPERATOR CLASS........................................................................................... 1040CREATE OPERATOR FAMILY..........................................................................................1044CREATE PROTOCOL........................................................................................................ 1045CREATE RESOURCE GROUP......................................................................................... 1046CREATE RESOURCE QUEUE..........................................................................................1049CREATE ROLE.................................................................................................................. 1052CREATE RULE...................................................................................................................1057CREATE SCHEMA.............................................................................................................1059CREATE SEQUENCE........................................................................................................ 1060CREATE SERVER............................................................................................................. 1063CREATE TABLE.................................................................................................................1064CREATE TABLE AS...........................................................................................................1079CREATE TABLESPACE.....................................................................................................1083CREATE TEXT SEARCH CONFIGURATION....................................................................1084CREATE TEXT SEARCH DICTIONARY............................................................................1085CREATE TEXT SEARCH PARSER...................................................................................1086CREATE TEXT SEARCH TEMPLATE...............................................................................1087CREATE TYPE...................................................................................................................1088CREATE USER.................................................................................................................. 1096CREATE USER MAPPING................................................................................................ 1097CREATE VIEW................................................................................................................... 1098DEALLOCATE.................................................................................................................... 1100DECLARE........................................................................................................................... 1101DELETE.............................................................................................................................. 1103DISCARD............................................................................................................................ 1106DO.......................................................................................................................................1107DROP AGGREGATE..........................................................................................................1109DROP CAST.......................................................................................................................1110DROP COLLATION............................................................................................................ 1111DROP CONVERSION........................................................................................................ 1111DROP DATABASE............................................................................................................. 1112DROP DOMAIN.................................................................................................................. 1113DROP EXTENSION............................................................................................................1114DROP EXTERNAL TABLE.................................................................................................1114DROP FOREIGN DATA WRAPPER..................................................................................1115


10

DROP FOREIGN TABLE................................................................................................... 1116DROP FUNCTION..............................................................................................................1117DROP GROUP................................................................................................................... 1118DROP INDEX..................................................................................................................... 1118DROP LANGUAGE.............................................................................................................1119DROP MATERIALIZED VIEW............................................................................................1120DROP OPERATOR............................................................................................................ 1121DROP OPERATOR CLASS............................................................................................... 1122DROP OPERATOR FAMILY.............................................................................................. 1122DROP OWNED...................................................................................................................1123DROP PROTOCOL............................................................................................................ 1124DROP RESOURCE GROUP..............................................................................................1125DROP RESOURCE QUEUE.............................................................................................. 1126DROP ROLE.......................................................................................................................1127DROP RULE.......................................................................................................................1128DROP SCHEMA................................................................................................................. 1129DROP SEQUENCE............................................................................................................ 1129DROP SERVER..................................................................................................................1130DROP TABLE..................................................................................................................... 1131DROP TABLESPACE.........................................................................................................1132DROP TEXT SEARCH CONFIGURATION........................................................................1133DROP TEXT SEARCH DICTIONARY................................................................................1133DROP TEXT SEARCH PARSER....................................................................................... 1134DROP TEXT SEARCH TEMPLATE...................................................................................1135DROP TYPE....................................................................................................................... 1136DROP USER...................................................................................................................... 1136DROP USER MAPPING.....................................................................................................1137DROP VIEW....................................................................................................................... 1138END.....................................................................................................................................1138EXECUTE........................................................................................................................... 1139EXPLAIN............................................................................................................................. 1140FETCH................................................................................................................................ 1145GRANT................................................................................................................................1147INSERT............................................................................................................................... 1152LOAD.................................................................................................................................. 1155LOCK.................................................................................................................................. 1156MOVE..................................................................................................................................1159PREPARE........................................................................................................................... 1160REASSIGN OWNED...........................................................................................................1162REFRESH MATERIALIZED VIEW..................................................................................... 1163REINDEX............................................................................................................................ 1164RELEASE SAVEPOINT......................................................................................................1166RESET................................................................................................................................ 1167REVOKE............................................................................................................................. 1167ROLLBACK......................................................................................................................... 1170ROLLBACK TO SAVEPOINT.............................................................................................1171SAVEPOINT........................................................................................................................1172SELECT.............................................................................................................................. 1174SELECT INTO.................................................................................................................... 1192SET..................................................................................................................................... 1193SET CONSTRAINTS.......................................................................................................... 1195SET ROLE.......................................................................................................................... 1196SET SESSION AUTHORIZATION..................................................................................... 1197SET TRANSACTION.......................................................................................................... 1198SHOW................................................................................................................................. 1201START TRANSACTION..................................................................................................... 1202


11

TRUNCATE.........................................................................................................................1203UPDATE..............................................................................................................................1205VACUUM.............................................................................................................................1208VALUES.............................................................................................................................. 1211

Data Types.................................................................................................................................... 1214Date/Time Types................................................................................................................ 1216Pseudo-Types..................................................................................................................... 1226Text Search Data Types.................................................................................................... 1228Range Types...................................................................................................................... 1230

Summary of Built-in Functions...................................................................................................... 1235Greenplum Database Function Types................................................................................1235Built-in Functions and Operators........................................................................................1236JSON Functions and Operators......................................................................................... 1239Window Functions.............................................................................................................. 1246Advanced Aggregate Functions......................................................................................... 1248Text Search Functions and Operators............................................................................... 1250Range Functions and Operators........................................................................................ 1254

Additional Supplied Modules......................................................................................................... 1257auto_explain........................................................................................................................1257citext....................................................................................................................................1257dblink...................................................................................................................................1258diskquota.............................................................................................................................1261fuzzystrmatch...................................................................................................................... 1265gp_sparse_vector................................................................................................................1266hstore.................................................................................................................................. 1270orafce.................................................................................................................................. 1270pageinspect.........................................................................................................................1272pgcrypto.............................................................................................................................. 1272sslinfo.................................................................................................................................. 1273

Character Set Support...................................................................................................................1274Setting the Character Set...................................................................................................1276Character Set Conversion Between Server and Client...................................................... 1276

Server Configuration Parameters..................................................................................................1279Parameter Types and Values.............................................................................................1279Setting Parameters............................................................................................................. 1279Parameter Categories.........................................................................................................1280Configuration Parameters...................................................................................................1290

System Catalogs........................................................................................................................... 1373System Tables.................................................................................................................... 1373System Views..................................................................................................................... 1374System Catalogs Definitions...............................................................................................1375

The gp_toolkit Administrative Schema..........................................................................................1480Checking for Tables that Need Routine Maintenance........................................................1480Checking for Locks.............................................................................................................1481Checking Append-Optimized Tables.................................................................................. 1483Viewing Greenplum Database Server Log Files................................................................ 1487Checking Server Configuration Files..................................................................................1490Checking for Failed Segments........................................................................................... 1491Checking Resource Group Activity and Status.................................................................. 1492Checking Resource Queue Activity and Status................................................................. 1496Checking Query Disk Spill Space Usage...........................................................................1498Viewing Users and Groups (Roles)....................................................................................1500Checking Database Object Sizes and Disk Space............................................................ 1500Checking for Uneven Data Distribution.............................................................................. 1504Including Data for Materialized Views................................................................................ 1505

The gpperfmon Database..............................................................................................................1507


12

database_*.........................................................................................................................1509diskspace_*....................................................................................................................... 1510interface_stats_*................................................................................................................ 1510log_alert_*..........................................................................................................................1512queries_*............................................................................................................................. 1513segment_*..........................................................................................................................1515socket_stats_*.....................................................................................................................1516system_*............................................................................................................................. 1517dynamic_memory_info........................................................................................................ 1519memory_info...................................................................................................................... 1519

Server Programmatic Interfaces....................................................................................................1521Greenplum Partner Connector API.................................................................................... 1521Developing a Background Worker Process....................................................................... 1540

SQL Features, Reserved and Key Words, and Compliance........................................................ 1543Summary of Greenplum Features...................................................................................... 1543Reserved Identifiers and SQL Key Words......................................................................... 1553SQL 2008 Optional Feature Compliance........................................................................... 1569

Chapter 9: Greenplum Client and Loader Tools Package................. 1599

Chapter 10: About the Tools Package................................................ 1600

Chapter 11: Installing the Client and Loader Tools Package............ 1601Supported Platforms......................................................................................................................1602Installation Procedure....................................................................................................................1603About Your Installation.................................................................................................................. 1604Running the UNIX Tools Installer................................................................................................. 1605

Prerequisites....................................................................................................................... 1605Procedure............................................................................................................................1605

Running the Windows Tools Installer........................................................................................... 1606Prerequisites....................................................................................................................... 1606Procedure............................................................................................................................1606

Chapter 12: Configuring Greenplum Database for Remote ClientAccess................................................................................................. 1607

Chapter 13: Configuring a Client System for KerberosAuthentication.................................................................................... 1608

Chapter 14: Using the Client and Loader Tools................................. 1609Prerequisites.................................................................................................................................. 1610Setting Up Your Greenplum Database Clients Runtime Environment.......................................... 1611Running the Client and Loader Programs.................................................................................... 1612Greenplum Database Documentation References........................................................................1613Windows Considerations............................................................................................................... 1614

Chapter 15: Client and Loader Utility Reference................................1615


13

Chapter 16: DataDirect ODBC Drivers for Pivotal Greenplum...........1616Prerequisites.................................................................................................................................. 1617Supported Client Platforms........................................................................................................... 1618Installing on Linux Systems.......................................................................................................... 1619

Configuring the Driver on Linux......................................................................................... 1620Testing the Driver Connection on Linux.............................................................................1621

Installing on Windows Systems.................................................................................................... 1623Verifying the Version on Windows..................................................................................... 1623Configuring and Testing the Driver on Windows................................................................1623

DataDirect Driver Documentation..................................................................................................1625

Chapter 17: DataDirect JDBC Driver for Pivotal Greenplum............. 1626Prerequisites.................................................................................................................................. 1627Downloading the DataDirect JDBC Driver.................................................................................... 1628Obtaining Version Details for the Driver....................................................................................... 1629Usage Information......................................................................................................................... 1630Configuring Prepared Statement Execution..................................................................................1631DataDirect Driver Documentation..................................................................................................1632

Pivotal Greenplum 6.9 Release Notes Release Notes

14

Chapter 2

Pivotal Greenplum 6.9 Release Notes

This document contains pertinent release information about Pivotal Greenplum Database 6.9 releases.For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 6 is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


15

Release 6.9.0Release Date: 2020-06-26

Pivotal Greenplum 6.9.0 is a minor release that includes changed features and resolves several issues.

FeaturesGreenplum Database 6.9.0 includes these new and changed features:

• Greenplum Streaming Server (GPSS) version 1.4.0 is included, which introduces many new andchanged features and bug fixes. Refer to the GPSS Release Notes for more information on releasecontent and to access the GPSS documentation.

Note: If you have previously used GPSS in your Greenplum 6.x installation, you are required toperform upgrade actions as described in Upgrading the Streaming Server.

• PXF version 5.13.0 is included, the first PXF release to also provide a separate download package thatenables you to install PXF in a file system location outside of the Greenplum install directory. Referto the PXF Release Notes for more information on release content and supported platforms, and toaccess the PXF documentation.

Resolved IssuesPivotal Greenplum 6.9.0 resolves these issues:

30630 - Segment Mirroring

In some cases during a failover of a segment instance, Greenplum Database returned theFATAL error requested WAL segment WAL_seg_ID has already been removed.Greenplum Database WAL replication incorrectly removed segment files before they wereprocessed during failover. This issue is resolved.

10216 - ALTER TABLE, ALTER DOMAIN

In some cases, heap table data is lost when performing concurrent ALTER TABLE orALTER DOMAIN commands where one command alters a table column and the otherrewrites or redistributes the table data. For example, performing concurrent ALTER TABLEcommands where one command changes a column data type from int to text mightcause data loss. This issue might also occur when altering a table column during thedata distribution phase of a Greenplum system expansion. Greenplum Database did notcorrectly capture the current state of the table during command execution. This issue isresolved.

10224 - ALTER TABLE

For a leaf partition of a partitioned table, ALTER TABLE allowed the distribution policyto be changed to REPLICATED. This is resolved. ALTER TABLE no longer allows thechange.

30647 - Postgres Planner

Some queries that performed multistage aggregation returned results in the incorrectorder. For example, some queries that perform a COUNT in the select list and also containa GROUP BY clause returned results in the incorrect order. This issue is resolved.

10013 - Postgres Planner

In some cases, Greenplum Database generated a PANIC when it encountered a lateralsubquery that included a LIMIT 1 or GROUP BY clause. Greenplum 6.9 resolves thisissue by forcing the gathering and materialization of any relation containing a GROUP BY orLIMIT clause.

https://greenplum.docs.pivotal.io/streaming-server/1-4/gpss-v14-release-notes.htmlhttps://greenplum.docs.pivotal.io/streaming-server/1-4/upgrading-gpss.html../../pxf/5-13/release/pxf_v5_13_release_notes.htmlhttps://github.com/greenplum-db/gpdb/issues/10216https://github.com/greenplum-db/gpdb/issues/10224https://github.com/greenplum-db/gpdb/issues/10013


16

10315 Postgres Planner

For some queries that perform a FULL JOIN using a subselect that contains aCOALESCE function, Greenplum Database returned "ERROR: could not find hashdistribution key expressions in target list". This issue is resolved.

8919 - MPP, Query Execution

Greenplum Database did not properly handle concurrent updating operations to a tablewhen one of the operations moved a table distribution key to another segment instance.Now when a table distribution key is moved to another segment instance, a concurrentupdating operation returns an error.

173243811 - Resource Groups

When resource groups are enabled and a user attempted to move a running query fromone resource group to a resource group configured using the memory_limit=0 with thepg_resgroup_move_query() function, Greenplum Database returned the error ERROR:group doesn't have enough memory on master. This issue isresolved.

172931886 - Transaction System

Greenplum 6.9 resolves an issue where restarting a primary could lead to a segmentprocess hang when there were prepared, but not yet committed or aborted, transactions inprogress at the time of shutdown.

Upgrading from Greenplum 6.x to Greenplum 6.9Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software toGreenplum 6.9.0.

https://github.com/greenplum-db/gpdb/issues/10315https://github.com/greenplum-db/gpdb/issues/8919


17

Deprecated FeaturesDeprecated features will be removed in a future major release of Greenplum Database. Pivotal Greenplum6.x deprecates:

• The gpsys1 utility.• The analzyedb option --skip_root_stats (deprecated since 6.2).

If the option is specified, a warning is issued stating that the option will be ignored.• The server configuration parameter gp_statistics_use_fkeys (deprecated since 6.2).• The server configuration parameter gp_ignore_error_table (deprecated since 6.0).

To avoid a Greenplum Database syntax error, set the value of this parameter to true when yourun applications that execute CREATE EXTERNAL TABLE or COPY commands that include the nowremoved Greenplum Database 4.3.x INTO ERROR TABLE clause.

• Specifying => as an operator name in the CREATE OPERATOR command (deprecated since 6.0).• The Greenplum external table C API (deprecated since 6.0).

Any developers using this API are encouraged to use the new Foreign Data Wrapper API in its place.• Commas placed between a SUBPARTITION TEMPLATE clause and its corresponding SUBPARTITION

BY clause, and between consecutive SUBPARTITION BY clauses in a CREATE TABLE command(deprecated since 6.0).

Using this undocumented syntax will generate a deprecation warning message.• The timestamp format YYYYMMDDHH24MISS (deprecated since 6.0).

This format could not be parsed unambiguously in previous Greenplum Database releases, and is notsupported in PostgreSQL 9.4.

• The createlang and droplang utilities (deprecated since 6.0).• The pg_resqueue_status system view (deprecated since 6.0).

Use the gp_toolkit.gp_resqueue_status view instead.• The GLOBAL and LOCAL modifiers when creating a temporary table with the CREATE TABLE and

CREATE TABLE AS commands (deprecated since 6.0).

These keywords are present for SQL standard compatibility, but have no effect in Greenplum Database.• Using WITH OIDS or oids=TRUE to assign an OID system column when creating or altering a table

(deprecated since 6.0).• Allowing superusers to specify the SQL_ASCII encoding regardless of the locale settings (deprecated

since 6.0).

This choice may result in misbehavior of character-string functions when data that is not encoding-compatible with the locale is stored in the database.

• The @@@ text search operator (deprecated since 6.0).

This operator is currently a synonym for the @@ operator.• The unparenthesized syntax for option lists in the VACUUM command (deprecated since 6.0).

This syntax requires that the options to the command be specified in a specific order.• The plain pgbouncer authentication type (auth_type = plain) (deprecated since 4.x).


18

Migrating Data to Greenplum 6Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Migrating Data from Greenplum 4.3 or 5 for guidelines and considerations for migrating existingGreenplum data to Greenplum 6, using standard backup and restore procedures.


19

Known Issues and LimitationsPivotal Greenplum 6 has these limitations:

• Upgrading a Greenplum Database 4 or 5 release, or Greenplum 6 Beta release, to Pivotal Greenplum 6is not supported.

• MADlib, GPText, and PostGIS are not yet provided for installation on Ubuntu systems.• Greenplum 6 is not supported for installation on DCA systems.• Greenplum for Kubernetes is not yet provided with this release.

The following table lists key known issues in Pivotal Greenplum 6.x.

Table 1: Key Known Issues in Pivotal Greenplum 6.x

Issue Category Description

N/A PXF Starting in 6.x, Greenplum does not bundle cURL and instead loadsthe system-provided library. PXF requires cURL version 7.29.0 ornewer. The officially-supported cURL for the CentOS 6.x and Red HatEnterprise Linux 6.x operating systems is version 7.19.*. GreenplumDatabase 6 does not support running PXF on CentOS 6.x or RHEL 6.x due to this limitation.

Workaround: Upgrade the operating system of your GreenplumDatabase 6 hosts to CentOS 7+ or RHEL 7+, which provides a cURLversion suitable to run PXF.

29703 Loading Datafrom ExternalTables

Due to limitations in the Greenplum Database external tableframework, Greenplum Database cannot log the following types oferrors that it encounters while loading data:

• data type parsing errors• unexpected value type errors• data type conversion errors• errors returned by native and user-defined functions

LOG ERRORS returns error information for data exceptions only.When it encounters a parsing error, Greenplum terminates the loadjob, but it cannot log and propagate the error back to the user via gp_read_error_log().

Workaround: Clean the input data before loading it into GreenplumDatabase.

30594 ResourceManagement

Resource queue-related statistics may be inaccurate in certain cases.Pivotal recommends that you use the resource group resourcemanagement scheme that is available in Greenplum 6.

30522 Logging Greenplum Database may write a FATAL message to the standbymaster or mirror log stating that the database system is in recoverymode when the instance is synchronizing with the master andGreenplum attempts to contact it before the operation completes.Ignore these messages and use gpstate -f output to determine ifthe standby successfully synchronized with the Greenplum master;the command returns Sync state: sync if it is synchronized.


20

Issue Category Description

30537 PostgresPlanner

The Postgres Planner generates a very large query plan that causesout of memory issues for the following type of CTE (common tableexpression) query: the WITH clause of the CTE contains a partitionedtable with a large number partitions, and the WITH reference is usedin a subquery that joins another partitioned table.

Workaround: If possible, use the GPORCA query optimizer. Withthe server configuration parameter optimizer=on, GreenplumDatabase attempts to use GPORCA for query planning andoptimization when possible and falls back to the Postgres Plannerwhen GPORCA cannot be used. Also, the specified type of querymight require a long time to complete.

170824967 gpfidsts For Greenplum Database 6.x, a command that accesses an externaltable that uses the gpfdists protocol fails if the external tabledoes not use an IP address when specifying a host system in theLOCATION clause of the external table definition.

n/a MaterializedViews

By default, certain gp_toolkit views do not display data formaterialized views. If you want to include this information in gp_toolkit view output, you must redefine a gp_toolkit internalview as described in Including Data for Materialized Views.

168957894 PXF The PXF Hive Connector does not support using the Hive* profilesto access Hive transactional tables.

Workaround: Use the PXF JDBC Connector to access Hive.

168548176 gpbackup When using gpbackup to back up a Greenplum Database 5.7.1 orearlier 5.x release with resource groups enabled, gpbackup returnsa column not found error for t6.value AS memoryauditor.

164791118 PL/R PL/R cannot be installed using the deprecated createlang utility,and displays the error:

createlang: language installation failed: ERROR:no schema has been selected to create in

Workaround: Use CREATE EXTENSION to install PL/R, as describedin the documentation.

N/A GreenplumClient/LoadTools onWindows

The Greenplum Database client and load tools on Windows have notbeen tested with Active Directory Kerberos authentication.


21

Differences Compared to Open Source GreenplumDatabase

Pivotal Greenplum 6.x includes all of the functionality in the open source Greenplum Database project andadds:

• Product packaging and installation script• Support for QuickLZ compression. QuickLZ compression is not provided in the open source version of

Greenplum Database due to licensing restrictions.• Support for data connectors:

• Greenplum-Spark Connector• Greenplum-Informatica Connector• Greenplum-Kafka Integration• Greenplum Streaming Server

• Data Direct ODBC/JDBC Drivers• gpcopy utility for copying or migrating objects between Greenplum systems• Support for managing Greenplum Database using Pivotal Greenplum Command Center• Support for full text search and text analysis using Pivotal GPText• Greenplum backup plugin for DD Boost• Backup/restore storage plugin API (Beta)

http://greenplum.org/

Installing and Upgrading Greenplum Release Notes

22

Chapter 3

Installing and Upgrading Greenplum

Information about installing, configuring, and upgrading Greenplum Database software and configuringGreenplum Database host machines.


23

Platform RequirementsThis topic describes the Pivotal Greenplum Database 6 platform and operating system softwarerequirements.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

• Operating Systems

• Software Dependencies• Java

• Hardware and Network• Storage• Tools and Extensions Compatibility

• Client Tools• Extensions• Data Connectors• GPText• Greenplum Command Center

• Hadoop Distributions

Operating SystemsPivotal Greenplum 6 runs on the following operating system platforms:

• Red Hat Enterprise Linux 64-bit 7.x (See the following Note.)• Red Hat Enterprise Linux 64-bit 6.x• CentOS 64-bit 7.x• CentOS 64-bit 6.x• Ubuntu 18.04 LTS

Important: Significant Greenplum Database performance degradation has been observed whenenabling resource group-based workload management on RedHat 6.x and CentOS 6.x systems.This issue is caused by a Linux cgroup kernel bug. This kernel bug has been fixed in CentOS 7.xand Red Hat 7.x systems.

If you use RedHat 6 and the performance with resource groups is acceptable for your use case,upgrade your kernel to version 2.6.32-696 or higher to benefit from other fixes to the cgroupsimplementation.

Note: For Greenplum Database that is installed on Red Hat Enterprise Linux 7.x or CentOS 7.xprior to 7.3, an operating system issue might cause Greenplum Database that is running largeworkloads to hang in the workload. The Greenplum Database issue is caused by Linux kernel bugs.

RHEL 7.3 and CentOS 7.3 resolves the issue.

Greenplum Database server supports TLS version 1.2 on RHEL/CentOS systems, and TLS version 1.3 onUbuntu systems.

Software DependenciesGreenplum Database 6 requires the following software packages on RHEL/CentOS 6/7 systems which areinstalled automatically as dependencies when you install the Pivotal Greenplum Database RPM package):

• apr• apr-util


24

• bash• bzip2• curl• krb5• libcurl• libevent• libxml2• libyaml• zlib• openldap• openssh• openssl• openssl-libs (RHEL7/Centos7)• perl• readline• rsync• R• sed (used by gpinitsystem)• tar• zip

Greenplum Database 6 client software requires these operating system packages:

• apr• apr-util• libyaml• libevent

On Ubuntu systems, Greenplum Database 6 requires the following software packages, which are installedautomatically as dependencies when you install Greenplum Database with the Debian package installer:

• libapr1• libaprutil1• bash• bzip2• krb5-multidev• libcurl3-gnutls• libcurl4• libevent-2.1-6• libxml2• libyaml-0-2• zlib1g• libldap-2.4-2• openssh-client• openssh-client• openssl• perl• readline• rsync• sed• tar• zip• net-tools


25

• less• iproute2

Greenplum Database 6 uses Python 2.7.12, which is included with the product installation (and notinstalled as a package dependency).

Important: SSL is supported only on the Greenplum Database master host system. It cannot beused on the segment host systems.

Important: For all Greenplum Database host systems, SELinux must be disabled. You shouldalso disable firewall software, although firewall software can be enabled if it is required for securitypurposes. See Disabling SELinux and Firewall Software.

JavaGreenplum 6 supports these Java versions for PL/Java and PXF:

• Open JDK 8 or Open JDK 11, available from AdoptOpenJDK• Oracle JDK 8 or Oracle JDK 11

Hardware and NetworkThe following table lists minimum recommended specifications for hardware servers intended to supportGreenplum Database on Linux systems in a production environment. All host servers in your GreenplumDatabase system must have the same hardwar

gpdb.docs.pivotal.io · contents release notes 3 contents chapter 2: pivotal greenplum 6.9 release...

Documents