pivotal greenplum databasecontents release notes 6 managing resources.....514

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 6.7

Pivotal Greenplum DatabaseDocumentationRev: A02

© 2020 Pivotal Software, Inc.

Copyright Release Notes

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2020 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised April 2020 (6.7.1)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents Release Notes

3

Contents

Chapter 2: Pivotal Greenplum 6.7 Release Notes.................................. 14Release 6.7.1.................................................................................................................................... 15Resolved Issues................................................................................................................................ 16Release 6.7.0.................................................................................................................................... 17

Features.................................................................................................................................. 17Resolved Issues..................................................................................................................... 17Upgrading from Greenplum 6.x to Greenplum 6.7.................................................................18

Deprecated Features.........................................................................................................................19Migrating Data to Greenplum 6........................................................................................................ 21Known Issues and Limitations.......................................................................................................... 22Differences Compared to Open Source Greenplum Database.........................................................24

Chapter 3: Installing and Upgrading Greenplum................................... 25Platform Requirements......................................................................................................................26

Operating Systems................................................................................................................. 26Hardware and Network...........................................................................................................28Storage....................................................................................................................................28Tools and Extensions Compatibility....................................................................................... 29Hadoop Distributions.............................................................................................................. 31

Introduction to Greenplum.................................................................................................................33The Greenplum Master...........................................................................................................34The Segments........................................................................................................................ 34The Interconnect.....................................................................................................................38ETL Hosts for Data Loading.................................................................................................. 40Greenplum Performance Monitoring...................................................................................... 41

Estimating Storage Capacity............................................................................................................. 43Calculating Usable Disk Capacity.......................................................................................... 43Calculating User Data Size.................................................................................................... 44Calculating Space Requirements for Metadata and Logs......................................................44

Configuring Your Systems.................................................................................................................45Disabling SELinux and Firewall Software.............................................................................. 45Recommended OS Parameters Settings............................................................................... 46Synchronizing System Clocks................................................................................................ 53Creating the Greenplum Administrative User.........................................................................54Next Steps.............................................................................................................................. 55

Installing the Greenplum Database Software................................................................................... 56Installing Greenplum Database.............................................................................................. 56Enabling Passwordless SSH.................................................................................................. 57Confirming Your Installation................................................................................................... 58About Your Greenplum Database Installation........................................................................58Next Steps.............................................................................................................................. 58

Creating the Data Storage Areas......................................................................................................59Creating Data Storage Areas on the Master and Standby Master Hosts...............................59Creating Data Storage Areas on Segment Hosts.................................................................. 59Next Steps.............................................................................................................................. 60

Validating Your Systems................................................................................................................... 61Validating Network Performance............................................................................................ 61Validating Disk I/O and Memory Bandwidth...........................................................................62


4

Initializing a Greenplum Database System.......................................................................................63Overview................................................................................................................................. 63Initializing Greenplum Database.............................................................................................63Setting Greenplum Environment Variables............................................................................ 67Next Steps.............................................................................................................................. 67

Installing Optional Extensions........................................................................................................... 69Procedural Language, Machine Learning, and Geospatial Extensions..................................69Python Data Science Module Package..................................................................................69R Data Science Library Package........................................................................................... 73Greenplum Platform Extension Framework (PXF)................................................................. 76

Installing Additional Supplied Modules..............................................................................................77Configuring Timezone and Localization Settings..............................................................................78

Configuring the Timezone...................................................................................................... 78About Locale Support in Greenplum Database..................................................................... 78Character Set Support............................................................................................................80Setting the Character Set.......................................................................................................82Character Set Conversion Between Server and Client..........................................................83

Upgrading to Greenplum 6................................................................................................................86Upgrading from an Earlier Greenplum 6 Release..................................................................86Migrating Data from Greenplum 4.3 or 5 to Greenplum 6..................................................... 88

Enabling iptables (Optional).............................................................................................................. 95Example iptables Rules..........................................................................................................95

Installation Management Utilities.......................................................................................................98Greenplum Environment Variables................................................................................................... 99

Required Environment Variables............................................................................................99Optional Environment Variables............................................................................................. 99

Example Ansible Playbook..............................................................................................................101

Chapter 4: Greenplum Database Administrator Guide........................ 103Greenplum Database Concepts...................................................................................................... 104

About the Greenplum Architecture....................................................................................... 104About Management and Monitoring Utilities........................................................................ 107About Concurrency Control in Greenplum Database...........................................................108About Parallel Data Loading................................................................................................ 115About Redundancy and Failover in Greenplum Database...................................................116About Database Statistics in Greenplum Database............................................................. 118

Managing a Greenplum System..................................................................................................... 126About the Greenplum Database Release Version Number................................................. 126Starting and Stopping Greenplum Database....................................................................... 126Accessing the Database.......................................................................................................129Configuring the Greenplum Database System.....................................................................137Enabling Compression..........................................................................................................139Enabling High Availability and Data Consistency Features................................................. 140Backing Up and Restoring Databases................................................................................. 158Expanding a Greenplum System..........................................................................................199Migrating Data with gpcopy..................................................................................................214Monitoring a Greenplum System..........................................................................................214Routine System Maintenance Tasks....................................................................................230Recommended Monitoring and Maintenance Tasks............................................................ 235

Managing Greenplum Database Access.........................................................................................242Configuring Client Authentication......................................................................................... 242Managing Roles and Privileges............................................................................................261

Defining Database Objects..............................................................................................................268Creating and Managing Databases......................................................................................268Creating and Managing Tablespaces...................................................................................270


5

Creating and Managing Schemas........................................................................................272Creating and Managing Tables............................................................................................ 274Choosing the Table Storage Model..................................................................................... 279Partitioning Large Tables......................................................................................................290Creating and Using Sequences........................................................................................... 303Using Indexes in Greenplum Database............................................................................... 306Creating and Managing Views............................................................................................. 310Creating and Managing Materialized Views.........................................................................310

Distribution and Skew..................................................................................................................... 312Local (Co-located) Joins.......................................................................................................312Data Skew............................................................................................................................ 312Processing Skew.................................................................................................................. 313

Inserting, Updating, and Deleting Data...........................................................................................316About Concurrency Control in Greenplum Database...........................................................316Inserting Rows...................................................................................................................... 317Updating Existing Rows........................................................................................................318Deleting Rows.......................................................................................................................318Working With Transactions...................................................................................................318Global Deadlock Detector.....................................................................................................320Vacuuming the Database..................................................................................................... 322Running Out of Locks...........................................................................................................322

Querying Data................................................................................................................................. 324About Greenplum Query Processing....................................................................................324About GPORCA....................................................................................................................327Defining Queries................................................................................................................... 340WITH Queries (Common Table Expressions)......................................................................352Using Functions and Operators............................................................................................356Working with JSON Data..................................................................................................... 367Working with XML Data........................................................................................................380Using Full Text Search.........................................................................................................392Using Greenplum MapReduce............................................................................................. 428Query Performance.............................................................................................................. 436Managing Spill Files Generated by Queries........................................................................ 437Query Profiling...................................................................................................................... 437

Working with External Data.............................................................................................................443Accessing External Data with PXF...................................................................................... 443Defining External Tables...................................................................................................... 443Accessing External Data with Foreign Tables..................................................................... 461Using the Greenplum Parallel File Server (gpfdist)..............................................................470

Loading and Unloading Data.......................................................................................................... 474Loading Data Using an External Table................................................................................ 475Loading and Writing Non-HDFS Custom Data.................................................................... 475Handling Load Errors............................................................................................................478Loading Data with gpload.....................................................................................................480Accessing External Data with PXF...................................................................................... 481Transforming External Data with gpfdist and gpload........................................................... 482Loading Data with COPY..................................................................................................... 492Running COPY in Single Row Error Isolation Mode............................................................493Optimizing Data Load and Query Performance................................................................... 493Unloading Data from Greenplum Database......................................................................... 493Formatting Data Files........................................................................................................... 496Example Custom Data Access Protocol.............................................................................. 499

Managing Performance................................................................................................................... 506Defining Database Performance.......................................................................................... 506Common Causes of Performance Issues............................................................................ 507Greenplum Database Memory Overview............................................................................. 510


6

Managing Resources............................................................................................................514Investigating a Performance Problem.................................................................................. 543

Chapter 5: Greenplum Database Security Configuration Guide......... 546Securing the Database....................................................................................................................547Greenplum Database Ports and Protocols..................................................................................... 548Configuring Client Authentication.................................................................................................... 552

Allowing Connections to Greenplum Database....................................................................552Editing the pg_hba.conf File.................................................................................................554Authentication Methods........................................................................................................ 555SSL Client Authentication.....................................................................................................558PAM-Based Authentication...................................................................................................560Radius Authentication...........................................................................................................561Limiting Concurrent Connections......................................................................................... 562Encrypting Client/Server Connections..................................................................................563

Configuring Database Authorization................................................................................................564Access Permissions and Roles............................................................................................564Managing Object Privileges..................................................................................................564Using SSH-256 Encryption...................................................................................................565Restricting Access by Time..................................................................................................567Dropping a Time-based Restriction.................................................................................... 569

Greenplum Command Center Security........................................................................................... 570Auditing............................................................................................................................................ 573Encrypting Data and Database Connections.................................................................................. 578

Encrypting gpfdist Connections............................................................................................ 578Encrypting Data at Rest with pgcrypto.................................................................................579

Security Best Practices................................................................................................................... 587

Chapter 6: Greenplum Database Best Practices..................................591Best Practices Summary.................................................................................................................592System Configuration...................................................................................................................... 598Schema Design............................................................................................................................... 603

Data Types........................................................................................................................... 603Storage Model...................................................................................................................... 603Compression......................................................................................................................... 604Distributions.......................................................................................................................... 605Partitioning............................................................................................................................ 608Indexes..................................................................................................................................610Column Sequence and Byte Alignment............................................................................... 610

Memory and Resource Management with Resource Groups......................................................... 612Memory and Resource Management with Resource Queues........................................................ 615System Monitoring and Maintenance..............................................................................................619

Monitoring............................................................................................................................. 619Updating Statistics with ANALYZE.......................................................................................620Managing Bloat in the Database..........................................................................................621Monitoring Greenplum Database Log Files..........................................................................625

Loading Data................................................................................................................................... 627INSERT Statement with Column Values..............................................................................627COPY Statement.................................................................................................................. 627External Tables.....................................................................................................................627External Tables with Gpfdist................................................................................................ 627Gpload...................................................................................................................................628Best Practices.......................................................................................................................629

Security............................................................................................................................................ 630


7

Encrypting Data and Database Connections.................................................................................. 633Tuning SQL Queries....................................................................................................................... 642

How to Generate Explain Plans........................................................................................... 642How to Read Explain Plans................................................................................................. 642Optimizing Greenplum Queries............................................................................................ 644

High Availability............................................................................................................................... 646Disk Storage......................................................................................................................... 646Master Mirroring....................................................................................................................646Segment Mirroring................................................................................................................ 647Dual Clusters........................................................................................................................ 648Backup and Restore.............................................................................................................648Detecting Failed Master and Segment Instances................................................................ 649Segment Mirroring Configuration..........................................................................................650

Chapter 7: Greenplum Database Utility Guide..................................... 655About the Greenplum Database Utilities.........................................................................................656

Referencing IP Addresses....................................................................................................656Running Backend Server Programs.....................................................................................656

Utility Reference.............................................................................................................................. 658analyzedb..............................................................................................................................659clusterdb................................................................................................................................663createdb................................................................................................................................ 665createlang............................................................................................................................. 667createuser............................................................................................................................. 668dropdb................................................................................................................................... 671droplang................................................................................................................................ 672dropuser................................................................................................................................ 673gpactivatestandby................................................................................................................. 675gpaddmirrors......................................................................................................................... 677gpbackup_manager.............................................................................................................. 680gpbackup...............................................................................................................................684gpcheckcat............................................................................................................................ 690gpcheckperf...........................................................................................................................693gpconfig.................................................................................................................................696gpcopy...................................................................................................................................699gpdeletesystem..................................................................................................................... 699gpexpand.............................................................................................................................. 700gpfdist....................................................................................................................................704gpinitstandby......................................................................................................................... 707gpinitsystem.......................................................................................................................... 709gpload................................................................................................................................... 716gplogfilter...............................................................................................................................727gpmapreduce........................................................................................................................ 730gpmapreduce.yaml................................................................................................................731gpmovemirrors...................................................................................................................... 738gpperfmon_install..................................................................................................................740gppkg.................................................................................................................................... 744gprecoverseg........................................................................................................................ 745gpreload................................................................................................................................ 749gprestore............................................................................................................................... 751gpscp.....................................................................................................................................758gpssh.....................................................................................................................................759gpssh-exkeys........................................................................................................................ 762gpstart................................................................................................................................... 764gpstate.................................................................................................................................. 766


8

gpstop................................................................................................................................... 770gpsys1...................................................................................................................................773pg_config...............................................................................................................................774pg_dump............................................................................................................................... 776pg_dumpall............................................................................................................................784pg_restore............................................................................................................................. 789pgbouncer............................................................................................................................. 794pgbouncer.ini.........................................................................................................................795pgbouncer-admin.................................................................................................................. 807plcontainer.............................................................................................................................816plcontainer Configuration File...............................................................................................821psql........................................................................................................................................825reindexdb.............................................................................................................................. 849vacuumdb..............................................................................................................................851

Additional Supplied Programs......................................................................................................... 854

Chapter 8: Greenplum Database Reference Guide.............................. 855SQL Commands.............................................................................................................................. 856

SQL Syntax Summary..........................................................................................................859ABORT..................................................................................................................................897ALTER AGGREGATE...........................................................................................................898ALTER COLLATION.............................................................................................................899ALTER CONVERSION......................................................................................................... 900ALTER DATABASE.............................................................................................................. 901ALTER DEFAULT PRIVILEGES.......................................................................................... 902ALTER DOMAIN...................................................................................................................905ALTER EXTENSION.............................................................................................................907ALTER EXTERNAL TABLE..................................................................................................909ALTER FOREIGN DATA WRAPPER...................................................................................911ALTER FOREIGN TABLE.................................................................................................... 912ALTER FUNCTION...............................................................................................................916ALTER GROUP.................................................................................................................... 918ALTER INDEX...................................................................................................................... 919ALTER LANGUAGE............................................................................................................. 921ALTER MATERIALIZED VIEW.............................................................................................921ALTER OPERATOR............................................................................................................. 923ALTER OPERATOR CLASS................................................................................................ 924ALTER OPERATOR FAMILY...............................................................................................924ALTER PROTOCOL............................................................................................................. 927ALTER RESOURCE GROUP.............................................................................................. 928ALTER RESOURCE QUEUE...............................................................................................931ALTER ROLE....................................................................................................................... 933ALTER SCHEMA..................................................................................................................938ALTER SEQUENCE............................................................................................................. 938ALTER SERVER.................................................................................................................. 940ALTER TABLE......................................................................................................................942ALTER TABLESPACE..........................................................................................................955ALTER TEXT SEARCH CONFIGURATION.........................................................................956ALTER TEXT SEARCH DICTIONARY................................................................................ 957ALTER TEXT SEARCH PARSER........................................................................................958ALTER TEXT SEARCH TEMPLATE....................................................................................959ALTER TRIGGER.................................................................................................................959ALTER TYPE........................................................................................................................960ALTER USER....................................................................................................................... 963ALTER USER MAPPING..................................................................................................... 964


9

ALTER VIEW........................................................................................................................ 965ANALYZE..............................................................................................................................966BEGIN................................................................................................................................... 970CHECKPOINT.......................................................................................................................972CLOSE.................................................................................................................................. 973CLUSTER............................................................................................................................. 973COMMENT............................................................................................................................975COMMIT................................................................................................................................978COPY.................................................................................................................................... 979CREATE AGGREGATE........................................................................................................991CREATE CAST.....................................................................................................................997CREATE COLLATION........................................................................................................1001CREATE CONVERSION.................................................................................................... 1002CREATE DATABASE......................................................................................................... 1003CREATE DOMAIN..............................................................................................................1005CREATE EXTENSION........................................................................................................1007CREATE EXTERNAL TABLE.............................................................................................1008CREATE FOREIGN DATA WRAPPER..............................................................................1018CREATE FOREIGN TABLE............................................................................................... 1019CREATE FUNCTION..........................................................................................................1022CREATE GROUP............................................................................................................... 1031CREATE INDEX................................................................................................................. 1032CREATE LANGUAGE........................................................................................................ 1036CREATE MATERIALIZED VIEW........................................................................................1039CREATE OPERATOR........................................................................................................ 1041CREATE OPERATOR CLASS........................................................................................... 1045CREATE OPERATOR FAMILY..........................................................................................1048CREATE PROTOCOL........................................................................................................ 1049CREATE RESOURCE GROUP......................................................................................... 1050CREATE RESOURCE QUEUE..........................................................................................1053CREATE ROLE.................................................................................................................. 1057CREATE RULE...................................................................................................................1061CREATE SCHEMA.............................................................................................................1063CREATE SEQUENCE........................................................................................................ 1065CREATE SERVER............................................................................................................. 1067CREATE TABLE.................................................................................................................1069CREATE TABLE AS...........................................................................................................1083CREATE TABLESPACE.....................................................................................................1086CREATE TRIGGER............................................................................................................1088CREATE TEXT SEARCH CONFIGURATION....................................................................1090CREATE TEXT SEARCH DICTIONARY............................................................................1091CREATE TEXT SEARCH PARSER...................................................................................1092CREATE TEXT SEARCH TEMPLATE...............................................................................1093CREATE TYPE...................................................................................................................1094CREATE USER.................................................................................................................. 1101CREATE USER MAPPING................................................................................................ 1102CREATE VIEW................................................................................................................... 1103DEALLOCATE.................................................................................................................... 1106DECLARE........................................................................................................................... 1106DELETE.............................................................................................................................. 1109DISCARD............................................................................................................................ 1112DO.......................................................................................................................................1113DROP AGGREGATE..........................................................................................................1114DROP CAST.......................................................................................................................1115DROP COLLATION............................................................................................................ 1116DROP CONVERSION........................................................................................................ 1117


10

DROP DATABASE............................................................................................................. 1118DROP DOMAIN.................................................................................................................. 1118DROP EXTENSION............................................................................................................1119DROP EXTERNAL TABLE.................................................................................................1120DROP FOREIGN DATA WRAPPER..................................................................................1121DROP FOREIGN TABLE................................................................................................... 1121DROP FUNCTION..............................................................................................................1122DROP GROUP................................................................................................................... 1123DROP INDEX..................................................................................................................... 1124DROP LANGUAGE.............................................................................................................1125DROP MATERIALIZED VIEW............................................................................................1125DROP OPERATOR............................................................................................................ 1126DROP OPERATOR CLASS............................................................................................... 1127DROP OPERATOR FAMILY.............................................................................................. 1128DROP OWNED...................................................................................................................1129DROP PROTOCOL............................................................................................................ 1130DROP RESOURCE GROUP..............................................................................................1130DROP RESOURCE QUEUE.............................................................................................. 1131DROP ROLE.......................................................................................................................1132DROP RULE.......................................................................................................................1133DROP SCHEMA................................................................................................................. 1134DROP SEQUENCE............................................................................................................ 1135DROP SERVER..................................................................................................................1135DROP TABLE..................................................................................................................... 1136DROP TABLESPACE.........................................................................................................1137DROP TEXT SEARCH CONFIGURATION........................................................................1138DROP TEXT SEARCH DICTIONARY................................................................................1139DROP TEXT SEARCH PARSER....................................................................................... 1140DROP TEXT SEARCH TEMPLATE...................................................................................1140DROP TRIGGER................................................................................................................ 1141DROP TYPE....................................................................................................................... 1142DROP USER...................................................................................................................... 1143DROP USER MAPPING.....................................................................................................1143DROP VIEW....................................................................................................................... 1144END.....................................................................................................................................1144EXECUTE........................................................................................................................... 1145EXPLAIN............................................................................................................................. 1146FETCH................................................................................................................................ 1151GRANT................................................................................................................................1153INSERT............................................................................................................................... 1158LOAD.................................................................................................................................. 1161LOCK.................................................................................................................................. 1162MOVE..................................................................................................................................1165PREPARE........................................................................................................................... 1166REASSIGN OWNED...........................................................................................................1168REFRESH MATERIALIZED VIEW..................................................................................... 1169REINDEX............................................................................................................................ 1170RELEASE SAVEPOINT......................................................................................................1172RESET................................................................................................................................ 1173REVOKE............................................................................................................................. 1173ROLLBACK......................................................................................................................... 1176ROLLBACK TO SAVEPOINT.............................................................................................1177SAVEPOINT........................................................................................................................1178SELECT.............................................................................................................................. 1180SELECT INTO.................................................................................................................... 1198SET..................................................................................................................................... 1199


11

SET CONSTRAINTS.......................................................................................................... 1201SET ROLE.......................................................................................................................... 1202SET SESSION AUTHORIZATION..................................................................................... 1203SET TRANSACTION.......................................................................................................... 1204SHOW................................................................................................................................. 1207START TRANSACTION..................................................................................................... 1208TRUNCATE.........................................................................................................................1209UPDATE..............................................................................................................................1211VACUUM.............................................................................................................................1214VALUES.............................................................................................................................. 1217

Data Types.................................................................................................................................... 1220Date/Time Types................................................................................................................ 1222Pseudo-Types..................................................................................................................... 1232Text Search Data Types.................................................................................................... 1234Range Types...................................................................................................................... 1236

Summary of Built-in Functions...................................................................................................... 1241Greenplum Database Function Types................................................................................1241Built-in Functions and Operators........................................................................................1242JSON Functions and Operators......................................................................................... 1245Window Functions.............................................................................................................. 1252Advanced Aggregate Functions......................................................................................... 1254Text Search Functions and Operators............................................................................... 1256Range Functions and Operators........................................................................................ 1260

Additional Supplied Modules......................................................................................................... 1263auto_explain........................................................................................................................1263citext....................................................................................................................................1264dblink...................................................................................................................................1264diskquota.............................................................................................................................1267fuzzystrmatch...................................................................................................................... 1271gp_sparse_vector................................................................................................................1272hstore.................................................................................................................................. 1276orafce.................................................................................................................................. 1276pageinspect.........................................................................................................................1278pgcrypto.............................................................................................................................. 1279sslinfo.................................................................................................................................. 1279

Character Set Support...................................................................................................................1280Setting the Character Set...................................................................................................1282Character Set Conversion Between Server and Client...................................................... 1282

Server Configuration Parameters..................................................................................................1285Parameter Types and Values.............................................................................................1285Setting Parameters............................................................................................................. 1285Parameter Categories.........................................................................................................1286Configuration Parameters...................................................................................................1296

System Catalogs........................................................................................................................... 1378System Tables.................................................................................................................... 1378System Views..................................................................................................................... 1379System Catalogs Definitions...............................................................................................1380

The gp_toolkit Administrative Schema..........................................................................................1480Checking for Tables that Need Routine Maintenance........................................................1480Checking for Locks.............................................................................................................1481Checking Append-Optimized Tables.................................................................................. 1483Viewing Greenplum Database Server Log Files................................................................ 1487Checking Server Configuration Files..................................................................................1490Checking for Failed Segments........................................................................................... 1491Checking Resource Group Activity and Status.................................................................. 1492Checking Resource Queue Activity and Status................................................................. 1496


12

Checking Query Disk Spill Space Usage...........................................................................1498Viewing Users and Groups (Roles)....................................................................................1500Checking Database Object Sizes and Disk Space............................................................ 1500Checking for Uneven Data Distribution.............................................................................. 1504Including Data for Materialized Views................................................................................ 1505

The gpperfmon Database..............................................................................................................1507database_*.........................................................................................................................1509diskspace_*....................................................................................................................... 1510interface_stats_*................................................................................................................ 1510log_alert_*..........................................................................................................................1512queries_*............................................................................................................................. 1513segment_*..........................................................................................................................1515socket_stats_*.....................................................................................................................1516system_*............................................................................................................................. 1517dynamic_memory_info........................................................................................................ 1519memory_info...................................................................................................................... 1519

Server Programmatic Interfaces....................................................................................................1521Greenplum Partner Connector API.................................................................................... 1521Developing a Background Worker Process....................................................................... 1540

SQL Features, Reserved and Key Words, and Compliance........................................................ 1543Summary of Greenplum Features...................................................................................... 1543Reserved Identifiers and SQL Key Words......................................................................... 1553SQL 2008 Optional Feature Compliance........................................................................... 1569

Chapter 9: Greenplum Client and Loader Tools Package................. 1599

Chapter 10: About the Tools Package................................................ 1600

Chapter 11: Installing the Client and Loader Tools Package............ 1601Supported Platforms......................................................................................................................1602Installation Procedure....................................................................................................................1603About Your Installation.................................................................................................................. 1604Running the UNIX Tools Installer................................................................................................. 1605

Prerequisites....................................................................................................................... 1605Procedure............................................................................................................................1605

Running the Windows Tools Installer........................................................................................... 1606Prerequisites....................................................................................................................... 1606Procedure............................................................................................................................1606

Chapter 12: Configuring Greenplum Database for Remote ClientAccess................................................................................................. 1607

Chapter 13: Configuring a Client System for KerberosAuthentication.................................................................................... 1608

Chapter 14: Using the Client and Loader Tools................................. 1609Prerequisites.................................................................................................................................. 1610Setting Up Your Greenplum Database Clients Runtime Environment.......................................... 1611Running the Client and Loader Programs.................................................................................... 1612


13

Greenplum Database Documentation References........................................................................1613Windows Considerations............................................................................................................... 1614

Chapter 15: Client and Loader Utility Reference................................1615

Chapter 16: DataDirect ODBC Drivers for Pivotal Greenplum...........1616Prerequisites.................................................................................................................................. 1617Supported Client Platforms........................................................................................................... 1618Installing on Linux Systems.......................................................................................................... 1619

Configuring the Driver on Linux......................................................................................... 1620Testing the Driver Connection on Linux.............................................................................1621

Installing on Windows Systems.................................................................................................... 1623Verifying the Version on Windows..................................................................................... 1623Configuring and Testing the Driver on Windows................................................................1623

DataDirect Driver Documentation..................................................................................................1625

Chapter 17: DataDirect JDBC Driver for Pivotal Greenplum............. 1626Prerequisites.................................................................................................................................. 1627Downloading the DataDirect JDBC Driver.................................................................................... 1628Obtaining Version Details for the Driver....................................................................................... 1629Usage Information......................................................................................................................... 1630Configuring Prepared Statement Execution..................................................................................1631DataDirect Driver Documentation..................................................................................................1632

Pivotal Greenplum 6.7 Release Notes Release Notes

14

Chapter 2

Pivotal Greenplum 6.7 Release Notes

This document contains pertinent release information about Pivotal Greenplum Database 6.7 releases.For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 6 is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


15

Release 6.7.1Release Date: 2020-04-30

Pivotal Greenplum 6.7.1 is a maintenance release that resolves several issues. Version 6.7.1 also updatesPostGIS to version 2.5.4, which removes several previous limitations. See Geospatial Analytics for moreinformation.


16

Resolved IssuesPivotal Greenplum 6.7.1 resolves these issues:

n/a - MADlib

In Greenplum 6.7.0 the MADlib download files that were originally provided,madlib-1.17.0+2-gp6-rhel7-x86_64.tar.gz and madlib-1.17.0+2-gp6-rhel6-x86_64.tar.gz, contained MADlib version 1.16 instead of version 1.17. Thisis resolved in Greenplum 6.7.1, and in Greenplum 6.7.0 with the newly-provided filesmadlib-1.17.0+3-gp6-rhel7-x86_64.tar.gz and madlib-1.17.0+3-gp6-rhel6-x86_64.tar.gz.

9790 - Server

A crash could occur when performing a SELECT query against a column-orientedtable, when the table was created using the WITH NO DATA clause. The problemoccurred because the WITH clause options were not correctly added to thepg_attribute_encoding table. This problem has been resolved.

30499 - Server: Execution

Fixed a memory leak that occurred when executing CHECKPOINT commands.

30559 - Query Optimizer

Queries that contain an IN clause with a large number of constants took a long time togenerate a query plan. Most of the time was spent estimating the cardinality of the INclause predicate. The cardinality estimation algorithm has been enhanced and significantlyreduces the cardinality estimation time for the specified type of query.

30579 - Interconnect

In some cases during query execution, the query hung with the query dispatcher (QD)waiting for the query executor (QE) on a few segment instances to complete. This issue isresolved.

172163076 - Server

A subtransaction would incorrectly use 1-phase commit instead of 2-phase commit if\set ON_ERROR_ROLLBACK interactive was enabled in a client's .psqlrc file. Thisproblem has been resolved.

172324858, 9891 - MPP: Locking, Signals, Processes

In some cases, Greenplum Database did not manage snapshots correctly whenprocessing concurrent distributed transactions. This caused a concurrent transaction toaccess a distributed log file that was no longer available and generated the error messageCould not open file ""pg_distributedlog/"": No such fileor directory. This issue is resolved.

172348849 - Postgres Planner

Some queries that contain a UNION ALL that combines the results from SELECTcommand that uses a replicated table with another SELECT command returns the errorERROR: could not build Motion path. This issue is resolved.

172284550 9823 - ALTER DATABASE

The ALTER DATABASE...FROM CURRENT command did not set a server configurationparameter for a database. This issue is resolved.

https://github.com/greenplum-db/gpdb/issues/9891https://github.com/greenplum-db/gpdb/issues/9823


17

Release 6.7.0Release Date: 2020-04-17

Pivotal Greenplum 6.7.0 is a minor release that includes changed features and resolves several issues.

FeaturesGreenplum Database 6.7.0 includes these new and changed features:

• Greenplum Database 6.7 introduces the new gp_resource_group_queuing_timeout serverconfiguration parameter. When the resource group-based resource management scheme is active,gp_resource_group_queuing_timeout specifies the maximum amount of time a transaction waitsfor execution in a queue on a resource group before Greenplum Database cancels the transaction. Bydefault, queued transactions in a resource group can wait indefinitely.

• Greenplum Database 6.7 includes MADlib version 1.17, which introduces new Deep Learning features,k-Means clustering, and other improvements and bug fixes. See the Apache MADlib page for additionalinformation and Release Notes.

Note: In Greenplum 6.7.0 the MADlib download files that were originally provided,madlib-1.17.0+2-gp6-rhel7-x86_64.tar.gz and madlib-1.17.0+2-gp6-rhel6-x86_64.tar.gz, contained MADlib version 1.16 instead of version 1.17. This is resolved inGreenplum 6.7.1, and in Greenplum 6.7.0 with the newly-provided files madlib-1.17.0+3-gp6-rhel7-x86_64.tar.gz and madlib-1.17.0+3-gp6-rhel6-x86_64.tar.gz.

Resolved IssuesPivotal Greenplum 6.7.0 resolves these issues:

8539 - Server

Using NOWAIT in a SELECT FOR UPDATE statement could result in the error, ERROR:relation "" does not exist, because locking was not correctly handledfor the NOWAIT clause. This problem has been resolved. Note, however, that NOWAITonly affects how the SELECT statement obtains row-level locks. A SELECT FOR UPDATENOWAIT statement will always wait for the required table-level lock; it behaves as ifNOWAIT was omitted.

9089 - Server

Fixed a problem where Greenplum Database failed to truncate an append-only, column-oriented table if the CREATE TABLE and TRUNCATE statements were executed in thesame transaction.

30305 - Resource Groups

A transaction may be queued for execution on a resource group for an extendedperiod of time, particularly when the resource group reached its concurrent transactionlimit. This could prevent queries initiated by Greenplum Database superusersfrom executing. Greenplum Database 6.7 resolves this issue by introducing thegp_resource_group_queuing_timeout server configuration parameter, which specifies themaximum amount of time a queued transaction waits for execution in a resource groupbefore Greenplum cancels the transaction.

30531 - Query Optimizer

An out of memory error occurred when running some queries that contain joins thatperform a comparison operation on citext data. The error occurred because the queryfalls back to the Postgres Planner. This issue is resolved. Now the query does not fall backto the Postgres planner, the query is executed using GPORCA.

http://madlib.apache.org/


18

30536 - PL/pgSQL

In a PL/pgSQL procedure, output text from a RAISE NOTICE statement was not displayedcorrectly if the text contained a newline (line feed) character. Only the text before thenewline character was displayed. This issue is resolved.

Upgrading from Greenplum 6.x to Greenplum 6.7Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software toGreenplum 6.7.0.


19

Deprecated FeaturesDeprecated features will be removed in a future major release of Greenplum Database. Pivotal Greenplum6.x deprecates:

• The analzyedb option --skip_root_stats (deprecated since 6.2).

If the option is specified, a warning is issued stating that the option will be ignored.• The server configuration parameter gp_statistics_use_fkeys (deprecated since 6.2).• The following PXF configuration properties (deprecated since 6.2):

• The PXF_USER_IMPERSONATION, PXF_PRINCIPAL, and PXF_KEYTAB settings in the pxf-env.sh file. You can use the pxf-site.xml file to configure Kerberos and impersonation settingsfor your new Hadoop server configurations.

• The pxf.impersonation.jdbc property setting in the jdbc-site.xml file. You can use thepxf.service.user.impersonation property to configure user impersonation for a new JDBCserver configuration.

• The server configuration parameter gp_ignore_error_table (deprecated since 6.0).

To avoid a Greenplum Database syntax error, set the value of this parameter to true when yourun applications that execute CREATE EXTERNAL TABLE or COPY commands that include the nowremoved Greenplum Database 4.3.x INTO ERROR TABLE clause.

• Specifying => as an operator name in the CREATE OPERATOR command (deprecated since 6.0).• The Greenplum external table C API (deprecated since 6.0).

Any developers using this API are encouraged to use the new Foreign Data Wrapper API in its place.• Commas placed between a SUBPARTITION TEMPLATE clause and its corresponding SUBPARTITION

BY clause, and between consecutive SUBPARTITION BY clauses in a CREATE TABLE command(deprecated since 6.0).

Using this undocumented syntax will generate a deprecation warning message.• The timestamp format YYYYMMDDHH24MISS (deprecated since 6.0).

This format could not be parsed unambiguously in previous Greenplum Database releases, and is notsupported in PostgreSQL 9.4.

• The createlang and droplang utilities (deprecated since 6.0).• The pg_resqueue_status system view (deprecated since 6.0).

Use the gp_toolkit.gp_resqueue_status view instead.• The GLOBAL and LOCAL modifiers when creating a temporary table with the CREATE TABLE and

CREATE TABLE AS commands (deprecated since 6.0).

These keywords are present for SQL standard compatibility, but have no effect in Greenplum Database.• The Greenplum Platform Extension Framework (PXF) HDFS profile names for the Text, Avro, JSON,

Parquet, and SequenceFile data formats (deprecated since 5.16).

Refer to Connectors, Data Formats, and Profiles in the PXF Hadoop documentation for moreinformation.

• Using WITH OIDS or oids=TRUE to assign an OID system column when creating or altering a table(deprecated since 6.0).

• Allowing superusers to specify the SQL_ASCII encoding regardless of the locale settings (deprecatedsince 6.0).

This choice may result in misbehavior of character-string functions when data that is not encoding-compatible with the locale is stored in the database.

• The @@@ text search operator (deprecated since 6.0).

This operator is currently a synonym for the @@ operator.

../pxf/access_hdfs.html#hadoop_connectors


20

• The unparenthesized syntax for option lists in the VACUUM command (deprecated since 6.0).

This syntax requires that the options to the command be specified in a specific order.• The plain pgbouncer authentication type (auth_type = plain) (deprecated since 4.x).


21

Migrating Data to Greenplum 6Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Migrating Data from Greenplum 4.3 or 5 for guidelines and considerations for migrating existingGreenplum data to Greenplum 6, using standard backup and restore procedures.


22

Known Issues and LimitationsPivotal Greenplum 6 has these limitations:

• Upgrading a Greenplum Database 4 or 5 release, or Greenplum 6 Beta release, to Pivotal Greenplum 6is not supported.

• MADlib, GPText, and PostGIS are not yet provided for installation on Ubuntu systems.• Greenplum 6 is not supported for installation on DCA systems.• Greenplum for Kubernetes is not yet provided with this release.

The following table lists key known issues in Pivotal Greenplum 6.x.

Table 1: Key Known Issues in Pivotal Greenplum 6.x

Issue Category Description

30522 Logging Greenplum Database may write a FATAL message to the standbymaster or mirror log stating that the database system is in recoverymode when the instance is synchronizing with the master andGreenplum attempts to contact it before the operation completes.Ignore these messages and use gpstate -f output to determine ifthe standby successfully synchronized with the Greenplum master;the command returns Sync state: sync if it is synchronized.

30537 PostgresPlanner

The Postgres Planner generates a very large query plan that causesout of memory issues for the following type of CTE (common tableexpression) query: the WITH clause of the CTE contains a partitionedtable with a large number partitions, and the WITH reference is usedin a subquery that joins another partitioned table.

Workaround: If possible, use the GPORCA query optimizer. Withthe server configuration parameter optimizer=on, GreenplumDatabase attempts to use GPORCA for query planning andoptimization when possible and falls back to the Postgres Plannerwhen GPORCA cannot be used. Also, the specified type of querymight require a long time to complete.

171883625 PXF pxf [cluster] init may fail to recognize a new JAVA_HOMEsetting when the value is provided via the shell environment.

Workaround: Edit $PXF_CONF/conf/pxf-env.sh and manuallyset JAVA_HOME to the new value, run pxf cluster sync tosynchronize this configuration change across the Greenplum cluster,and then re-run pxf [cluster] init.

170824967 gpfidsts For Greenplum Database 6.x, a command that accesses an externaltable that uses the gpfdists protocol fails if the external tabledoes not use an IP address when specifying a host system in theLOCATION clause of the external table definition.

n/a MaterializedViews

By default, certain gp_toolkit views do not display data formaterialized views. If you want to include this information in gp_toolkit view output, you must redefine a gp_toolkit internalview as described in Including Data for Materialized Views.


23

Issue Category Description

168689202 PXF PXF fails to run any query on Java 11 that specifies a Hive* profiledue to this Hive known issue: ClassCastException when initializingHiveMetaStoreClient on JDK10 or newer.

Workaround: Run PXF on Java 8 or use the PXF JDBC Connectorto access Hive.

168957894 PXF The PXF Hive Connector does not support using the Hive* profilesto access Hive transactional tables.

Workaround: Use the PXF JDBC Connector to access Hive.

169200795 GreenplumStream Server

When loading Kafka data into Greenplum Database in UPDATE andMERGE modes, GPSS requires that a MAPPING exist for each columnname identified in the MATCH_COLUMNS and UPDATE_COLUMNS lists.

170202002 Greenplum-KafkaIntegration

Updating the METADATA:SCHEMA property and restarting apreviously-run load job could cause gpkafka to re-read Kafkamessages published to the topic, and load duplicate messages intoGreenplum Database.

168548176 gpbackup When using gpbackup to back up a Greenplum Database 5.7.1 orearlier 5.x release with resource groups enabled, gpbackup returnsa column not found error for t6.value AS memoryauditor.

164791118 PL/R PL/R cannot be installed using the deprecated createlang utility,and displays the error:

createlang: language installation failed: ERROR:no schema has been selected to create in

Workaround: Use CREATE EXTENSION to install PL/R, as describedin the documentation.

N/A GreenplumClient/LoadTools onWindows

The Greenplum Database client and load tools on Windows have notbeen tested with Active Directory Kerberos authentication.

https://issues.apache.org/jira/browse/HIVE-21508https://issues.apache.org/jira/browse/HIVE-21508


24

Differences Compared to Open Source GreenplumDatabase

Pivotal Greenplum 6.x includes all of the functionality in the open source Greenplum Database project andadds:

• Product packaging and installation script• Support for QuickLZ compression. QuickLZ compression is not provided in the open source version of

Greenplum Database due to licensing restrictions.• Support for data connectors:

• Greenplum-Spark Connector• Greenplum-Informatica Connector• Greenplum-Kafka Integration• Greenplum Stream Server

• Data Direct ODBC/JDBC Drivers• gpcopy utility for copying or migrating objects between Greenplum systems• Support for managing Greenplum Database using Pivotal Greenplum Command Center• Support for full text search and text analysis using Pivotal GPText• Greenplum backup plugin for DD Boost• Backup/restore storage plugin API (Beta)

http://greenplum.org/

25

Chapter 3

Installing and Upgrading Greenplum

Information about installing, configuring, and upgrading Greenplum Database software and configuringGreenplum Database host machines.

Installing and Upgrading Greenplum Release Notes

26

Platform RequirementsThis topic describes the Pivotal Greenplum

pivotal greenplum databasecontents release notes 6 managing resources.....514

Documents