PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 5.6.1

Pivotal Greenplum DatabaseDocumentationRev: A02

© 2018 Pivotal Software, Inc.

Copyright OpenTopic

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2018 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised March 2018 (5.6.1)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents OpenTopic

3

Contents

Chapter 2: Pivotal Greenplum 5.6.1 Release Notes............................... 14Welcome to Pivotal Greenplum 5.6.1............................................................................................... 15New Features.................................................................................................................................... 16

Query Optimization Enhancements........................................................................................16PL/Container Enhancements..................................................................................................16gpbackup and gprestore Enhancements................................................................................16Parameter to Ignore 4.3.x INTO ERROR TABLE Clause...................................................... 17Experimental Pivotal Greenplum-Informatica Connector........................................................17Greenplum Metrics Collection Extension............................................................................... 17

Changed Features.............................................................................................................................18Experimental Features...................................................................................................................... 19Differences Compared to Open Source Greenplum Database.........................................................20Supported Platforms..........................................................................................................................21

Veritas NetBackup.................................................................................................................. 22Supported Platform Notes...................................................................................................... 22

Pivotal Greenplum Tools and Extensions Compatibility................................................................... 24Client Tools.............................................................................................................................24Extensions...............................................................................................................................25Pivotal Greenplum Data Connectors......................................................................................25Pivotal GPText Compatibility.................................................................................................. 26Pivotal Greenplum Command Center.................................................................................... 26

Hadoop Distribution Compatibility..................................................................................................... 27Upgrading to Greenplum Database 5.6.1......................................................................................... 28

Upgrading from 5.x to 5.6.1................................................................................................... 29Troubleshooting a Failed Upgrade.........................................................................................31

Migrating Data to Pivotal Greenplum 5.x..........................................................................................32Pivotal Greenplum on DCA Systems................................................................................................33Resolved Issues................................................................................................................................ 34Known Issues and Limitations.......................................................................................................... 37Update for gp_toolkit.gp_bloat_diag Issue........................................................................................44

Chapter 4: Greenplum Database Installation Guide...............................46Introduction to Greenplum.................................................................................................................47

The Greenplum Master...........................................................................................................47The Segments........................................................................................................................ 48The Interconnect.....................................................................................................................49ETL Hosts for Data Loading.................................................................................................. 50Greenplum Performance Monitoring...................................................................................... 51

Estimating Storage Capacity............................................................................................................. 52Calculating Usable Disk Capacity.......................................................................................... 52Calculating User Data Size.................................................................................................... 52Calculating Space Requirements for Metadata and Logs......................................................53

Configuring Your Systems and Installing Greenplum....................................................................... 54System Requirements.............................................................................................................54Setting the Greenplum Recommended OS Parameters........................................................ 56Creating the Greenplum Database Administrative User Account.......................................... 60Installing the Greenplum Database Software.........................................................................61Installing and Configuring Greenplum on all Hosts................................................................62

Contents OpenTopic

4

Installing Oracle Compatibility Functions............................................................................... 64Installing Optional Modules.................................................................................................... 64Installing Greenplum Database Extensions........................................................................... 65Installing and Configuring the Greenplum Platform Extension Framework (PXF)..................65Creating the Data Storage Areas...........................................................................................65Synchronizing System Clocks................................................................................................ 66Enabling iptables.................................................................................................................... 67Amazon EC2 Configuration (Amazon Web Services)............................................................70Next Steps.............................................................................................................................. 75

Installing the Data Science Packages.............................................................................................. 76Python Data Science Module Package..................................................................................76R Data Science Library Package........................................................................................... 78

Validating Your Systems................................................................................................................... 81Validating OS Settings............................................................................................................81Validating Hardware Performance..........................................................................................81Validating Disk I/O and Memory Bandwidth...........................................................................82

Configuring Localization Settings...................................................................................................... 84About Locale Support in Greenplum Database..................................................................... 84Character Set Support............................................................................................................85Setting the Character Set.......................................................................................................88Character Set Conversion Between Server and Client..........................................................88

Initializing a Greenplum Database System.......................................................................................91Overview................................................................................................................................. 91Initializing Greenplum Database.............................................................................................91Setting Greenplum Environment Variables............................................................................ 94Next Steps.............................................................................................................................. 95

About Implicit Text Casting in Greenplum Database........................................................................96Workaround: Manually Creating Missing Operators...............................................................97

Installation Management Utilities.....................................................................................................100Greenplum Environment Variables................................................................................................. 101

Required Environment Variables..........................................................................................101Optional Environment Variables........................................................................................... 101

Chapter 6: Greenplum Database Administrator Guide........................ 103Greenplum Database Concepts...................................................................................................... 104

About the Greenplum Architecture....................................................................................... 104About Management and Monitoring Utilities........................................................................ 106About Concurrency Control in Greenplum Database...........................................................107About Parallel Data Loading................................................................................................ 115About Redundancy and Failover in Greenplum Database...................................................116About Database Statistics in Greenplum Database............................................................. 118

Managing a Greenplum System..................................................................................................... 125About the Greenplum Database Release Version Number................................................. 125Starting and Stopping Greenplum Database....................................................................... 125Accessing the Database.......................................................................................................127Configuring the Greenplum Database System.....................................................................137Enabling High Availability and Data Consistency Features................................................. 147Backing Up and Restoring Databases................................................................................. 166Expanding a Greenplum System..........................................................................................203Migrating Data with gptransfer............................................................................................. 218Monitoring a Greenplum System..........................................................................................224Routine System Maintenance Tasks....................................................................................244Recommended Monitoring and Maintenance Tasks............................................................ 248

Managing Greenplum Database Access.........................................................................................256Configuring Client Authentication......................................................................................... 256

Contents OpenTopic

5

Managing Roles and Privileges............................................................................................282Defining Database Objects..............................................................................................................289

Creating and Managing Databases......................................................................................289Creating and Managing Tablespaces...................................................................................291Creating and Managing Schemas........................................................................................294Creating and Managing Tables............................................................................................ 295Choosing the Table Storage Model..................................................................................... 298Partitioning Large Tables......................................................................................................309Creating and Using Sequences........................................................................................... 321Using Indexes in Greenplum Database............................................................................... 324Creating and Managing Views............................................................................................. 327

Distribution and Skew..................................................................................................................... 328Local (Co-located) Joins.......................................................................................................328Data Skew............................................................................................................................ 328Processing Skew.................................................................................................................. 329

Inserting, Updating, and Deleting Data...........................................................................................332About Concurrency Control in Greenplum Database...........................................................332Inserting Rows...................................................................................................................... 333Updating Existing Rows........................................................................................................334Deleting Rows.......................................................................................................................334Working With Transactions...................................................................................................334Vacuuming the Database..................................................................................................... 336

Querying Data................................................................................................................................. 337About Greenplum Query Processing....................................................................................337About GPORCA....................................................................................................................340Defining Queries................................................................................................................... 353WITH Queries (Common Table Expressions)......................................................................362Using Functions and Operators............................................................................................365Working with JSON Data..................................................................................................... 374Working with XML Data........................................................................................................378Query Performance.............................................................................................................. 390Managing Spill Files Generated by Queries........................................................................ 390Query Profiling...................................................................................................................... 390

Working with External Data.............................................................................................................396Defining External Tables...................................................................................................... 396Accessing External Data with PXF...................................................................................... 414Accessing HDFS Data with gphdfs...................................................................................... 414Using the Greenplum Parallel File Server (gpfdist)..............................................................437

Loading and Unloading Data.......................................................................................................... 441Loading Data Using an External Table................................................................................ 442Loading and Writing Non-HDFS Custom Data.................................................................... 442Handling Load Errors............................................................................................................445Loading Data with gpload.....................................................................................................447Transforming External Data with gpfdist and gpload........................................................... 448Loading Data with COPY..................................................................................................... 458Running COPY in Single Row Error Isolation Mode............................................................458Optimizing Data Load and Query Performance................................................................... 458Unloading Data from Greenplum Database......................................................................... 459Formatting Data Files........................................................................................................... 461Example Custom Data Access Protocol.............................................................................. 464

Managing Performance................................................................................................................... 471Defining Database Performance.......................................................................................... 471Common Causes of Performance Issues............................................................................ 472Greenplum Database Memory Overview............................................................................. 475Managing Resources............................................................................................................478Investigating a Performance Problem.................................................................................. 501

Contents OpenTopic

6

Chapter 8: Greenplum Database Security Configuration Guide......... 505Securing the Database....................................................................................................................506Greenplum Database Ports and Protocols..................................................................................... 507Configuring Client Authentication.................................................................................................... 511

Allowing Connections to Greenplum Database....................................................................511Editing the pg_hba.conf File.................................................................................................512Authentication Methods........................................................................................................ 513SSL Client Authentication.....................................................................................................516PAM Based Authentication...................................................................................................518Radius Authentication...........................................................................................................518Limiting Concurrent Connections......................................................................................... 518Encrypting Client/Server Connections..................................................................................519

Configuring Database Authorization................................................................................................521Access Permissions and Roles............................................................................................521Managing Object Privileges..................................................................................................521Using SSH-256 Encryption...................................................................................................522Restricting Access by Time..................................................................................................524Dropping a Time-based Restriction.................................................................................... 526

Greenplum Command Center Security........................................................................................... 527Auditing............................................................................................................................................ 530Encrypting Data and Database Connections.................................................................................. 535

Encrypting gpfdist Connections............................................................................................ 535Encrypting Data at Rest with pgcrypto.................................................................................536

Enabling gphdfs Authentication with a Kerberos-secured Hadoop Cluster.....................................544Prerequisites......................................................................................................................... 544Configuring the Greenplum Cluster......................................................................................544Creating and Installing Keytab Files.................................................................................... 545Configuring gphdfs for Kerberos.......................................................................................... 546Testing Greenplum Database Access to HDFS...................................................................547Troubleshooting HDFS with Kerberos..................................................................................548

Security Best Practices................................................................................................................... 550

Chapter 10: Greenplum Database Best Practices................................ 554Best Practices Summary.................................................................................................................555System Configuration...................................................................................................................... 561Schema Design............................................................................................................................... 565

Data Types........................................................................................................................... 565Storage Model...................................................................................................................... 565Compression......................................................................................................................... 566Distributions.......................................................................................................................... 567Partitioning............................................................................................................................ 570Indexes..................................................................................................................................572Column Sequence and Byte Alignment............................................................................... 572

Memory and Resource Management with Resource Queues........................................................ 574System Monitoring and Maintenance..............................................................................................578

Monitoring............................................................................................................................. 578Updating Statistics with ANALYZE.......................................................................................579Managing Bloat in the Database..........................................................................................580Monitoring Greenplum Database Log Files..........................................................................584

Loading Data................................................................................................................................... 586INSERT Statement with Column Values..............................................................................586COPY Statement.................................................................................................................. 586External Tables.....................................................................................................................586

Contents OpenTopic

7

External Tables with Gpfdist................................................................................................ 586Gpload...................................................................................................................................587Best Practices.......................................................................................................................588

Migrating Data with Gptransfer....................................................................................................... 589Security............................................................................................................................................ 594Encrypting Data and Database Connections.................................................................................. 597Accessing a Kerberized Hadoop Cluster........................................................................................ 606

Prerequisites......................................................................................................................... 606Configuring the Greenplum Cluster......................................................................................606Creating and Installing Keytab Files.................................................................................... 607Configuring gphdfs for Kerberos.......................................................................................... 608Testing Greenplum Database Access to HDFS...................................................................609Troubleshooting HDFS with Kerberos..................................................................................610

Tuning SQL Queries....................................................................................................................... 612How to Generate Explain Plans........................................................................................... 612How to Read Explain Plans................................................................................................. 612Optimizing Greenplum Queries............................................................................................ 614

High Availability............................................................................................................................... 616Disk Storage......................................................................................................................... 616Master Mirroring....................................................................................................................616Segment Mirroring................................................................................................................ 617Dual Clusters........................................................................................................................ 618Backup and Restore.............................................................................................................618Detecting Failed Master and Segment Instances................................................................ 619Segment Mirroring Configuration..........................................................................................620

Chapter 12: Greenplum Database Utility Guide................................... 626Management Utility Reference........................................................................................................ 627

Backend Server Programs................................................................................................... 628analyzedb..............................................................................................................................629gpactivatestandby................................................................................................................. 633gpaddmirrors......................................................................................................................... 635gpbackup...............................................................................................................................638gpcheck.................................................................................................................................642gpcheckcat............................................................................................................................ 644gpcheckperf...........................................................................................................................647gpconfig.................................................................................................................................650gpcrondump.......................................................................................................................... 653gpdbrestore........................................................................................................................... 667gpdeletesystem..................................................................................................................... 674gpexpand.............................................................................................................................. 675gpfdist....................................................................................................................................678gpfilespace............................................................................................................................ 681gpinitstandby......................................................................................................................... 684gpinitsystem.......................................................................................................................... 686gpload................................................................................................................................... 693gplogfilter...............................................................................................................................703gpmapreduce........................................................................................................................ 706gpmfr..................................................................................................................................... 707gpmovemirrors...................................................................................................................... 711gpperfmon_install..................................................................................................................712gppkg.................................................................................................................................... 716gprecoverseg........................................................................................................................ 718gpreload................................................................................................................................ 723gprestore............................................................................................................................... 725

Contents OpenTopic

8

gpscp.....................................................................................................................................728gpseginstall........................................................................................................................... 730gpssh.....................................................................................................................................732gpssh-exkeys........................................................................................................................ 735gpstart................................................................................................................................... 737gpstate.................................................................................................................................. 739gpstop................................................................................................................................... 743gpsys1...................................................................................................................................745gptransfer.............................................................................................................................. 746pgbouncer............................................................................................................................. 758

Client Utility Reference....................................................................................................................778Client Utility Summary.......................................................................................................... 778

Additional Supplied Modules........................................................................................................... 828citext Data Type................................................................................................................... 828dblink Functions....................................................................................................................830hstore Functions................................................................................................................... 831Oracle Compatibility Functions.............................................................................................834passwordcheck..................................................................................................................... 855

Chapter 14: Greenplum Database Reference Guide............................ 857SQL Command Reference..............................................................................................................858

SQL Syntax Summary..........................................................................................................860ABORT..................................................................................................................................887ALTER AGGREGATE...........................................................................................................888ALTER CONVERSION......................................................................................................... 889ALTER DATABASE.............................................................................................................. 890ALTER DOMAIN...................................................................................................................891ALTER EXTENSION.............................................................................................................893ALTER EXTERNAL TABLE..................................................................................................896ALTER FILESPACE............................................................................................................. 897ALTER FUNCTION...............................................................................................................898ALTER GROUP.................................................................................................................... 900ALTER INDEX...................................................................................................................... 901ALTER LANGUAGE............................................................................................................. 903ALTER OPERATOR............................................................................................................. 903ALTER OPERATOR CLASS................................................................................................ 904ALTER OPERATOR FAMILY...............................................................................................905ALTER PROTOCOL............................................................................................................. 907ALTER RESOURCE GROUP.............................................................................................. 908ALTER RESOURCE QUEUE...............................................................................................910ALTER ROLE....................................................................................................................... 912ALTER SCHEMA..................................................................................................................916ALTER SEQUENCE............................................................................................................. 917ALTER TABLE......................................................................................................................919ALTER TABLESPACE..........................................................................................................929ALTER TYPE........................................................................................................................930ALTER USER....................................................................................................................... 930ALTER VIEW........................................................................................................................ 931ANALYZE..............................................................................................................................932BEGIN................................................................................................................................... 935CHECKPOINT.......................................................................................................................937CLOSE.................................................................................................................................. 937CLUSTER............................................................................................................................. 938COMMENT............................................................................................................................939COMMIT................................................................................................................................941

Contents OpenTopic

9

COPY.................................................................................................................................... 942CREATE AGGREGATE........................................................................................................952CREATE CAST.....................................................................................................................956CREATE CONVERSION...................................................................................................... 959CREATE DATABASE........................................................................................................... 960CREATE DOMAIN................................................................................................................961CREATE EXTENSION..........................................................................................................963CREATE EXTERNAL TABLE...............................................................................................964CREATE FUNCTION............................................................................................................973CREATE GROUP................................................................................................................. 979CREATE INDEX................................................................................................................... 980CREATE LANGUAGE.......................................................................................................... 983CREATE OPERATOR.......................................................................................................... 985CREATE OPERATOR CLASS............................................................................................. 989CREATE OPERATOR FAMILY............................................................................................992CREATE PROTOCOL.......................................................................................................... 993CREATE RESOURCE GROUP........................................................................................... 994CREATE RESOURCE QUEUE............................................................................................996CREATE ROLE.................................................................................................................... 999CREATE RULE...................................................................................................................1004CREATE SCHEMA.............................................................................................................1005CREATE SEQUENCE........................................................................................................ 1007CREATE TABLE.................................................................................................................1010CREATE TABLE AS...........................................................................................................1021CREATE TABLESPACE.....................................................................................................1024CREATE TYPE...................................................................................................................1025CREATE USER.................................................................................................................. 1031CREATE VIEW................................................................................................................... 1031DEALLOCATE.................................................................................................................... 1033DECLARE........................................................................................................................... 1034DELETE.............................................................................................................................. 1036DISCARD............................................................................................................................ 1038DO.......................................................................................................................................1039DROP AGGREGATE..........................................................................................................1041DROP CAST.......................................................................................................................1042DROP CONVERSION........................................................................................................ 1042DROP DATABASE............................................................................................................. 1043DROP DOMAIN.................................................................................................................. 1044DROP EXTENSION............................................................................................................1044DROP EXTERNAL TABLE.................................................................................................1045DROP FILESPACE.............................................................................................................1046DROP FUNCTION..............................................................................................................1047DROP GROUP................................................................................................................... 1048DROP INDEX..................................................................................................................... 1048DROP LANGUAGE.............................................................................................................1049DROP OPERATOR............................................................................................................ 1050DROP OPERATOR CLASS............................................................................................... 1051DROP OPERATOR FAMILY.............................................................................................. 1051DROP OWNED...................................................................................................................1052DROP PROTOCOL............................................................................................................ 1053DROP RESOURCE GROUP..............................................................................................1054DROP RESOURCE QUEUE.............................................................................................. 1055DROP ROLE.......................................................................................................................1056DROP RULE.......................................................................................................................1057DROP SCHEMA................................................................................................................. 1058DROP SEQUENCE............................................................................................................ 1058

Contents OpenTopic

10

DROP TABLE..................................................................................................................... 1059DROP TABLESPACE.........................................................................................................1060DROP TYPE....................................................................................................................... 1060DROP USER...................................................................................................................... 1061DROP VIEW....................................................................................................................... 1062END.....................................................................................................................................1062EXECUTE........................................................................................................................... 1063EXPLAIN............................................................................................................................. 1064FETCH................................................................................................................................ 1066GRANT................................................................................................................................1069INSERT............................................................................................................................... 1073LOAD.................................................................................................................................. 1075LOCK.................................................................................................................................. 1075MOVE..................................................................................................................................1078PREPARE........................................................................................................................... 1079REASSIGN OWNED...........................................................................................................1081REINDEX............................................................................................................................ 1082RELEASE SAVEPOINT......................................................................................................1083RESET................................................................................................................................ 1084REVOKE............................................................................................................................. 1085ROLLBACK......................................................................................................................... 1087ROLLBACK TO SAVEPOINT.............................................................................................1087SAVEPOINT........................................................................................................................1088SELECT.............................................................................................................................. 1090SELECT INTO.................................................................................................................... 1104SET..................................................................................................................................... 1105SET ROLE.......................................................................................................................... 1107SET SESSION AUTHORIZATION..................................................................................... 1108SET TRANSACTION.......................................................................................................... 1109SHOW................................................................................................................................. 1111START TRANSACTION..................................................................................................... 1112TRUNCATE.........................................................................................................................1113UPDATE..............................................................................................................................1114VACUUM.............................................................................................................................1117VALUES.............................................................................................................................. 1119

SQL 2008 Optional Feature Compliance......................................................................................1122Greenplum Environment Variables............................................................................................... 1151

Required Environment Variables........................................................................................1151Optional Environment Variables......................................................................................... 1151

System Catalog Reference........................................................................................................... 1153System Tables.................................................................................................................... 1153System Views..................................................................................................................... 1154System Catalogs Definitions...............................................................................................1155

The gp_toolkit Administrative Schema..........................................................................................1240Checking for Tables that Need Routine Maintenance........................................................1240Checking for Locks.............................................................................................................1241Checking Append-Optimized Tables.................................................................................. 1243Viewing Greenplum Database Server Log Files................................................................ 1247Checking Server Configuration Files..................................................................................1250Checking for Failed Segments........................................................................................... 1251Checking Resource Group Activity and Status.................................................................. 1252Checking Resource Queue Activity and Status................................................................. 1254Checking Query Disk Spill Space Usage...........................................................................1256Viewing Users and Groups (Roles)....................................................................................1258Checking Database Object Sizes and Disk Space............................................................ 1259Checking for Uneven Data Distribution.............................................................................. 1263

Contents OpenTopic

11

The gpperfmon Database..............................................................................................................1264database_*.........................................................................................................................1266diskspace_*....................................................................................................................... 1267interface_stats_*................................................................................................................ 1267log_alert_*..........................................................................................................................1269queries_*............................................................................................................................. 1270segment_*..........................................................................................................................1272socket_stats_*.....................................................................................................................1273system_*............................................................................................................................. 1274dynamic_memory_info........................................................................................................ 1276memory_info...................................................................................................................... 1276

Greenplum Database Data Types.................................................................................................1278Character Set Support...................................................................................................................1282

Setting the Character Set...................................................................................................1284Character Set Conversion Between Server and Client...................................................... 1284

Server Configuration Parameters..................................................................................................1287Parameter Types and Values.............................................................................................1287Setting Parameters............................................................................................................. 1287Parameter Categories.........................................................................................................1288Configuration Parameters...................................................................................................1298

Summary of Built-in Functions...................................................................................................... 1378Greenplum Database Function Types................................................................................1378Built-in Functions and Operators........................................................................................1379JSON Functions and Operators......................................................................................... 1382Window Functions.............................................................................................................. 1385Advanced Aggregate Functions......................................................................................... 1387

Greenplum MapReduce Specification...........................................................................................1389Greenplum MapReduce Document Format........................................................................1389Greenplum MapReduce Document Schema......................................................................1390Example Greenplum MapReduce Document..................................................................... 1397

Greenplum PL/pgSQL Procedural Language............................................................................... 1403About Greenplum Database PL/pgSQL............................................................................. 1403PL/pgSQL Plan Caching.....................................................................................................1405PL/pgSQL Examples...........................................................................................................1405References..........................................................................................................................1409

Greenplum PostGIS Extension..................................................................................................... 1410About PostGIS.................................................................................................................... 1410Enabling and Removing PostGIS Support......................................................................... 1411Usage..................................................................................................................................1412PostGIS Extension Support and Limitations...................................................................... 1413PostGIS Support Scripts.....................................................................................................1414

Greenplum PL/R Language Extension..........................................................................................1417About Greenplum Database PL/R......................................................................................1417

Greenplum PL/Python Language Extension................................................................................. 1423About Greenplum PL/Python..............................................................................................1423Enabling and Removing PL/Python support...................................................................... 1423Developing Functions with PL/Python................................................................................1424Installing Python Modules...................................................................................................1427Examples............................................................................................................................ 1430References..........................................................................................................................1432

Greenplum PL/Container Language Extension.............................................................................1433About the PL/Container Language Extension.................................................................... 1433PL/Container Docker Images............................................................................................. 1434Prerequisites....................................................................................................................... 1434Installing the PL/Container Language Extension............................................................... 1434Installing PL/Container Docker Images.............................................................................. 1436

Contents OpenTopic

12

Uninstalling PL/Container................................................................................................... 1437Using PL/Container.............................................................................................................1438About PL/Container Running PL/Python............................................................................ 1440About PL/Container Running PL/R.....................................................................................1441Configuring PL/Container....................................................................................................1442Installing Docker................................................................................................................. 1451References..........................................................................................................................1452

Greenplum PL/Java Language Extension.....................................................................................1453About PL/Java.................................................................................................................... 1453About Greenplum Database PL/Java.................................................................................1454Installing PL/Java................................................................................................................1455Uninstalling PL/Java........................................................................................................... 1456Enabling PL/Java and Installing JAR Files........................................................................ 1457Writing PL/Java functions................................................................................................... 1457Using JDBC........................................................................................................................ 1463Exception Handling.............................................................................................................1463Savepoints.......................................................................................................................... 1463Logging............................................................................................................................... 1464Security............................................................................................................................... 1464Some PL/Java Issues and Solutions..................................................................................1465Example.............................................................................................................................. 1466References..........................................................................................................................1467

Greenplum PL/Perl Language Extension......................................................................................1468About Greenplum PL/Perl...................................................................................................1468Greenplum Database PL/Perl Limitations.......................................................................... 1468Trusted/Untrusted Language.............................................................................................. 1468Enabling and Removing PL/Perl Support...........................................................................1469Developing Functions with PL/Perl.....................................................................................1469

Greenplum MADlib Extension for Analytics.................................................................................. 1473About MADlib......................................................................................................................1473Installing MADlib................................................................................................................. 1473Upgrading MADlib...............................................................................................................1474Uninstalling MADlib.............................................................................................................1475Examples............................................................................................................................ 1475References..........................................................................................................................1481

Greenplum Fuzzy String Match Extension....................................................................................1483Soundex Functions............................................................................................................. 1483Levenshtein Functions........................................................................................................1484Metaphone Functions......................................................................................................... 1484Double Metaphone Functions.............................................................................................1485Installing and Uninstalling the Fuzzy String Match Functions............................................ 1485

Summary of Greenplum Features.................................................................................................1486Greenplum SQL Standard Conformance........................................................................... 1486Greenplum and PostgreSQL Compatibility.........................................................................1488

Chapter 16: Greenplum Database UNIX Client Documentation........ 1497Greenplum Database Client Tools for UNIX.................................................................................1498

Installing the Greenplum Client Tools................................................................................ 1498Client Tools Reference.......................................................................................................1501

Greenplum Database Load Tools for UNIX..................................................................................1502Installing the Greenplum Load Tools................................................................................. 1502Load Tools Reference........................................................................................................ 1503

Chapter 17: Greenplum Database Windows Client Documentation..1505

Contents OpenTopic

13

Greenplum Database Client Tools for Windows...........................................................................1506Installing the Greenplum Client Tools................................................................................ 1506Running the Greenplum Client Tools.................................................................................1509Client Tools Reference.......................................................................................................1510

Greenplum Database Load Tools for Windows............................................................................ 1512Installing Greenplum Loader.............................................................................................. 1512Running Greenplum Loader............................................................................................... 1514Running gpfdist as a Windows Service..............................................................................1518Loader Program Reference................................................................................................ 1519

Chapter 18: DataDirect ODBC Drivers for Pivotal Greenplum...........1520Prerequisites.................................................................................................................................. 1521Supported Client Platforms........................................................................................................... 1522Installing on Linux Systems.......................................................................................................... 1523

Configuring the Driver on Linux......................................................................................... 1524Testing the Driver Connection on Linux.............................................................................1525

Installing on Windows Systems.................................................................................................... 1526Verifying the Version on Windows..................................................................................... 1526Configuring and Testing the Driver on Windows................................................................1526

DataDirect Driver Documentation..................................................................................................1528

Chapter 19: DataDirect JDBC Driver for Pivotal Greenplum............. 1529Prerequisites.................................................................................................................................. 1530Downloading the DataDirect JDBC Driver.................................................................................... 1531Obtaining Version Details for the Driver....................................................................................... 1532Usage Information......................................................................................................................... 1533DataDirect Driver Documentation..................................................................................................1534

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

14

Chapter 2

Pivotal Greenplum 5.6.1 Release Notes

Updated: June, 2018

• Welcome to Pivotal Greenplum 5.6.1• New Features• Changed Features• Experimental Features• Differences Compared to Open Source Greenplum Database• Supported Platforms• Pivotal Greenplum Tools and Extensions Compatibility• Hadoop Distribution Compatibility• Upgrading to Greenplum Database 5.6.1• Migrating Data to Pivotal Greenplum 5.x• Pivotal Greenplum on DCA Systems• Resolved Issues• Known Issues and Limitations• Update for gp_toolkit.gp_bloat_diag Issue

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

15

Welcome to Pivotal Greenplum 5.6.1Pivotal Greenplum Database is a massively parallel processing (MPP) database server that supports nextgeneration data warehousing and large-scale analytics processing. By automatically partitioning dataand running parallel queries, it allows a cluster of servers to operate as a single database supercomputerperforming tens or hundreds times faster than a traditional database. It supports SQL, MapReduce parallelprocessing, and data volumes ranging from hundreds of gigabytes, to hundreds of terabytes.

This document contains pertinent release information about Pivotal Greenplum Database 5.6.1. Forprevious versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 5.x software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 5.x is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

Pivotal Greenplum 5.6.1 is the first Pivotal release of Greenplum 5.6, which adds and changes severalfeatures.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

16

New FeaturesGreenplum Database 5.6.1 includes these new features and enhancements.

• PL/Container Enhancements• Query Optimization Enhancements• gpbackup and gprestore Enhancements• Parameter to Ignore 4.3.x INTO ERROR TABLE Clause• Experimental Pivotal Greenplum-Informatica Connector• Greenplum Metrics Collection Extension

Query Optimization EnhancementsGreenplum Database 5.6.1 includes these GPORCA enhancements:

• To improve GPORCA query optimization performance, resource usage information is retrieved onlywhen required when generating optimization statistics. In previous releases, the information was beingretrieved whenever the GPORCA timer was reset.

• When optimizing queries against partitioned tables, GPORCA calculates predicate conditions that arepartitioning conditions. To improve optimization performance, these partitioning conditions are cachedand reused during query optimization. In previous releases, the partitioning conditions were calculatedmultiple times during query optimization.

• To improve query optimization, GPORCA exhaustive join order generation maintains a top-n list oforder alternatives making the join order processing pick the join order with the lowest cost estimatedcost. In previous releases, the ordering of the join alternative did not guarantee the lowest cost joinordering.

PL/Container EnhancementsPL/Container has been updated to version 1.1. Greenplum Database 5.6.1 supports PL/Container 1.1. PL/Container 1.1 includes these enhancements:

• PL/Container is installed as a Greenplum Database extension. PL/Container 1.1 is installed by gppkgas a Greenplum Database extension, while PL/Container 1.0 extension is installed as a GreenplumDatabase language. Upgrading to PL/Container 1.1 from 1.0 requires uninstalling PL/Container 1.0.

• PL/Container configuration supports adjusting the CPU usage for Docker containers that are controlledby PL/Container.

In the PL/Container configuration file, the cpu_share attribute of a settings attribute is a relativeweighting of CPU usage compared to other containers. For example, a container with a cpu_share of2048 is allocated double the CPU slice time compared with container with the default value of 1024.

For information about the PL/Container extension in Greenplum Database including upgrading to PL/Container 1.1, see PL/Container Language Extension in the Greenplum Database Reference Guide.

gpbackup and gprestore EnhancementsGreenplum Database 5.6.1 includes these gpbackup and gprestore enhancements.

• The gpbackup and gprestore utilities are included in the Greenplum Databse SuSE distribution.• The utilities support Data Domain DD Boost File System Plug-In (BoostFS). For information about using

BoostFS, see Using gpbackup and gprestore with BoostFS in the Greenplum Database AdministratorGuide.

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

17

Parameter to Ignore 4.3.x INTO ERROR TABLE ClauseThe Greenplum Database server configuration parameter gp_ignore_error_table controls GreenplumDatabase behavior when the INTO ERROR TABLE clause is specified in a CREATE EXTERNAL TABLE orCOPY command. The INTO ERROR TABLE clause was removed in Greenplum Database 5.

The default value is false, Greenplum Database returns an error if the INTO ERROR TABLE clause isspecified in a command.

If the value is true, Greenplum Database ignores the clause, issues a warning, and executes thecommand and logs information about data rows with formatting errors. In Greenplum Database 5.x, youaccess the error log information with built-in SQL functions. See the CREATE EXTERNAL TABLE or COPYcommand.

You can set this value to true to avoid the Greenplum Database 5 error when you run applications thatexecute CREATE EXTERNAL TABLE or COPY commands that include the Greenplum Database 4.3.x INTOERROR TABLE clause.

Value Range Default Set Classifications

Boolean false master

system

reload

For information about Greenplum Database server configuration parameters, see Server ConfigurationParameters in the Greenplum Database Reference Guide.

Experimental Pivotal Greenplum-Informatica ConnectorGreenplum Database 5.6 provides integration with the new Pivotal Greenplum-Informatica Connector(experimental). The Pivotal Greenplum-Informatica Connector supports high speed data transfer from anInformatica PowerCenter cluster to a Pivotal Greenplum Database cluster for batch and streaming ETLoperations.

Greenplum Database 5.6 introduces the Greenplum Streaming Server (GPSS) component, whichmanages all communication between the Informatica PowerCenter node and the Pivotal GreenplumDatabase cluster. See the Pivotal Greenplum-Informatica Connector (Experimental) Documentation.

Greenplum Metrics Collection ExtensionThe Greenplum Metrics Collection Extension has been added to this release. This extension is aprerequisite for an upcoming Pivotal Greenplum Command Center release on Red Hat and CentOSplatforms, and is not yet enabled.

The extension is installed in the $GPHOME/share/postgresql/extension/metrics_collectiondirectory, and its library files are installed in the $GPHOME/lib/postgresql directory.

The server configuration parameter shared_preload_libraries in the master and segmentpostgresql.conf files is updated to include the name of the extension's shared library,metrics_collection.so.

https://greenplum-informatica.docs.pivotal.io

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

18

Changed FeaturesGreenplum Database 5.6.1 includes these changed features.

• For PL/Container 1.1, the configuration attribute use_container_network is not supported. PL/Container 1.1 determines the appropriate network protocol. When upgrading to PL/Container 1.1 theattribute must be removed from the configuration file. For information about PL/Container 1.1, see NewFeatures.

• For the Python installed with Greenplum Database, the Python paramiko library has been updated toversion 1.18.1. The update helps resolve issues with cryptography libraries used on RHEL 7.4 systemswith FIPS enabled. See resolved issue 29089.

• Resource group cgroup configuration has changed. The gpdb.conf cgroup configuration file mustnow include a memory control group. The memory control group is a placeholder for future supportof cgroup-based memory management for Greenplum Database external components. If you haveenabled resource group-based resource management in your current Greenplum Database installation,you must re-configure cgroups. Refer to Upgrading to Greenplum Database 5.6.1 for resource groupcgroup upgrade instructions. See Using Resource Groups in the Greenplum Database AdministratorGuide for general resource group configuration and usage information.

../admin_guide/workload_mgmt_resgroups.html#topic71717999

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

19

Experimental FeaturesBecause Pivotal Greenplum Database is based on the open source Greenplum Database project code,it includes several experimental features to allow interested developers to experiment with their use ondevelopment systems. Feedback will help drive development of these features, and they may becomesupported in future versions of the product.

Warning: Experimental features are not recommended or supported for production deployments.These features may change in or be removed from future versions of the product based on furthertesting and feedback. Moreover, any features that may be visible in the open source code butthat are not described in the product documentation should be considered experimental andunsupported for production use.

Greenplum Database 5.6.1 includes these experimental features:

• Integration with the Pivotal Greenplum-Informatica Connector (experimental). The connector supportshigh speed data transfer from an Informatica PowerCenter cluster to a Pivotal Greenplum Databasecluster for batch and streaming ETL operations. See the Pivotal Greenplum-Informatica Connector(Experimental) Documentation.

• Recursive WITH Queries (Common Table Expressions). See WITH Queries (Common TableExpressions) in the Pivotal Greenplum Database Documentation.

• Resource groups remain an experimental feature only on the SuSE 11 platform, due to limited cgroupsfunctionality in the kernel.

SuSE 12 resolves the Linux cgroup issues that caused the performance degradation when GreenplumDatabase resource groups are enabled.

http://greenplum.org/https://greenplum-informatica.docs.pivotal.iohttps://greenplum-informatica.docs.pivotal.iohttps://gpdb.docs.pivotal.io/

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

20

Differences Compared to Open Source GreenplumDatabase

Pivotal Greenplum 5.x includes all of the functionality in the open source Greenplum Database project andadds:

• Product packaging and installation script.• Support for QuickLZ compression. QuickLZ compression is not provided in the open source version of

Greenplum Database due to licensing restrictions.• Support for managing Greenplum Database using Pivotal Greenplum Command Center.• Support for monitoring and managing queries with Pivotal Greenplum Workload Manager.• Support for full text search and text analysis using Pivotal GPText.• Spark Connector and Gemfire Connector• Data Direct ODBC/JDBC Drivers

http://greenplum.org/

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

21

Supported PlatformsPivotal Greenplum 5.6.1 runs on the following platforms:

• Red Hat Enterprise Linux 64-bit 7.x (See the following Note)• Red Hat Enterprise Linux 64-bit 6.x• SuSE Linux Enterprise Server 64-bit 12 SP2 and SP3 with kernel version greater than 4.4.73-5. (See

the following Note)• SuSE Linux Enterprise Server 64-bit 11 SP4 (See the following Note)• CentOS 64-bit 7.x• CentOS 64-bit 6.x

Note: For the supported Linux operating systems, Pivotal Greenplum Database is supportedon system hosts using either AMD or Intel CPUs based on the x86-64 architecture. Pivotalrecommends using a homogeneous set of hardware (system hosts) in a Greenplum Databasesystem.

Important: Significant Greenplum Database performance degradation has been observed whenenabling resource group-based workload management on Red Hat 6.x, CentOS 6.x, and SuSE11 systems. This issue is caused by a Linux cgroup kernel bug. This kernel bug has been fixed inCentOS 7.x and Red Hat 7.x systems.

If you use Red Hat 6 and the performance with resource groups is acceptable for your use case,upgrade your kernel to version 2.6.32-696 or higher to benefit from other fixes to the cgroupsimplementation.

SuSE 11 does not have a kernel version that resolves this issue; resource groups are stillconsidered to be an experimental feature on this platform. Resource groups are not supported onSuSE 11 for production use. See known issue 149789783.

Pivotal Greenplum on SuSE 12 supports resource groups for production use. SuSE 12 resolves theLinux cgroup kernel issues that caused the performance degradation when Greenplum Databaseresource groups are enabled.

Note: For Greenplum Database that is installed on Red Hat Enterprise Linux 7.x or CentOS 7.xprior to 7.3, an operating system issue might cause Greenplum Database that is running largeworkloads to hang in the workload. The Greenplum Database issue is caused by Linux kernel bugs.

RHEL 7.3 and CentOS 7.3 resolves the issue.

Note: Greenplum Database on SuSE Linux Enterprise systems does not support these features.

• The PL/Perl procedural language• The gpmapreduce tool• The PL/Container language extension• The Greenplum Platform Extension Framework (PXF)

Greenplum Database support on Dell EMC DCA.

Note: This release of Pivotal Greenplum Database is not certified for use on Dell EMC DCAsystems, because of an incompatibility in configuring timezone information. This problem is beingtracked in issues 29368 and 29320. DCA customers should continue to use Greenplum Databaseversion 5.4.1 or earlier until this issue is resolved in later releases of the product.

Pivotal Greenplum 5.6.1 supports these Java versions:

• 8.xxx• 7.xxx

Pivotal Greenplum 5.6.1 Release Notes OpenTopic

22

Greenplum Database 5.6.1 software that runs on Linux systems uses OpenSSL 1.0.2l (with FIPS 2.0.16),cURL 7.54, OpenLDAP 2.4.44, and Python 2.7.12.

Greenplum Database client software that runs on Windows and AIX systems uses OpenSSL 0.9.8zg.

The Greenplum Database s3 external table protocol supports these data sources:

• Amazon Simple Storage Service (Amazon S3)• Dell EMC Elastic Cloud Storage (ECS), an Amazon S3 compatible service

Pivotal Greenplum 5.6.1 supports Data Domain Boost on Red Hat Enterprise Linux.

This table lists the versions of Data Domain Boost SDK and DDOS supported by Pivotal Greenplum 5.x.

Table 1: Data Domain Boost Compatibility

Pivotal Greenplum Data Domain Boost2 DDOS

5.5.0

5.4.0

5.2.0

5.1.0

5.0.0

3.3

3.0.0.316.1 (all versions)

6.0 (all versions)

Note: In addition to the DDOS versions listed in the previous table, Pivotal Greenplum 5.0.0 andlater supports all minor patch releases (fourth digit releases) later than the certified version.1Support for Data Domain Boost 3.0.0.3 is deprecated.