devops: automate db2 schema management

DevOps: Automate Db2 Schema ManagementTIM WELTZER – SOFTWARE ENGINEER – [email protected]

JAE CHUNG – SOFTWARE ENGINEER – [email protected]

1.19.2020

Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Disclaimer

Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of June 2020 and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.

Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any contract.

Copyright © 2020 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies.

THIS PRESENTATION IS FOR YOUR INFORMATIONAL PURPOSES ONLY. CA assumes no responsibility for the accuracy or completeness of the information. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENT “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. In no event will CA be liable for any loss or damage, direct or indirect, in connection with this presentation, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised in advance of the possibility of suchdamages.

2


Db2 DevOps Using JCL Scripting

3

Problem:

• Automate the management of Db2 object changes across multiple Db2 environments including

development, quality assurance, and production subsystems. Enable a self-service capability

for developers (Dev) and minimize the need for DBA (Ops) involvement.

Solution SO13807 is a DevOps enhancement to CA RC/Migrator and CA RC/Compare that helps

to guide you in the building of your own DevOps pipeline to address and automate the needs of

development and operations. The enhancement includes:

• A set of JCL templates

• A sample DevOps JCL workflow using sample JCL and PROCs derived from the templates

• Documentation that describes using JCL and/or open source DevOps tools such as Zowe CLI

and Jenkins to help you build your own workflow.

The set of JCL templates can be used for two different DevOps scripting contexts: JCL only and

open source tooling.

This presentation is focused on a JCL only scripting context.


JCL Creation Templates

4

Four templates are used as building blocks to create a DevOps workflow. The templates can be

used as input to an Apache Velocity compatible template engine. They can also be used as

models to build a pure JCL pipeline. The table below provides a summary of each template.

Template and Purpose CA Product Usage Input Output

RMATM001

Generate DDL

CA RC/Migrator is used to

generate DDL for selected

objects

-Source SSID

-Db2 object selections

-Options

-DDL file

RMATM002

Syntax Check DDL

CA RC/Migrator Syntax

Checker

-DDL File

-target SSID (for object verification)

-Validated DDL file

-Report of invalid DDL

RMATM003

Create an SSID update script

Three step process:

• Create strategy

• Analyze strategy

• Generate Incremental

Change Language (ICL)

CA RC/Compare is used to:

• Create a strategy

• Analyze a strategy

• Generate ICL

• For create strategy step

-DDL, -Target SSID, -Options

• For analyze strategy step

-Strategy, -Target SSID, -Options

• For generate ICL step

-Strategy, -Target SSID, -Options

• For create strategy step

-Compare strategy

• For analyze strategy step

-CA Batch Processor script

• For generate ICL step

-ICL script used as a report

RMATM004

Execute an update script

CA Batch Processor (BP) -Target SSID

-BP script

- The target SSID is updated

according to script


Template RMATM001 – Generate DDL for Selected Objects

5

JCL variables ‘&’

Template engine variables ‘$’

Quick Migrate Db2 object selection

Invoke RC/Migrator

Template engine statements ‘#’

Generate DDL option

Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.6

DEV QA PROD

Example DevOps Workflow Based on Templates

Changes and movement of Db2 Objects are managed in DDL format.

1. Provision DEV with Db2 objects extracted from PRD – uses RMATM001, RMATM003, and RMATM004

2. Iterate: edit, update, and test Db2 object changes – uses RMATM002, RMATM003, and RMATM004

3. Promote Db2 object changes from DEV to QA – uses RMATM001, RMATM003, and RMATM004

4. Perform QA testing

5. Promote Db2 object changes from QA to PROD - Uses RMATM001, RMATM003, and RMATM004

1

53

2

4


Sample DevOps PROC and JCL to Promote Db2 Object Changes

7

Sample DevOps PROC and JCL are provided as an example pipeline which promotes Db2 object

changes from a source environment to a target environment. The table below provides a brief

description of the PROC and JCLs.

JCL/PROC CA Product Used Input Output

RMAPROMP - PROC

Multi-step process to:

• Generate DDL

• Create a strategy

• Analyze a strategy

• Generate ICL

• Send email (via SMTP)

• CA RC/Migrator

• CA RC/Compare

• source SSID

• target SSID

• submitter email address

• Db2 object selection

• BP update script

• ICL report

• Email notifications

RMAPROMJ - JCL

Invoke RMAPROMP PROC

• N/A • source SSID

• target SSID

• submitter email address

• Db2 object selection

• BP update script

• ICL report

• Email notifications

RMAPROMX - JCL

Execute a BP Script

• CA Batch Processor • submitter email address

• LOGSSID

• BP script

• Target SSID is updated

with changes

• Recovery BP script


One-time configuration parameters

Values supplied by RMAPROMJ

SYSUT1 overridden by RMAPROMJ to specify Db2 Objects

RMAPROMJ JCL Invokes RMAPROMP PROC

8

RMAPROMX is an instream

PROC to execute the update

script produced by RMAPROMJ


The Anatomy of the RMAPROMP Procedure

9

RMAPROMP is a sample PROC that shows how to shield a developer from implementation

details while minimizing the inputs needed to promote Db2 object changes from one environment

to another. The steps contained inside RMAPROMP are:

1. .

2. If copy failed then email submitter and exit

3. Generate DDL for selected objects

4. Delete analysis data sets created from a prior execution

5. If DDL generation failed then email submitter and exit

6. Create compare strategy between DDL and subsystem

7. If strategy creation failed then email submitter and exit

8. Analyze compare strategy

9. If analyze failed then email submitter and exit

10. Generate Incremental Change Language (ICL) for reporting

11. Email submitter with success notification

12. Why are you looking under here? Congratulations! You have found an Easter egg!

Input:

object selection

source ssid, target ssid, submitter email

Output:

update script

ICL report

One-time configuration values:

Db2 data sets DSNLOAD and DSNEXIT

HLQ of CA products

Product options

Site domain

Approver email address

Copy object selection to a temporary file

Generate DDL for selected objects

Create compare strategy between DDL and subsystem

Generate Incremental Change Language (ICL) report

Email request-for-approval to approver with attached details: update script and ICL report

Analyze compare strategy to create update script


DevOps Use Case: Promote Db2 Changes to Production

10

The diagram below depicts changes made to a Db2 application’s object hierarchy. It includes object additions

and changes to existing objects.

Developer Initiates Db2 Object Promotion

Execute RMAPROMJ to create an update script specifying:

• SRCSSID=QA ssid

• TRGSSID=PROD ssid

• [email protected]

• Object selection specification: DATABASE CHUJAA5 DBA

RMAPROMJ invokes RMAPROMP which is configured to use the

QUICKME keyword to include all dependent objects.

Administrator Completes Db2 Object Promotion

Execute RMAPROMX to execute an update script specifying:

• [email protected] (to notify developer)

• LOGSSID=PROD ssid

• INPRPRT=update script generated by RMAPROMJ

The result is emailed to the administrator and developer.

The administrator also receives a recovery script which can be

executed to undo the changes if needed.

Execution of Two Jobs Will Promote the

Changes from QA to Production

+TS1B

+TG1B

+TB1B

IX1AAdd column

DBA

TB1AAdd column

Add constraint

SGA

+FK1B1A

+SQ

TS1A

Data Structure Changes

• Solid lines are existing

objects on QA and

production subsystems

• Dashed lines are new

objects on QA

subsystem

• Red text are attributes

added to existing

production objects and

are only on the QA

subsystem


RMAPROMJ Creates Update Script and Emails an Approver

11

+TS1B

+TG1B

+TB1B

IX1AAdd column

DBA

TB1AAdd column

Add constraint

SGA

+FK1B1A

+SQ

TS1A

RMAPROMJ generates an update script.

This snippet shows some of the changes

to be made. Utility calls are not shown.

The DATABASE CHUJAA5 DBA

is the only object that is specified

in RMAPROMJ because the

invoked RMAPROMP PROC is

configured to include all

dependent objects.

RMAPROMJ


Results of RMAPROMJ Emailed to Approver for Approval

12

Snippets of ICL_REPORT.TXTApproval request emailed to approverThis pause in the flow allows a DBA to

approve or reject the changes.

These parameters are used when

submitting RMAPROMX to deploy the

changes by executing the update script

in z/OS data set WELTI01.ANALYSIS.

The ICL report provides a terse report of the

Db2 object changes being made compared

to the update script.


Results of RMAPROMX Emailed to Submitter and Approver

13

Results emailed to submitter and approver

Recovery Script and execution output

Also saved in a z/OS data set

Snippet of recovery script

Snippet of execution output

RMAPROMX

Objects

successfully

promoted!


Other Things You can Include in a DevOps Pipeline Using JCL

14

A DevOps pipeline is not constrained to only Db2 schema management. Some ideas for

other processes (JCL steps) to consider adding to your pipeline include:

• Provisioning environment(s) with test data

• Provisioning environment(s) with production Db2 statistics to tune SQL queries using Explain

• Maintaining a Db2 application in sync with Db2 object changes

• Recording and Reporting Db2 Explain results

• Comparing Explain results between development and production

• Backing-up and building a changed Db2 application i.e. pre-compiling, compiling, linking, Db2

binding

• Enforcing standards

• Tracking DevOps activity

• Archiving Db2 DDL, application source, build processes, etc.

• Etc.

Broadcom Proprietary and Confidential. Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.15

Automate Db2 Schema Management

Live demonstration of an open source tools approach to Db2 schema automation using python, Zowe, Git,

and Jenkins.

Thank You

devops: automate db2 schema management

Documents