chicago z council december 4, 2018 - ibm · ibm’ statements regarding its plans, directions, and...

© 2018 IBM Corporation

Db2 for z/OS: What’s New?

Mark Rader Tadas Varaneckas Dominique Parker

Db2 for z/OS, IBM WSC Db2 for z/OS, IBM Cloud Db2 for z/OS, IBM WSC

Chicago Z Council

December 4, 2018

© 2018 IBM Corporation2 © 2018 IBM Corporation

Db2 for z/OS: What’s New?

Transparent Data Set Encryption

Mark Rader [email protected]

Db2 DevOps Experience for z/OS

Tadas Varaneckas [email protected]

Db2ZAI – Db2 and Machine Learning

Dominique Parker [email protected]

mailto:[email protected]



Please note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice and at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.

The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

3


Db2 for z/OS: Transparent Data Set

Encryption

Mark Rader

Db2 for z/OS, IBM Washington Systems Center

Chicago Z Council

December 4, 2018

IBM z Analytics

5

Today’s security challenges

European Union General Data Protection Regulation

Payment Card Industry Data Security Standard

Health Insurance Portability and Accountability Act

Source: 2017 Ponemon Cost of Data Breach Study: Global Analysis --http://www.ibm.com/security/data-breach/

1,2

$ 3.62MAverage cost of a data breach in 2017 1

27.7%Likelihood of an organization having a data breach over the next two years 2

IBM z Analytics

6

Multiple layers of encryption of data

Coverage

IBM z Analytics

7

Db2 support of z/OS data set encryption and pervasive encryption

• Db2 11 and Db2 12 support transparent data set encryption (TDE)

• Without database downtime or requiring the administrator to redefine objects which could cause disruption to operations

• No application changes required

• Encrypt active and archive log datasets

• Encrypt catalog and directory table spaces

• Encrypt user table spaces

• Utilizes new z/OS DFSMS data set encryption support delivered in z/OS 2.3 and z/OS 2.2 (with OA50569 and OA53951)

• Extended format data sets only

• Db2 12 V12R1M502 adds additional controls to set up encryption policies using Db2 interfaces

IBM z Analytics

8

Understanding DFSMS policy-based dataset encryption (reference)

Data sets are defined as encrypted by specifying a key label during the creation of a new data set:

• RACF data set profile

• JCL, Dynamic Allocation, TSO Allocate, IDCAMS DEFINE

• SMS DATACLAS

During data set open, DFSMS:

• Checks the user access to the key label

• Specifies the key label to ICSF to retrieve the secure / protected key from the CKDS

ICSF:

• Locates the secure key in the CKDS using the key label specified by DFSMS

• Calls the adapter to unwrap the key value from the Master key

• Rewraps the key value under a CPACF wrapping key to make it a protected key

• Protected key stored in ICSF cache

IBM z Analytics

9

DFSMS dataset encryption

•Application transparency

• Data remains encrypted during backup/recovery, migration/recall

• In memory system or application data buffers remain in the clear

• Access to the key label is controlled through SAF permissions, in addition to traditional data set permissions

•Segregation of duties

• Storage administrators need access to the data set but not access to the key label

IBM z Analytics

10

Steps to enable encryption

Generates an encryption key and key label.Stores it in the ICSF CKDS

ICSF AdminSets up RACF for use of key label

Allows secure key to be used as protected key via ICSF segment- SYMCPACFWRAP- SYMCPACFRET

Permits Db2 use of the key label

– AND –

In RACF, permit access to new resource in FACILITY class

Security Admin

Online REORG

Migrate to encrypted data

Non-disruptive

Database Admin

Associate the key label with the desired data set.

In RACF, alter DFP segment in data set profile - DATAKEY()

IDCAMS DEFINE, TSO Allocate, etc. (V12 only) In Db2, set key label for system objects and user objects

– OR –

In DFSMS, assign to data class

– OR –

Security / Database / Storage Admin

1 2 3 4

IBM z Analytics

11

Encrypting Db2 system objects (see prior page)

•Options to define a key label used by Db2 (precedence order):

• 1. Security Admin can set a key label in the DFP segment of RACF data set profile using the new DATAKEY keyword

• 2. Database System Admin can set a key label using ENCRYPTION_KEYLABEL system parameter (V12R1M502 only)

• -SET SYSPARM command is required for the zparm value to take effect

• Group scope: Takes effect on all the members of a data sharing group immediately

• Security related parameter: Requires installation SYSADM or SECADM authority to set the zparm

• Db2 DBM1 and MSTR address space IDs must be permitted access to the key label

OR

• 2. Storage Admin can set a key label using IDCAMS DEFINE

• Only option to encrypt active logs; ENCRYPTION_KEYLABEL will not apply

• 3. Storage Admin can set a key label in the DFSMS data class

3


– OR –


In Db2, set key label using system parameter

ORIDCAMS DEFINE, etc.

– OR –

Security / Database System Admin / Storage Admin

IBM z Analytics

12

Encrypting Db2 system objects

•Active logs

• Encrypt new active logs

• Define active log data set as encrypted and issue the SET LOG command NEWLOG option to add the newly defined active log data set to the active log inventory without stopping Db2

• Encrypt all active logs

• Stop Db2. Copy the contents of the active log data set to an encrypted data set. Restart Db2.

•Archive logs

• New archive logs automatically encrypted based on the key label setting

• Must be defined on disk

•Catalog and directory table spaces

• Execute REORG TABLESPACE utility to encrypt table spaces and index spaces in DSNDB06 and DSNDB01

• Encrypt DSNDB01.SYSUTILX – Execute RECOVER utility followed by REBUILD INDEX ALL.

Online REORG

Database Admin

4

Can use ENCRYPTION_KEYLABELsystem parameter

IBM z Analytics

13

Encrypting user objects

•Options to define a key label for user objects encryption (precedence order):

• 1. Security Admin can set a key label in the RACF data set profile DFP segment using the new DATAKEY keyword

• 2. Storage Admin (or Database Admin) can set a key label via IDCAMS, TSO, etc…

OR

• 2. Application Database Admin can set a key label using SQL interfaces:CREATE / ALTER with STOGROUP / TABLE (V12R1M502 only)

• Enabled with APPLCOMPAT V12R1M502

• 3. Storage Admin can set a key label in the DFSMS data class


– OR –


Security / Database / Storage Admin

– OR –

IDCAMS DEFINE, etc.OR

In Db2, set key label using SQL interfaces

3

IBM z Analytics

14

Utilities considerations•Db2 managed table spaces and index spaces

• Utilities used to convert to encrypted data sets (except when REUSE option is specified)

• REORG TABLESPACE or REORG INDEX

• LOAD REPLACE

• REBUILD INDEX

• RECOVER from image copies – PIT or full recovery

• PART or DSNUM option to encrypt / decrypt at the partition level

•User managed table spaces and index spaces

• IDCAMS DELETE / DEFINE with the KEYLABEL option

• Execute RECOVER and/or REBUILD INDEX utilities to restore the data

•FlashCopy image copies (FCIC), DFSMSdss concurrent image copies, shadow data sets

• Allocated with the same key label as the table space or index

IBM z Analytics

15

Db2 data set encryption considerations

Compression

• Db2 compression works seamlessly with data set encryption

• Compression is performed first

Performance

• There will be some CPU cost for encryption

• Internal benchmarks show significant savings on z14 vs. z13

UI55093

IBM z Analytics

16

Data set encryption summary

•For Db2 11 and Db2 12, a key label can be defined by the security administrator or storage administrator, a data base administrator can use Db2 REORG utility to seamlessly migrate Db2 data sets to encrypted data sets with no application outages

•For Db2 12, function level V12R1M502 provides new Db2 interfaces to configure and manage Db2 key labels

•Make sure all disaster recover user ids and sites have access to any key labels used to protect Db2 data sets and the key management system is fully deployed across the enterprise

•Recommendation: plan and implement enterprise security and encryption strategy

Thank you

Further reference:

Db2 V11 APAR PI81900 (UI51358)Db2 V12 base APAR PI81907 (UI51499)

https://www.ibm.com/support/knowledgecenter/SSEPEK_11.0.0/seca/src/tpc/db2z_dfsmsencryptionsupport.html

Db2 V12 V12R1M502 APAR PI95511 (UI55093)

https://www.ibm.com/support/knowledgecenter/en/SSEPEK_12.0.0/wnew/src/tpc/db2z_fl_v12r1m502.html#db2z_fl_v12r1m502__48635

Systems Technical University 2018 / © 2018 IBM Corporation

17

https://www.ibm.com/support/knowledgecenter/SSEPEK_11.0.0/seca/src/tpc/db2z_dfsmsencryptionsupport.html

https://www.ibm.com/support/knowledgecenter/en/SSEPEK_12.0.0/wnew/src/tpc/db2z_fl_v12r1m502.html#db2z_fl_v12r1m502__48635


Db2 DevOps Experience for z/OS

Tadas Varaneckas

Db2 for z/OS, IBM Cloud

Chicago Z Council

December 4, 2018


• Enterprises need to innovate rapidly to capitalize on new market opportunities, and reducethe cycle time to collect and react to customer feedback.

• DevOps methods provide the ability for continuous software delivery and management.

What is DevOps?

for faster time to value

for increased capacity to innovate

for improved customer experience

Process

Culture

Technology

Business and Practitioner Adoption of Lean and Agile principles

Business are Seeking to: Driving Changes to:

19


Why DevOps matters

Source: Forrester20


• Lines of Business (LOB) owners can respond quickly to customer needs and competition with faster delivery of new or changed Db2 applications

• IT can establish and enforce the criteria for App Dev to create their own Db2 for z/OS test environments, and retains control of rules and standards that protect Db2 databases and data

• App Dev avoids the wait time for IT to create Db2 for z/OS test environments, and can work in teams using more modern agile development methods

IBM Db2 DevOps Experience for z/OSBrings Db2 applications to market faster with lower costs and less risk

21


Where does the Db2 DevOps Experience for z/OS fit?

Db2 for z/OS

• Allows Db2 for z/OS to participate in customer DevOps pipelines

• Db2 DDL as code• Application and Infrastructure have

been managed as code• Now Db2 fits this model

• On demand, self service provisioning and deployment of Db2 objects and data

• Enable the upstream and downstream portions of DevOps

• With controls to enable object handling, limits, and approvals for deployment

• UI and REST API support• Fits into customer’s existing DevOps

tooling

22


• Need DBA to create Db2 environment• Creates / changes applications• Need DBA to reset test environment• Application deadlines missed

• Create mainframe test environment for application development

• Cannot service # of app dev requests• Critical DBA tasks not addressed• Application deadlines missed

• Business rules• Enterprise data

App Dev

DBA

Before IBM Db2 DevOps Experience for z/OS

DB2

DBA controls DB2 rules and data

23


IBM Db2 DevOps Experience for z/OS – Features

• Ability to create and delete Db2 for z/OS sandbox environment for testing without IT time and assistance

• Make Db2 syntax changes using object-aware Data Definition Language (DDL) editor

• Integrate Db2 database object management into existing continuous delivery development processes

App Dev/Engineer DBA

• Define teams of users, environments, and application components for developers to use in their application development

• Discover and select application’s Db2 database components to be managed and processed together

• Set rules and storage limits for the defined teams

• Review and approve developer schema changes

zowe.org© 2018 IBM Corporation24


IBM Db2 DevOps Experience for z/OS

Ready

© 2018 IBM Corporation25


• Create own Db2 test environment• Resets test environment when needed• Delete test environment when complete• Application delivered on time!

• Set up policies for APP DEV to self-create Db2 test environment

• Policy limits frees DBA from app service requests• Can focus on critical Db2 tasks• Retain control over Db2 databases• Application delivered on time!

• Business rules• Enterprise data

App Dev

DBA

After IBM Db2 DevOps Experience for z/OS

DB2

DBA controls DB2 rules and data

26



• A platform for the definition administration and operation of DevOps services in support of Db2 for z/OS.

• Establish the Users, Teams, and Subsystems that will participate in DevOps

• Define Site Rules to be used in the naming and management of objects under DevOps control

• Register the Subsystem & establish Environments which will be the targets for provisioning operations

• Define Applications as set of Db2 for z/OS objects- Establishes a master branch in Git for those objects

Users

Teams

Subsystems

Environments

Applications

Site Rules / Limits

Instances

27



• Provision instances of the Application into defined Environments

- Will create an Instance of the Application tables- Optionally copy data into the provisioned Instance

objects• DDL for the Provisioned instance can be modified

and deployed to the Instance for testing• Deprovision object to start over (fail fast)• Commit changes to the master branch when

preparing to deploy to other landscapes- The UI/API will issue a Git pull request where

Approvals can be added to the process before being accepted

Users

Teams

Subsystems

Environments

Applications

Site Rules / Limits

Instances

28


Additional References

Db2 Tools for z/OS News – Announcement Replayhttps://ibm.co/2R31zwD

IBM Db2 DevOps Experience for z/OS White Paperhttps://bit.ly/2znfTcx

IBM Marketplacehttps://www.ibm.com/us-en/marketplace/db2-devops-experience

29

https://ibm.co/2R31zwD

https://bit.ly/2znfTcx

https://www.ibm.com/us-en/marketplace/db2-devops-experience


Machine Learning and Db2 for z/OS:

the Db2 Optimizer

Dominique Parker

Db2 for z/OS, IBM Washington Systems Center

Chicago Z Council

December 4, 2018


Agenda

The query optimization challenge

A brief overview of machine learning

Db2 AI for z/OS: exploiting machine learning for query optimization


The query optimization challenge


One aspect of the optimization challenge

Determining the degree of row-filtering for each predicate (i.e., the filter factor) is a

key input to the optimizer’s access path cost formulas

Question: how many of you were born before… parameter marker?

It depends…

• If the substitution value is ‘1900-01-01’, the answer is: nobody

• If the substitution value is ‘2018-10-01’, the answer is: everybody

When a query has a range predicate with a parameter marker or host variable…

…the Db2 optimizer uses a default filter factor (these are documented)

BIRTHDATE < ?


The trouble with default filter factors

They may not ALWAYS predict the way an organization accesses its Db2 data

Analogy: transportation recommendation for a local trip

To get from point A to point B, should you take a personal vehicle, bus, commuter rail?

Answer depends not only on factors such as distance to be traveled and time of day, but also on

the particular area in which the trip will be taken

• Best answer for New York may not be best for Dallas, best for Los Angeles may not be best for Charlotte

• It would be nice if we could provide a location-specific answer…


Tuning just ONE query: the user challenge

Access path tuning can be a very complex, time-consuming task

Depends on several factors (user skill level, available tooling, etc.)

Solutions that we like to see:

Determine right RUNSTATS options for targeted database objects

• Sources: SYSIBM.SYSSTATFEEDBACK (system-level) or DSN_STAT_FEEDBACK (query-level)

Indexing – add or modify

Add OPTIMIZE FOR n ROWS to query (“paging-style” SQL, only part of result set fetched)

Solutions that are less preferred:

BIND/REBIND with APREUSE or OPTHINTS

Query rewrites

Manipulating catalog statistics

Query tricks (OR 0=1 for example)

Skill often missing

Organizations often revert to these


REBIND – user challenge

Suppose you REBIND 1000 static packages using APCOMPARE(WARN)

After migration or maintenance apply

If 90% show no change, and 10% (100 packages) have differences, how do you

validate that changes are acceptable?

Review changes identified in PLAN_TABLE.REMARKS due to APCOMPARE(WARN)

• Do access paths look better (e.g., increase in MATCHCOLS)? Any red flags?

If access path change looks OK, go with new instance of package

What if access path change does NOT look OK?

• APREUSE, based on prior instance of package?

• Create OPTHINT?

• Try to tune affected queries?

What if access path change validation work has to be done for several hundred

packages (or more)?

Approaches most commonly taken


Users want performance, but also stability

Db2 for z/OS optimizer

Is a cost-based model

Has evolved and improved over 35+ years.

• User feedback

• Internal analysis by Db2 development

Provides stability and reliability

• RUNSTATS utility provides inputs to the optimizer cost model

• Static SQL, Dynamic Statement Cache, and (with Db2 12) dynamic plan stability persist access path

choices

• Plan management (for static SQL) enables quick and easy switch to earlier instance of a package

in the event of an access path change that negatively impacts performance


Optimizer development goal

Improve performance and/or

stability of access path

choices…

…without compromising the

quality and stability of existing

access path choices


A brief overview of machine learning


What is machine learning?

A twist on traditional data processing The basic areas of activity related to

exploitation of machine learning technology

Identify

patterns in

datanot readily

discerned by

humans

Build

predictive

modelsof behavior

from those

patterns

Obtain “scores” from

deployed modelsthat predict behavior

Computer

data

programoutput

Traditional data processing

Computer

data

outputprogram*

Machine learning

* How can you get from these data input values to those data output values?


How does Db2 optimizer exploit machine learning?

Machine learning…• Constantly learns and adapts

• Avoids making the same mistakes

• Provides faster, deeper, improved insights

Resulting in… More-favorable business outcomes

Reduced risks and costs

New opportunities


A TrainOps (DevOps) story

Model trainingFeature

engineering

Feature

engineering Scoring

Labeled examples

Training

Scoring

Newdata

Model

ModelPredicted

data

Deploy

Data Scientist/Data analyst

Operational system

Dev

Ops

Training, deployment, scoring


Db2 AI for z/OS: exploiting machine learning

for query optimization


Long-term goal for machine learning within Db2 for z/OS

Make Db2 self-tuning and self-managing

Query optimization

Rebinds

Memory and storage allocations (buffer pools, work files, RID pool, sort pool, etc.)

Utility execution (REORG, RUNSTATS,…)

Table space and index PCTFREE/FREEPAGE specifications based on insert patterns

Fast traverse block (FTB) usage

Analytics Accelerator query offload decisioning

Application of maintenance (how, when, with what expected benefits?)

…


Initial use case – access path selection

Construct machine learning models to assist Db2 optimizer in choosing best

access path for a query

Models are based on actual costs observed for queries that are executed on

target system

What that means: the optimizer is customized for the needs of a particular environment and

workload

Machine learning models stay current and adapt to changing conditions


Machine learning and access path selection – phase one

Initial focus on optimizer cost model – could it be improved via machine learning?

Analysis using supervised learning techniques (neural networks, random forest, linear regression)

Analysis was used to validate the existing optimizer cost model

Multiple costing deficiencies found in optimizer base code – will be addressed through future APARs

After that work, focus shifted to using machine learning to improve the quality and

effectiveness of inputs to the optimizer cost model

Machine learning does not replace optimizer (any more than IBM Watson replaces an oncologist)

Watson provides an oncologist with better inputs to patient care decisions – we use machine

learning to provide the Db2 optimizer with better inputs to query access path decisions


Phase two: focusing on what the optimizer doesn’t know

What inputs are currently “estimated” or unknown?

Two of the most important of those factors are:

Execution time values for host variables / parameter markers

Number of rows that will actually be fetched by the application

• Sometimes OPTIMIZE FOR n ROWS has been added to a query – often not…

Among others…


Db2 AI for z/OS (Db2ZAI) enables the Db2 optimizer to determine the best-performing

query access paths based on your workload characteristics, using machine learning

New offering: IBM Db2 AI for z/OS

• Learns patterns from workload data collected in an

organization’s unique operating environment and uses

derived insight in determining optimal access paths for SQL

statements

• Built on top of the IBM Machine Learning for z/OS stack (MLz)

o Leverages all MLz services without requiring data scientist

support

o Db2 generates model training data, deploys and monitors

and retrains models via MLz services

Db2


Db2ZAI – prerequisites

Db2 12 for z/OS

z/OS Db2 JDBC driver (FMID JDBCC12)

Function level V12R1M503 must be activated (APAR PH00506)

Db2ZAI product ID: 5698-CGN

IBM Machine Learning for z/OS, V1.2 (5698-ML1)

Fix pack 1.2.0.1

IBM Open Data Analytics for z/OS (IzODA)


Db2ZAI – architecture

Note: data collected to build predictive models is stored in user-managed tables, but Db2 takes care of “cleaning up” the tables as needed

SQLApplications

“Learning”Task(zIIP Offloaded*)

ModelLifeCycleAutomationSubtasks

PerformanceHistory

Db212forz/OS

- Db2ZAI- MachineLearningforz/OS- z/OS- x86*zIIP Offload

MachineLearningforz/OSV1.2

MLz ModelServicesDb2ZAIUserInterface

MLScoringService*

Db2Optimizer

IBMOpenDataAnalytics*(IZODA)

-x86 or Linux on z * zIIP offload


What’s next?

We’re evaluating the next targeted use cases

Including soliciting customer input

Related, parallel efforts: improving the REBIND experience and boosting

regression avoidance

There is NO commitment on timeline or deliverables – “this is a journey”

Important to know:

The Db2 for z/OS development organization is committed to the safe and cost-effective

exploitation of machine learning technology, for the benefit of Db2-using organizations


Recap

The optimizer’s goal for each new release/enhancement of Db2 for z/OS:

Provide new access path choices that improve performance and/or stability, without

compromising the quality and stability of existing choices

It is understood that improving the BIND/REBIND experience is CRITICAL to

ensuring that optimizer exploitation of machine learning technology does not

degrade performance and/or stability while we’re “learning”


Where to find more information on Db2ZAI

The announcement letter:

https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/1/897/ENUS218-

341/index.html&request_locale=en

The IBM Knowledge Center:

https://www.ibm.com/support/knowledgecenter/SSGKMA_1.1.0/src/ai/ai_home.html

Overview video:

https://www.youtube.com/watch?v=t5fTNxfehQA

https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/1/897/ENUS218-341/index.html&request_locale=en

https://www.ibm.com/support/knowledgecenter/SSGKMA_1.1.0/src/ai/ai_home.html

https://www.youtube.com/watch?v=t5fTNxfehQA


Thank you!

Question?

Contact us:

• Mark [email protected]

• Tadas [email protected]

• Dominique [email protected]




chicago z council december 4, 2018 - ibm · ibm’ statements regarding its plans, directions, and...

Documents