Using IBM PureData System for Analytics Workload ManagementSession Number 2839Gordon Booman, IBM, gbooman@us.ibm.com

Torsten Steinbach, IBM, torsten@de.ibm.com

© 2013 IBM Corporation

Outline

Why WLM WLM Principles Feature Overview Usage Scenarios Work Load Analysis Best Practices

PureData for Analytics (Striper, Full Rack)

PureData for Analytics Architecture

SPU Blade 1

Disk 1

Slice 1

Disk 40

Q3

Q2

Dat

a R

ecei

ve

Slice 40

Q1Q3

Q2

Dat

a R

ecei

veCentral Host

...

...

Q1Q3

Q2Load 1

Load 2

BI App• Operational Query: Q1• Analytics Query: Q2 Power User

Heavy ad-hoc query: Q3

ETL 1Load 1

ETL 2Load 2

SPU Blade 6

Disk 1

Slice 1

Disk 40

Q3

Q2

Dat

a R

ecei

ve

Slice 40

Q3

Q2

Dat

a R

ecei

ve

...

...

Standby Host

. . .

Challenging Workload Situations Large amount of concurrent workload

Large queries can run out of memory and temp space Throughput can be worse than with lower concurrency

Concurrent mix of short queries & heavy queries Large queries can starve out the short ones

Concurrent ingest & queries Loads can starve out queries

Rushing (workload shifts) Sudden arrival of large amount of a set of users/app can

monopolize the system Runaway queries

Carelessly submitted heavy queries (e.g. by power user) can occupy system without business value (e.g. cartesian join)

10 min

Resources that matter for Workload Management

Allocation Memory Temporary storage

Utilization CPU I/O bandwidth Network bandwidth

In use even when query is not actively worked on

We call these the fixed resources

In use only when query is actively worked on

We call these the renewable resources

Meeting User Objectives through WLM Simple user-oriented way to specify performance goals

Ability to sub-divide system resources and assign to different users, tenants or applications

Low level control knobs (such as declarative concurrency limits) should not be the primary user model of WLM

Ensure consistent performance for a tenant Don’t “spoil” users just because the system could at the moment Ability to declare maximum resource limit for a tenant

Respect declared relative priorities Allow explicit declaration of query priority by the application/user Higher priority queries always go before lower priority queries

60%25%

15%

The Control Tool Box

Page 7

Admission Sequence

I/O Priority

Process CPU Priority

Delay

Allocation & Concurrency Limits

20 min

Admission Control through Scheduler Queues

GA

TEKEEPER

GR

A

SNIPPET

Disk

Fabric

CPU

JOBS PLANS SNIPPETSPLA

NN

ER

Control admission by priority & duration

Control admission & execution by renewable resource share

Control admission by fixed resource fit

Declaring Priorities

Page 9

Four priority levels: Critical, High, Medium, Low Higher priority queries get served first within the

same resource sharing group

System level default priority: SET SYSTEM DEFAULT [DEFPRIORITY | MAXPRIORITY ]

TO [CRITICAL | HIGH | NORMAL | LOW | NONE]

Set default priority per permission group:CREATE GROUP <group name> WITH DEFPRIORITY <prio>;

Change default priority of specific user: ALTER USER <user> WITH DEFPRIORITY LOW MAXPRIORITY HIGH;

Changing priority of existing session:nzsession priority -high –u <user> -pw <pw> -id <session ID>, orALTER SESSION [<session ID>] SET PRIORITY TO <prio>;

Gatekeeper

Page 10

Limits how many plans can run concurrently By priority and estimated duration host.gkEnabled=yes

Priority Queues Critical & High (host.gkHighPriQueries),

Normal(host.gkMaxPerQueue), Low (host.gkLowPriQueries)

Duration queues Split Normal by estimated duration host.gkMaxPerQueue=20,5,3,1 host.gkQueueThreshold=1,10,60,-1

Passes jobs to GRA

GRA & Resource Sharing Groups

Page 11

Resource Sharing Groups (RSGs) Different from user groups for permissions A group with a resource minimum: CREATE GROUP Analysts

WITH RESOURCE MINIMUM 50; User in only one RSG

By default: public Optionally: Job Limit

GRA(Guaranteed Resource Allocation) Accuracy: +/- 5% resource use

CPU, Disk, Network; Host & SPU Averaged over trailing window of one hour

Control Mechanisms Admission: Job order by groups’ compliance with goals Execution: feedback loop that modifies weights & applies “greed” waits

Sum of all currently active groups 100%≙

Short Query Bias – SQB Reserving resources for short queries

Part of memory is reserved for short queries only Short queries in special queue per group that is always served first host.schedSQBEnabled=true host.schedSQBNominalSecs=2

Cache retention priority for transient data (nzlocal)

Reserved resourceshost.schedSQBReservedGraSlots, host.schedSQBReservedSnSlotshost.schedSQBReservedSnMb, host.schedSQBReservedHostMb

30 min

GRA Ceilings Data Service Providers need to control user experience

Give the user only the performance that he paid for Don’t “spoil” users

GRA can hard limit a group’s resources share ALTER GROUP ... WITH RESOURCE MAXIMUM 30; MAX can be larger than MIN (allow limited headroom)

Controlled by inserting delay Delay at end of each snippet Until it would have ended

13

30%

Res

ourc

es

Time

A

delay

A A A

B B B B

A A A A ...

...

Res

ourc

es

Time

B ...delayB delayB delayB

GRA Load

Page

14

Load: Insert from an external tableINSERT INTO JUNK SELECT * FROM EXTERNAL '<stdin>‘ ... Host only snippet – no SPU snippet!

Data sent to common data receive thread per slice

GRA’s execution control using weights has no bite on loads Can’t balance load requirements, queries get clobbered

GRA load as an additional mechanism on top of GRA host.schedGRALoadEnabled=true Controls load rates based on a load performance model

“How fast could the load go without any concurrent stuff” Limits data send rate according to GRA group goal Tracks system utilization and actual rates

WLM Mechanisms Review Priority Gatekeeper GRA SQB GRA Ceilings GRA Load

30%

60%25%

15%

Usage Scenarios Application Consolidation Mixed workload: ELT v reports v interactive Departmental Chargeback Data Analysis Service Provider System rollout / Application migration Power Users

Usage Scenarios: Application Consolidation Combine applications from separate systems

Need to maintain SLAs, provide fair share Use RESOURCE MINIMUM per application

If one group has no jobs, others will expand

Workload App A App B App CSetup 50% 30% 20%

No App A - 60% 40%

No A, B - - 100%

Usage Scenarios: Mixed Workload Uncontrolled ELT may affect queries Big queries (reports / analytics) may delay little ones Interactive are highly variable and sensitive

* Limit loads only when you want other groups to fully expand

Workload MINIMUM MAXIMUM JOB LIMITELT 10-30% 10% * 4-10 or OFF

Reports 20-40% 4-10 or OFF

Prompts 40-70% 100% OFF

Usage Scenarios: Department control System used by independent departments

Or applications Want to control them; want some balance

But OK to use more if nobody else needs it Create a RESOURCE GROUP for each Set RESOURCE MINIMUM as expected

Monitor / change share over time based on _V_SCHED_GRA May even have chargebacks

System charged to departments / cost centers Track via _V_SCHED_GRA

Usage Scenarios: Service Provider Data Analysis Service Provider

Paying customers – need to limit Fixed Departmental Chargeback

System charged to departments / cost centers Not variable: FIXED

They paid for 10%, refused to pay more; They only get 10%!

RESOURCE MAXIMUM Limits use of system; does not expand

Usage Scenarios New system rollout

Consistent experience as applications arrive Set RESOURCE MAXIMUM for early users

Increase over time; eventually remove

Power Users Individuals that write raw SQL Killer / crazy queries; and lots of them! Use JOB LIMIT, GK queue limits, runaway query event

Work Load Analysis Capacity planning & application fit Query History - Shared, remotable DB (NPS)

Query text, start / end, queue time … Virtual table: _v_sched_gra

Per group: jobs started / completed, resource details, compliance, busy%, …

Virtual table: _v_system_util Host & SPU resources: CPU, disk (table & temp), network

Nzportal

WLM Guidelines & Best Practices No more than 8 -10 Resource Sharing Groups

Want each group to be able to run concurrently Roughly N max size snippets at once

Approximately 11

+/- 5% means that a 5% group could get NO TIME and be compliant

Smaller groups are harder to keep balanced

WLM Guidelines & Best Practices RESOURCE MINIMUMs should add up to 100%

Not strictly necessary Easier to think about

OK to change RSG minimums on the fly e.g. to have different day/night balances

ADMIN Gets at least 50% of system resources Avoid using the Admin user account for normal work Gets a lot of boosts, can ruin balance for other groups Like "root" - use in emergencies, occasional privileged access

WLM Guidelines & Best Practices Short Query Bias (SQB)

Many boosts for "short" queries Go to the head of the queue

Can use reserved memory etc.

More CPU, preferential disk access

Default: estimate less than 2 seconds May not be right for you

Make sure short queries are short! Check plan files

WLM Guidelines & Best Practices PRIORITY

Control queries within a group E.g. interactive queries v reports v loads

Users, groups have defaults

Can set in a session & for running queries

Two impacts: Select higher first; go to the head of the queue

Increase resource share for a query -- within the group Normal gets 2X Low, High gets 2X normal, …

WLM Guidelines & Best Practices RESOURCE MAXIMUM

Limit a RSG To protect other RSGs

Other cases: pay for use, control growth experience

Generally 5% accuracy: average over an hour Uses delay latency variation

Values should be between 30% and 80%. Larger values are sort of meaningless, not very effective

Smaller values introduce a lot of variability

WLM Guidelines & Best Practices Limiting Jobs: two ways

RSG JOB LIMIT works for a specific RSG Example: limit the ETL group to 10 loads

ALTER GROUP … WITH JOB LIMIT 10

Gatekeeper queues: limit jobs across RSGs Example: limit large jobs across the entire system

Set query priority to LOW (user or session)

Limit the GK LOW queue size

Example: limit long jobs across the system Split GK normal queue at (say) 300 seconds

<300 seconds 48, >300 seconds 5

WLM Guidelines & Best Practices JOB LIMIT

Limits one RSG; protects others Consider job type & peak throughput

A few medium queries can hit peak

Maybe ten or so loads

Small queries? May need dozens

Limits shorts, longs, all priorities JOB LIMIT best for groups with big queries, loads

WLM Guidelines & Best Practices

Experiment: Limit changes, record, verify. Your workload is not the same as others Your workload today is not the same as yesterday’s Effects may depend on subtle workload differences Effects can be hard to predict

Thank You Your feedback is important!

• Access the Conference Agenda Builder to complete your session surveys

o Any web or mobile browser at http://iod13surveys.com/surveys.html

o Any Agenda Builder kiosk onsite