prligence empowering intelligence summary management by advanced queues paper # 419 arup nanda...

pr ligence Empowering Intelligence

Summary ManagementBy Advanced Queues

Paper # 419

Arup Nanda

Proligence, Inc

www.proligence.com


Real Time Datawarehouses

SourceDW

Source

Source

Summary

PullLog Push


Complex Query

• Query to find out the eligibility of a transaction based on data in other tables – complex joins

• Takes up too much time in the Query


Actual ExampleSELECT DISTINCT NVL(C.COL1, R.COL1) AS COL1,RQ.COL2, COUNT(C.COL3) AS COUNTERFROM TAB1 R, TAB2 C, TAB3 CP, TAB4 RQ WHERE R.COL4 = C.COL4 AND C.COL1 = RQ.COL1 AND ( CP.COL5 = :b0OR (CP.COL5 = :b1 AND CP.COL6 = :b2)) AND CP.COL3 = C.COL3 AND NOT EXISTS (SELECT 1 FROM TAB5 CCL,TAB6 CL, TAB7 CHO WHERE CCL.COL7 = CL.COL7AND CL.COL8 = CHO.COL8AND CCL.COL3 = C.COL3AND CHO.COL1 = 'Y') GROUP BY NVL(C.COL1, R.COL1), RQ.COL2 ORDER BY RQ.COL2


Options

• Materialized View– Fast Refresh Needed– Complex – Fast Refresh Not Allowed

• Materialized View of MVs– Too Many Levels – Difficult to Administer– Refresh Lags– Failure Prompts Calls to DBMS_MVIEW()


Options

• A Summary Table to Hold the Data• Triggers Set Up on Source Tables

T1

T2

T3

T4

SUMMARYTABLE

Trigger

Trigger

Trigger

Trigger


Example

• Tables– DEPT– EMP

• QuerySELECT DEPTNO, COUNT(*) EMP_COUNTSFROM EMPWHERE STATUS = 'ACTIVE'GROUP BY DEPTNO


Example Summary Table

• Table DEPT_COUNTS– DEPTNO– EMP_COUNTS

DEPTNO EMP_COUNTS1 1202 3403 230


LogicA Row Is Inserted

STATUS = ‘ACTIVE’?

Check the Existence of the DEPTNO in DEPT_COUNTS

Exists?

Upadate DEPT_COUNTS

Set EMP_COUNTS=EMP_COUNTS+1

Insert into DEPT_COUNTS

EMP_COUNTS=1DEPTNO=..

Y

NY

SELECT DEPTNO, COUNT(*) EMP_COUNTSFROM EMPWHERE STATUS = 'ACTIVE'GROUP BY DEPTNO


Flaw

Time 0

DEPTNO EMP_COUNTS1 1202 340


Flaw contd…

Time 1Session1 INSERTS into EMP (DEPTNO=3, EMPNO=5679)

DEPTNO EMP_COUNTS1 1202 340


Flaw contd…

Time 2Trigger Finds No Record for DEPTNO=3INSERTS into DEPT_COUNTS

DEPTNO EMP_COUNTS1 1202 3403 1 New


Flaw contd…

Time 3Session2 Inserts into EMP(DEPTNO=3, EMPNO=4567)

DEPTNO EMP_COUNTS1 1202 3403 1 Not Visible


Flaw contd…

Time 4Trigger tries to INSERT into DEPT_COUNTS(DEPTNO=3, EMP_COUNTS=1)

DEPTNO EMP_COUNTS1 1202 3403 1 Primary Key Violated

WAITS!


Flaw contd…

Time 5Session1 Commits!

DEPTNO EMP_COUNTS1 1202 3403 1 Row Is Present


Flaw contd…

Time 6Session2 Errors Out!

DEPTNO EMP_COUNTS1 1202 3403 1 Row Is Present

Error!


Problem

• Locking• Primary Key Violation• Solution

Session1

Session2

Trans1

Trans2 SummaryTable

ApplyingSession


Advanced Queues

User

Producer

User

Message Payload Consumer

enqueue dequeue


Complex

Producer1

Tags

Producer2

Producer3

Consumer1

Consumer2


Types of AQs

• Persistent• Non Persistent• Single Producer/Consumer• Multiple Producers/Consumers• Publisher-Subscriber Model


Queue Table (QT)Payload defined here

QueueException QueueDBMS_AQADMDBMS_AQ

AQ Components


Solution

Table

Trigger

Queue

QueueTable

EnQueue DeQueue

SummaryTable

Payload


Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Database Prep

• Turn on the queue timer processes– aq_tm_processes = 2– ALTER SYSTEM possible too– Process Identified as QMNn

• Grant privileges– GRANT EXECUTE ON DBMS_AQADM TO SCOTT;

– GRANT EXECUTE ON DBMS_AQ TO SCOTT;


Payload

• Oracle Object Type

create or replace type dept_counts_type as object

( action_type char(1), old_deptno number(2), new_deptno number(2))

Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Processing

• Procedure PROCESS_DEPT_COUNTS()• Input Parameters

– Action – I, D, U, T– Old DeptNo– New DeptNo

Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Queue Table

begin DBMS_AQADM.CREATE_QUEUE_TABLE ( queue_table => 'DEPT_COUNTS_QT', queue_payload_type=> 'DEPT_COUNTS_TYPE', multiple_consumers=> FALSE, storage_clause => 'TABLESPACE USR INITRANS 10 STORAGE (FREELISTS 10 FREELIST GROUPS 2)',

compatible => '8.1');end;

Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Queue

begin DBMS_AQADM.CREATE_QUEUE ( queue_name =>'DEPT_COUNTS_Q', queue_table=>'DEPT_COUNTS_QT', max_retries=>'5', retry_delay=>'0');

dbms_aqadm.start_queue( 'DEPT_COUNTS_Q',TRUE,TRUE);

end;

Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Enqueue Procedure

create or replace procedure enq_dept_counts_q (p_msg in dept_counts_type)as enq_opt dbms_aq.enqueue_options_t; msg_prop dbms_aq.message_properties_t; msg_id raw(16);begin sys.dbms_aq.enqueue ( 'DEPT_COUNTS_Q',enq_opt, msg_prop, p_msg, msg_id);end;

Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Dequeue Procedure

create or replace procedure deq_dept_counts_qas deq_opt dbms_aq.dequeue_options_t; msg_prop dbms_aq.message_properties_t; payload dept_counts_type; msgid raw(16);begin loop deq_opt.wait := dbms_aq.forever; deq_opt.navigation := dbms_aq.next_message; dbms_aq.dequeue( 'DEPT_COUNTS_Q', deq_opt, msg_prop, payload, msgid); process_dept_counts(payload.action_type, payload.old_deptno, payload.new_deptno); commit; end loop;end;

Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Trigger

create or replace trigger tr_ar_iud_emp after insert or delete or update on empfor each rowdeclare l_action char(1); l_old_deptno number(2); l_new_deptno number(2);begin enq_dept_counts_q( dept_counts_type( l_action, l_old_deptno, l_new_deptno));end;

Table

Trigger Queue

QueueTable

EnQueue

DeQueue

SummaryTable

Payload


Solution Revisited

Table

Trigger

Queue

QueueTable

EnQueue DeQueue

SummaryTable

Payload


Important Considerations

• Asynchronous• DecoupledTable

Trigger

Queue

QueueTable

EnQueue DeQueue

SummaryTable

Payload

Transaction 1 Transaction 2


Administration

• Number of Messages in Queue

select count(*) from AQ$DEPT_COUNTS_QT where queue = 'DEPT_COUNTS_Q'


Usage

• Datawarehouse• OLTP Complex Queries• Data from Heterogeneous Sources

– MQ Series• Maintaining Flattened Tables in OLTP


Thank you!

www.proligence.cowww.proligence.comm

Summary ManagementBy Advanced Queues

Paper # 419

prligence empowering intelligence summary management by advanced queues paper # 419 arup nanda...

Documents

table queue

deptno slide

counts emp

queue queue

counts deptno emp

counts deptnoemp

query slide

queue dept