building python messaging apps with oracle database · 2019-06-27 · cx_oracle background • open...

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|

BuildingPythonMessagingAppswithOracleDatabaseCON7344

AnthonyTuiningaConsulGngMemberofTechnicalStaffDataAccessDevelopment,OracleDatabaseOctober3rd,2017


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirecGon.ItisintendedforinformaGonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncGonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andGmingofanyfeaturesorfuncGonalitydescribedforOracle’sproductsremainsatthesolediscreGonofOracle.

2

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.| 3

AboutMe

• AnthonyTuininga– SoQwareDeveloperforOracleDatabase,ScripGngLanguageDrivers(andrelatedtechnology)

– Creator/maintainerofcx_Oracle

• ContactinformaGon– [email protected]– @AnthonyTuininga


ProgramAgenda

IntroducGon

Code

Futurecx_OracleEnhancements

AQFeatures

1

2

3

4

4


ProgramAgenda

IntroducGon

Code


AQFeatures

1

2

3

4

5

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|

IntroducGon

ConfidenGal–OracleInternal/Restricted/HighlyRestricted 6


cx_OracleBackground• OpenSourcepackagegivingCPythonaccesstoOracleDatabase• CoversallofPythonDatabaseAPIspecificaGonwithmanyaddiGonstosupportOracleadvancedfeatures

• BegunbyAnthonyTuiningain1998usingOracle8iandPython1.5– 30+releasescoveringOracle8ithrough12c

• Currentcx_Oracleversionis6.0,releasedAugust2017•  Supportforadvancedqueueing(AQ)addedin5.3

7


WhatisAdvancedQueuing(AQ)?•  Storesmessagesinabstractstorageunitscalledqueues

– Messagesareenqueued(addedtothequeue)bytheproducer– Messagesaredequeued(removedfromthequeue)bytheconsumer

• Producersandconsumersdonotneedtobeawareofeachother• MessagesstayinthequeueunGltheyareconsumedortheyexpire• Consumerscanbeotherqueuesinthesameoraremotedatabase• Messagescanbepersistentorbuffered(in-memory)

8


AQFeaturesInApplicaGon• CondiGonscanbespecifiedbyconsumerstolimitthemessagestheysee• MulGpleconsumerqueuesallowmulGpleconsumerstoreadthesamemessageindependentlyofeachother

• Messagescanhavea“shelflife”aQerwhichtheyexpire• MessagescanspecifyanidenGfier(“correlaGon”)whichisnotmanagedbyAQinanyway

9


AddiGonalAQFeatures• MessagescanbedelayedaperiodofGmebeforetheycanbedequeued•  Theorderinwhichmessagesaredequeuedcanbespecified• Messagescanbetransformedfromonetypetoanotherbeforebeingenqueuedordequeued

• Can“peek”atmessageswithoutconsumingthem• CandefineexcepGonqueuesformanagingexcepGons• Messagescanbein-memoryonlyforincreasedperformancefortransientmessageswhichdon’trequirepersistence

10


ApplicaGonScenario

11

Customers

Stores

Manufacturers


ProgramAgenda

IntroducGon

Code


AQFeatures

1

2

3

4

12


Code



ApplicaGonArchitecture

14

ProcessQuotes

ProcessOrders

Quotes

ProcessOrders

ProcessQuotes

Quotes

SendOrders

RFQs

Orders

HeadOffice StoreS StoreT

ProcessRFQs

QuoteResponses

ManufacturerA ManufacturerB

ProcessRFQs

QuoteResponses


SendOrdersArchitecture

15

SendOrders

Orders

HeadOffice


SendOrders–Overview• Microservicesendsordersfromcustomerstothestoresusingasingleconsumerqueuecalled“OrderQueue”

• Objecttype“UDT_ORDER”usedasmessagepayload• AQproperty“correlaGon”usedtoidenGfytheordernumber

16


SendOrders–SQLCode–Part1 1 create type udt_OrderLine as object ( 2 WidgetTypeCode varchar2(30), 3 NumBoxes number(9) 4 ); 5 / 6 7 create type udt_OrderLines as varray(20) of udt_OrderLine; 8 / 9 10 create type udt_Order as object ( 11 CustomerName varchar2(60), 12 CustomerEmailAddress varchar2(250), 13 StoreCode varchar2(30), 14 Lines udt_OrderLines 15 ); 16 /

17


SendOrders–SQLCode–Part2 1 begin 2 dbms_aqadm.create_queue_table('OrderMessages', 'UDT_ORDER'); 3 dbms_aqadm.create_queue('OrderQueue', 'OrderMessages'); 4 dbms_aqadm.start_queue('OrderQueue'); 5 end; 6 /

18


SendOrders–PythonCode–Part1 1 orderObjType = connection.gettype("UDT_ORDER") 2 orderObj = orderObjType.newobject() 3 orderObj.CUSTOMERNAME = order.customerName 4 orderObj.CUSTOMEREMAILADDRESS = order.customerEmailAddress 5 orderObj.STORECODE = order.storeCode 6 7 orderLineObjType = connection.gettype("UDT_ORDERLINE") 8 orderLinesObjType = connection.gettype("UDT_ORDERLINES") 9 orderLinesObj = orderLinesObjType.newobject() 10 for line in order.lines: 11 lineObj = orderLineObjType.newobject() 12 lineObj.WIDGETTYPECODE = line.widgetTypeCode 13 lineObj.NUMBOXES = line.numBoxes 14 orderLinesObj.append(lineObj) 15 orderObj.LINES = orderLinesObj

19


SendOrders–PythonCode–Part216 options = connection.enqoptions() 17 messageProperties = connection.msgproperties() 18 messageProperties.correlation = order.orderNumber 19 connection.enq("OrderQueue", options, messageProperties, orderObj) 20 connection.commit()

20


ProcessOrdersArchitecture

21

ProcessOrders

ProcessOrders

RFQs

Orders

HeadOffice StoreS StoreT


ProcessOrders–Overview• Aseparatemicroserviceisusedbyeachstoretoprocessitsordersfromthe“OrderQueue”queue– TheAQproperty“condiGon”isusedtoensurethatonlymessagesfortheparGcularstorearedequeued

• Aseachorderisdequeued,aquoteisrequestedforeachorderline– ThestorelocaGonisincludedinthemessagesothatthemanufacturerscanusethatinformaGontomakeadecisiononwhethertorespondtotherequestforaquote

– AmulG-consumerqueue“RFQQueue”isusedsothateachmanufacturercanmanageitsmessagesindependentlyofallothermanufacturers

•  Theobjecttype“UDT_RFQ”isusedasthemessagepayload

22


ProcessOrders–SQLCode–Part1 1 create type udt_RFQ as object ( 2 Location sdo_geometry, 3 WidgetTypeCode varchar2(30), 4 NumBoxes number(9), 5 QueueName varchar2(30) 6 ); 7 /

23


ProcessOrders–SQLCode–Part2 1 begin 2 dbms_aqadm.create_queue_table('RFQMessages', 'UDT_RFQ', 3 multiple_consumers => true); 4 dbms_aqadm.create_queue('RFQQueue', 'RFQMessages'); 5 dbms_aqadm.start_queue('RFQQueue'); 6 end; 7 /

24


ProcessOrders–PythonCode–Part1 1 orderObjType = connection.gettype("UDT_ORDER") 2 orderObj = orderObjType() # or orderObjType.newobject() 3 deqOptions = connection.deqoptions() 4 deqOptions.navigation = cx_Oracle.DEQ_FIRST_MSG 5 deqOptions.wait = cx_Oracle.DEQ_NO_WAIT 6 deqOptions.condition = "tab.user_data.StoreCode = '%s'" % STORE_CODE 7 messageProperties = connection.msgproperties() 8 9 rfqObjType = connection.gettype("UDT_RFQ") 10 rfqObj = rfqObjType.newobject() 11 rfqObj.LOCATION = STORE_LOCATION 12 enqOptions = connection.enqoptions()

25


ProcessOrders–PythonCode–Part213 while connection.deq("OrderQueue", deqOptions, messageProperties, orderObj): 14 orderNumber = messageProperties.correlation 15 for lineNum, lineObj in enumerate(orderObj.LINES.aslist()): 16 rfqObj.WIDGETTYPECODE = lineObj.WIDGETTYPECODE 17 rfqObj.NUMBOXES = lineObj.NUMBOXES 18 rfqObj.QUEUENAME = "QuoteQueue" + STORE_CODE 19 messageProperties.correlation = "%s_%s" % (orderNumber, lineNum + 1) 20 connection.enq("RFQQueue", enqOptions, messageProperties, rfqObj) 21 connection.commit()

26


ProcessRFQsArchitecture

27

ProcessRFQs

ProcessRFQs

QuotesQuotesRFQs

HeadOffice StoreS StoreT ManufacturerA ManufacturerB


ManufacturerResponsetoRFQ?

28


ProcessRFQs–Overview• Aseparatemicroserviceisusedbyeachmanufacturertoprocessrequestsforquotesfromstoresandsendquotesbacktothestores

•  EachmanufacturersubscribestothemulGpleconsumerqueue“RFQQueue”usingitsownrule– SpaGalDataObjects(SDO)isusedtolimitthemessagesreceivedtoonlythosefromstoreswithinacertaindistance

– AQfiltersthemessagesforsubscribersasmessagesareaddedtothequeue

•  Theobjecttype“UDT_QUOTE”isusedasthemessagepayload•  TheAQproperty“expiraGon”isusedtospecifyhowlongthequoteisvalid

– OncetheGmehasexpiredwithoutbeingdequeued,itisputintoanexcepGonqueue

29


ProcessRFQs–SQLCode–Part1 1 create type udt_Quote as object ( 2 WidgetTypeCode varchar2(30), 3 NumBoxes number(9), 4 AvailableDate date, 5 TotalCost number(11, 2), 6 ManufacturerCode varchar2(30), 7 QueueName varchar2(30) 8 ); 9 /

30


ProcessRFQs–SQLCode–Part2 1 begin 2 for r in (select * from Manufacturers) loop 3 dbms_aqadm.add_subscriber('RFQQueue', 4 sys.aq$_agent('Manufacturer' || r.Code, null, null), 5 rule => 'sdo_within_distance(tab.user_data.location, ' || 6 '( select location ' || 7 ' from Manufacturers ' || 8 ' where Code = ' || r.Code || 9 '), ''distance=' || r.MaxDistanceToStore || 10 ' unit=km'') = ''TRUE'''); 11 end loop; 12 dbms_aqadm.create_queue_table('QuoteMessages', 'UDT_QUOTE'); 13 for r in (select * from Stores) loop 14 dbms_aqadm.create_queue('QuoteQueue' || r.Code, 'QuoteMessages'); 15 dbms_aqadm.start_queue('QuoteQueue' || r.Code); 16 end loop; 17 end; 18 /

31


ProcessRFQs–PythonCode–Part1 1 rfqObjType = connection.gettype("UDT_RFQ") 2 rfqObj = rfqObjType.newobject() 3 deqOptions = connection.deqoptions() 4 deqOptions.navigation = cx_Oracle.DEQ_FIRST_MSG 5 deqOptions.wait = cx_Oracle.DEQ_NO_WAIT 6 deqOptions.consumername = "Manufacturer" + MANUFACTURER_CODE 7 messageProperties = connection.msgproperties() 8 9 quoteObjType = connection.gettype("UDT_QUOTE") 10 quoteObj = quoteObjType.newobject() 11 enqOptions = connection.enqoptions()

32


ProcessRFQs–PythonCode–Part213 while connection.deq("RFQQueue", deqOptions, messageProperties, rfqObj): 14 quoteObj.WIDGETTYPECODE = rfqObj.WIDGETTYPECODE 15 quoteObj.NUMBOXES = rfqObj.NUMBOXES 16 quoteObj.AVAILABLEDATE = datetime.datetime.today() + \ 17 datetime.timedelta(DAYS[rfqObj.WIDGETTYPECODE]) 18 quoteObj.TOTALCOST = rfqObj.NUMBOXES * PRICES[rfqObj.WIDGETTYPECODE] 19 quoteObj.QUEUENAME = "QuoteResponseQueue" + MANUFACTURER_CODE 20 quoteObj.MANUFACTURERCODE = MANUFACTURER_CODE 21 messageProperties.expiration = 24 * 60 * 60 22 connection.enq(rfqObj.QUEUENAME, enqOptions, messageProperties, quoteObj) 23 connection.commit()

33


ProcessQuotesArchitecture

34

ProcessQuotes

ProcessQuotes

QuoteResponses

QuoteResponsesQuotesQuotes

HeadOffice StoreS StoreT ManufacturerA ManufacturerB


ProcessQuotes–Overview• Aseparatemicroserviceisusedbyeachstoretoprocessquotesfrommanufacturersandsendaresponsebacktothem

•  Theobjecttype“UDT_QUOTERESPONSE”isusedasthemessagepayload

35


ProcessQuotes–SQLCode–Part1 1 create type udt_QuoteResponse as object ( 2 Accepted char(1), 3 Comments varchar2(4000) 4 ); 5 /

36


ProcessQuotes–SQLCode–Part2 1 begin 2 dbms_aqadm.create_queue_table('QuoteResponseMessages', 3 'UDT_QUOTERESPONSE'); 4 for r in (select * from Manufacturers) loop 5 dbms_aqadm.create_queue('QuoteResponseQueue' || r.Code, 6 'QuoteResponseMessages'); 7 dbms_aqadm.start_queue('QuoteResponseQueue' || r.Code); 8 end loop; 9 end; 10 /

37


ProcessQuotes–PythonCode–Part1 1 quoteObjType = connection.gettype("UDT_QUOTE") 2 quoteObj = quoteObjType.newobject() 3 deqOptions = connection.deqoptions() 4 deqOptions.navigation = cx_Oracle.DEQ_FIRST_MSG 5 deqOptions.wait = cx_Oracle.DEQ_NO_WAIT 6 messageProperties = connection.msgproperties() 7 8 quoteResponseObjType = connection.gettype("UDT_QUOTERESPONSE") 9 quoteResponseObj = quoteResponseObjType.newobject() 10 enqOptions = connection.enqoptions()

38


ProcessQuotes–PythonCode–Part211 queueName = "QuoteQueue" + STORE_CODE 12 while connection.deq(queueName, deqOptions, messageProperties, quoteObj): 13 responseComments = "Some comments if not accepted and None if accepted" 14 quoteResponseObj.ACCEPTED = "Y" if responseComments is None else "N" 15 quoteResponseObj.COMMENTS = responseComments 16 connection.enq(quoteObj.QUEUENAME, enqOptions, messageProperties, 17 quoteResponseObj) 18 connection.commit()

39


Furthersteps• Manufacturersneedtoprocessquoteresponsesandensurethattheyareabletoshipbythedateoriginallypromised– Messagesaredequeuedfrommanufacturerspecific“QuoteResponseQueue”

•  Storesneedtosendane-mailtocustomerswhenallorderlineshavebeensaGsfiedbysuccessfulquotesfrommanufacturers– Failuretosendane-mailshouldnotabortthetransacGon– FailuretocommitthetransacGoncannot“unsend”e-mail– Aqueueisneededtoensurethate-mailsaresentindependently,butaresGllguaranteedtobesent

40


ProgramAgenda

IntroducGon

Code


AQFeatures

1

2

3

4

41


Futurecx_OracleEnhancements?•  Supportfor“raw”queues•  SupportfornoGficaGonofavailabilityofmessagestodequeue

43


ProgramAgenda

IntroducGon

Code


AQFeatures

1

2

3

44

4


AQFeatures



AboutMe

• AnilMadan– SoQwareDevelopmentDirector,OracleAdvancedQueuing

• ContactinformaGon– [email protected]


AdvancedQueuingOverview

Transformation & Rules Engine

Publish Advanced Queues

Subscribe Priority > 2

Subscribe corrid= ‘ORDER_ID’

ORACLE

Advanced Queues

Application 3

PYTHONPL/SQLJMSJDBC

Queue Tables

ORACLE

Application 1

Application 2

SOAPWebServices/XML

IBM MQ

OCI

TIBCO


AQUsage• AQisacriGcalpartoftheFusionApplicaGonsSaaSandMiddlewarePaaS•  Severalreasonsforthis

– Queuesandasynchronousmessagingisanestablishedprogrammingmodel– TradiGonallytheseweredoneviaJavaMessagingService(JMS)– Withthedrivetostatelessmid-Gers(pupngstateinthedatabaseinstead),theyhavemigratedfrommid-GerJMSprovidertousingAQ•  ThissimplifiesHA,DR,backup/recovery

– MajorMiddlewarePaaSservices(e.g.,IdenGtyCloudService)arebeingre-architectedtousemicroservices

– Thegoalistoscalesubsystemsindependently,greaterresiliency,andagiledevelopment


AQUsage•  SampleofOracleSaaS,MiddlewareandDatabaseusingAQ

–  FusionApplicaGons(SupplyChainManagement,DistributedOrderOrchestraGon,EnterpriseSchedulerService,…)

–  OracleRetail–  OracleDaaS(DataasaService)–  SOASuite(EventDeliveryNetwork)–  Eloqua–  E-BusinessSuitehasover100queuesintheschema–  EnterpriseManager–  Databaseinternalusers

•  DatabaseChangeNoGficaGons,DatabaseAlerts,DatabaseScheduler,MessagingGateway,Datapump,EMalertevents,FANnoGficaGonandmanymore


ShardedQueues

•  Singlelogicalqueuemodel

ArchitectureforScalabilityandPerformance

§ Physical Sharded Queue: automatic management of partitions on RAC


• Higherthroughput•  LesssystemresourceconsumpGon•  Largenumberofsubscribers•  Event-basedlistenerwithfewerdatabaseconnecGons• ManyconcurrentenqueuersanddequeuersacrossmulGpleRACinstances• BackwardcompaGbleforStandardJMSandPL/SQLbasedapplicaGons

– justrecreatetheAQinthedatabase

ShardedQueuesKeybenefits


InteresGngSessionsforyourOpenWorldExperience• Hands-onLabSession:PythonandOracleDatabase12c:ScripGngfortheFuture[HOL7605]– Wednesday,Oct04,9:45a.m.-10:45a.m.|HiltonSanFranciscoUnionSquare(BallroomLevel)-ConGnentalBallroom6

• PythonandOracleDatabase:Tips,Tricks,andtheBestNewFeatures[CON6714]– Wednesday,Oct04,4:30p.m.-5:15p.m.|MosconeWest-Room3012

• DemoBooth:ApplicaGonDevelopmentwithNode.js,Python,PHP,R,C,andC++– SOA-042,MosconeWest

52


InteresGngSessionsforyourOpenWorldExperience• Node.js:JavaScriptApplicaGonDevelopmentforOracleDatabase[CON6712]– Tuesday,Oct03,5:45p.m.-6:30p.m.|MosconeWest-Room3008

• BestPracGcesforApplicaGonHighAvailability[CON6711]– Wednesday,Oct04,3:30p.m.-4:15p.m.|MosconeWest-Room3012

• PerformanceandScalabilityTechniquesforOracleDatabaseApplicaGons[CON6710]– Wednesday,Oct04,5:30p.m.-6:15p.m.|MosconeWest-Room3012

53


Resources• Mail: [email protected]

•  Twiwer: @AnthonyTuininga

• GroupBlog: hAps://blogs.oracle.com/opal

•  cx_OracleHome: hAps://oracle.github.io/python-cx_Oracle•  Installfrom: hAps://pypi.python.org/pypi/cx_Oracle• Coderepository: hAps://github.com/oracle/python-cx_Oracle

54


SafeHarborStatementTheprecedingisintendedtooutlineourgeneralproductdirecGon.ItisintendedforinformaGonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncGonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andGmingofanyfeaturesorfuncGonalitydescribedforOracle’sproductsremainsatthesolediscreGonofOracle.

55

building python messaging apps with oracle database · 2019-06-27 · cx_oracle background • open...

Documents