event processing and integration with ias data processors
TRANSCRIPT
InvenireAude.com Copyright (C) 2015 Invenire Aude Ltd.
Practical Guide to
Data Processors
Data Processors
Data processors can be configured to act as:
Data transformation nodes, using PASCAL-like script language, Gateways and bridges (e.g. HTTP/JSON and Queues/XML), SQL Database interfaces using the data mapping script extension.
You can configure and use the Data Processors as single threaded programs but you can define many logic implementations and run them in parallel as threads.
You can choose the transaction support from the three available modes: auto-commit, single phase (independent) commits, distributed two phase commit with XA when the supported coordination software is used.
And last but not least, one can find the Data Processors as a very helpful command line admin's tool.
Architecture
Concept
The Data Processor is designed to process the incoming messages calling desired action accordingly.
1. Begin remote transactions (optional),2. Receive message,3. Parse content (if configured) and call prerequisit actions (if defined).4. Invoke the configured logic.5. Commit or rollback transaction (in case of an exception).
The program can interact with databases and send messages according to the requirements.
First Usage
export IAS_LANG_XSD=${your_project}/data/customer.xsd
File operations:
ias_qs_processor i file:customer.xml o file:stdout l fwd
ias_qs_processor i file:cust.xml o file:stdout_fmt l fwd
ias_qs_processor i file:c1.xml o file:somedir/c2.xml l fwd
l fwd vs l fwd! Turns the parsing off.
ias_qs_processor i file:customer.xml \ o file:stdout_fmt?format=JSON l fwd
Directories
ias_qs_processor i dir:data/xml o file:output l fwd
The above command will display subsequent messages on the terminal output. In order to direct the messages into series of files use the dir protocol in output specification.
ias_qs_processor i dir:data/xml \o 'dir:output/${ID}.${COUNT}.out' l fwd
The parameter is a directory name (output in this case) and the file name pattern.
Timer
The next command will produce two messages with a one second (1000ms) delay.
ias_qs_processor N 2 i timer:clock?timeout=1000 \ o file:stdout l fwd
Queue Operations
ias_qs_processor i shm://sample/Q.TEST \o file:stdout l fwd
By default, the messages (and files) are read in the shared browser mode.
ias_qs_processor i shm://sample/Q.TEST?mode=input \o file:stdout l fwd
Note that if the transport layer does not support browsing (e.g. HTTP/FCGI) omitting the modifier yields in an error.
By default, if there are no messages available, the reading ends immediately. You can specify the timeout parameter (a number of milliseconds or -1 for an infinite wait).
ias_qs_processor \ i shm://sample/Q.TEST?mode=input?timeout=10000 \ o file:stdout l fwd
Network Operations 1
When the receiver is going to be a server, one can start it with the command:
ias_qs_processor i nets://localhost:10000/?mode=input o file:stdout_fmt l fwd
and invoke a client with the matching address specified on its output:
ias_qs_processor i file:data/xml/customer.xml o netc://127.0.0.1:10000/ l fwd
Network Operations 2
The data flow in the other direction is also possible.
Server:
ias_qs_processor o nets://localhost:10000/?mode=input i file:data/xml/customer.xml l fwd
Client:
ias_qs_processor i netc://127.0.0.1:10000/?mode=input o file:stdout_fmt l fwd
Modes
Typical operation patterns are:
● Producer,● Consumer,● Requester,● Responder,● Publisher,● Subscriber,● Browser,● Shared Worker Browser (default!)
Producer and Consumer
Requester and Responder
Publish-Subscribe
The application calls the send() and receive() procedures and is not aware of such a configuration.
Not all transports support this pattern.
Implementing a Service
In this case API allows to perform send() and receive() calls on both i/o descriptors.
One can specify the same or different names for the requester input and output.
Browser
autoRecycle specifies if the processing should resume after end of data condition occurs.
● never Browser is not reset when no message is available.
● next Browser will be reset where no message is available but the exception is thrown this time. The next call will grab a new message.
● immediate Browser will be reset where no message is available and the program attempts to read a message immediately.
Shared Browser
This time, the dedicated thread reads messages on behalf of all logic threads. This pattern allows to take full advantage of parallel processing.
Built-in Actions
Typical operation patterns are:
● Forward.● Execute.● Input data type match execute.● Attribute based selection execute.● Job Daemon.● Synchronous proxy.● Asynchronous proxy.
● Message preview.● Message extraction.● Queue System statistics producer.● Writing custom logic.● Java Script execute.● Simplistic Process Server.
Considerations
Performance Notes
Options:
T <no. of threads>
C <commit count>
N <no. of messages to process> O <no. of messages to skip>
Parsing
export IAS_LANG_XSD=${your_project}/data/customer.xsd
Note the exclamation mark right after the logic name:
l fwd vs l fwd! Turns the parsing off.
Command Line Hacks
Data Format Change
ias_qs_processor i mqm://MQTEST/Q1?mode=input \ o mqm://MQTEST/Q2?format=JSON l fwd
Working with Meta Attributes
ias_qs_processor i mqm://MQTEST/Q1?mode=input \ o 'mqm://MQTEST/Q2?*Folder.ATTR1=VALUE' l fwd
The '*' attributes provided in URL are considered as message meta attributes and will be seen in RFH2 or message properties according to the application or IBM WebSphere MQ version.
Protocol Gateway
Example Queue Setup
DEF QL(GATEWAY.IN)DEF QL(SERVICE.IN)DEF QA(GATEWAY.OUT) TARGQ(SERVICE.IN)
Echo Service
If you do not have a service for this example just start the following command to setup the echo program.
ias_qs_processor \ i 'mqm://MQTEST/SERVICE.IN?mode=responder&timeout=1' \ o dummy: l fwd!
Gateway Synchronous Proxy
ias_qs_processor \ i 'srvhttp://localhost:50000/mode=input&responderName=input' \ o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=20000' \ l proxy! m server
Note the srvhttp service and the proxy logic.
Gateway Asynchronous Proxy
ias_qs_processor \ i 'asrvhttp://localhost:50000/mode=input&responderName=input' \ o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=1' \ l aproxy!
Note the asrvhttp service and the aproxy logic.
Testing
curl X POST d "{ 'key': 'value1234' }" \ H "ContentType: application/json" \ http://localhost:50000/
You can use the HTTP client of your choice.
Modes
processor Each processor thread opens a session and processes incoming messages until the end conditions are met. The threads are not restarted and program terminates when all threads terminate. This is a default mode.
server The session processing is as in the processor mode. However when the thread encounters the end of data condition, the session is recreated. This mode is useful for the synchronous network servers and when one wants to automatically release resources after some period of inactivity.
Transaction Control
NONTXN No transaction call are executed.
TXN After each iteration the commit (or rollback) API call will be executed on success (error).
XA No transaction control calls are executed and it is assumed that proper XA resource manger is configured.
Configuration Files
ias_qs_processor f <configuration_file>
Configuration (XML or JSON)
- input and output specification (Queues, HTTP(S), Files)
- data sources (data bases)
- logic(s) configuration
You can split the configuration into separate definitions of inputs, outputs or/and datasources.
ias_qs_processor i inputs.xml o outputs.xml d ds.xml
Environment Specific Settings
Writing Data Processor Configuration
Follow the XSD files from the IAS-QsystemLib project. You can write in XML or JSON.
File name Contents
qsystem.service.xsd QSystem service data schema.qsystem.workers.xsd QSystem main processor schema.qsystem.workers.logic.xsd QSystem processor logic schema.qsystem.workers.ds.xsd QSystem data sources (SQL) schema.qsystem.workers.io.xsd QSystem input/output (messaging) schema.qsystem.workers.txm.xsd QSystem XA/TX schema.qsystem.workers.spec.xsd QSystem combined specification schema.qsystem.workers.cache.xsd QSystem cache module schema.
Input Configuration
<inputSpec> <inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output"> <connection alias="qs.esb"/> <destination>CIF.INQ.IN</destination> <txnMode>NONTXN</txnMode> <timeout>-1</timeout> </inputs> </inputSpec>
The above configuration creates a server.
The responderName field creates corresponding output to send replies.
The connection details can be specified here or are referenced via an alias name from the registry file (as shown above).
Output Configuration
The above configuration creates a client.
The inputDestination field specified where the underlying protocol expects the replies .
<outputSpec> <outputs xsi:type="io:RequesterOutput" outputName="output"> <connection alias="qs.esb"/> <destination>WWW.INQ.OUT</destination> <timeout>2000</timeout> <attrUpdates name="FMT" value="XML" override="true"/> <inputDestination>WWW.INQ.IN</inputDestination> </outputs> </outputSpec>
Exit Configuration
The above configuration specifies that the cb::ValidateCustomer program is to be called.The exit program parameters are matched against the input data type.
Available error actions are: IGNORE_ALL, IGNORE_PROGRAM, IGNORE_NONE.
<inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output" > <connection> <protocol>mqm</protocol> <host>MQTSTBRK</host> </connection> <destination>DB.INQ.IN</destination> <txnMode>TXN</txnMode> <timeout>-1</timeout> <attrUpdates name="DB" value="@timestamp@" override="true"/>
<onSend> <program>cb::ValidateCustomer</program> <errorAction>IGNORE_ALL</errorAction> </onSend>
</inputs>
Data Source
<datasourceSpec> <datasources name="ds.mcif"> <connection> <protocol>db2</protocol> <user>user</user> <password>password</password> <location>CIFDB</location> </connection> <txnMode>TXN</txnMode> </datasources> </datasourceSpec>
This example provides all configuration without referencing the registry.
Logic – Explicit Execute
In this scenario the processor will start two threads.Note how “input” and “output”, “error” i/o destinations are specified.
<logicSpec> <logics xsi:type="logic:ExplicitExecute" error="brk.error" instances="2"> <load>esb::broker::inq::routeInput</load> </logics> <logics xsi:type="logic:ExplicitExecute"
input="CIF" output="output" error="brk.error"> <load>esb::broker::inq::routeOutput</load> </logics> </logicSpec>
Logic – Mach Execute
<logicSpec> <logics xsi:type="logic:MatchExecute"> <load>mcif::svc::party::getParties</load> <load>mcif::svc::role::getContextRoles</load> <load>mcif::svc::role::getPartyProducts</load> <load>mcif::svc::comm::getCommunicators</load> </logics> </logicSpec>
In this scenario the processor will find programs that match the input data.
Logic – Input and Output
Each logic has at least the following defaults:- input- output- error (optional)
Above are used by the predefined logic implementations and the script execution framework.
You can specify many input and output definitions in the configuration file. Then you can either assign them as defaults forthe desired logic or use them explicitly via std::send() or std::receive()Script calls.
Basics of Scripting
Environment
Data model:IAS_LANG_XSD=/home/ias/data/xsd/model.xsd
Script Sources:IAS_LANG_SRC_DIRS=/home/ias/lang
First Script
vi /home/ias/lang/update_msgs.y
The Eclipse source highlighting plugin is available.
See the Script Language for more details (for example on writing procedures accessing databases and files and more).
Processing Data
Testing, browsing and displaying result on the screen:
ias_qs_processor i mqm://TEST/Q1 \ o file:stdout_fmt l exe update
Processing, moving and saving changed content in the output queue:
ias_qs_processor i mqm://TEST/Q1?mode=input \ o mqm://TEST/Q2 l exe update
Learn More
http://www.invenireaude.com/content/books/QSystemProc/index.html?show=chapters/ref/index.html