enable database service over http or ibm websphere mq in 15_minutes with ias
TRANSCRIPT
InvenireAude.com Copyright (C) 2015 Invenire Aude Ltd.
Practical Guide to
Building Database Services with Invenire Aude System
Enabling Database Services in less than 15 minutes ....
Task
Enable the database content as: - HTTP service - IBM WebSphere MQ Service- files/directories (testing)
Database
Data Processor
http://localhost:5000RESTful JSON/XML Service
Data Processor
IBM Websphere MQ Queuebased Service
Example Project
The examples presented here are available on the internet.
However, as you can see, the whole configuration can be made in less then 15 minutes from scratch.
As the ThreeSteps Eclipse Project.
Create Database
ds/crt_db.sh
sqlite3 /var/tmp/customer.ias.db <<EOF
CREATE TABLE CUSTOMER( ID INTEGER PRIMARY KEY, FIRSTNAME VARCHAR(64), LASTNAME VARCHAR(64));
INSERT INTO CUSTOMER VALUES(1,'Walt','Kowalski');INSERT INTO CUSTOMER VALUES(2,'Nick','Pulovski');INSERT INTO CUSTOMER VALUES(3,'Frank','Corvin');
SELECT * FROM CUSTOMER;
EOF
sqlite3 /var/tmp/customer.ias.db <<EOF
CREATE TABLE CUSTOMER( ID INTEGER PRIMARY KEY, FIRSTNAME VARCHAR(64), LASTNAME VARCHAR(64));
INSERT INTO CUSTOMER VALUES(1,'Walt','Kowalski');INSERT INTO CUSTOMER VALUES(2,'Nick','Pulovski');INSERT INTO CUSTOMER VALUES(3,'Frank','Corvin');
SELECT * FROM CUSTOMER;
EOF... or use the IBM DB2
or Oracle database instead.
Create Data Model
xsd/customer.xsd
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:demo="http://www.invenireaude.org/demo" xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="http://www.invenireaude.org/demo">
<xsd:complexType name="Selector"> <xsd:sequence> <xsd:element name="ids" type="xsd:integer" minOccurs="0" maxOccurs="unbounded"/> </xsd:sequence></xsd:complexType>
<xsd:complexType name="Customer"> <xsd:sequence> <xsd:element name="id" type="xsd:string" minOccurs="1" maxOccurs="1"/> <xsd:element name="firstname" type="xsd:string" minOccurs="1" maxOccurs="1"/> <xsd:element name="lastname" type="xsd:string" minOccurs="1" maxOccurs="1"/> </xsd:sequence> </xsd:complexType> <xsd:complexType name="GetCustomers"> <xsd:sequence> <xsd:element name="selector" type="demo:Selector" minOccurs="1" maxOccurs="1"/> <xsd:element name="customers" type="demo:Customer" minOccurs="0" maxOccurs="unbounded"/> </xsd:sequence> </xsd:complexType> <xsd:element name="getCustomers" type="demo:GetCustomers"/> </xsd:schema>
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:demo="http://www.invenireaude.org/demo" xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="http://www.invenireaude.org/demo">
<xsd:complexType name="Selector"> <xsd:sequence> <xsd:element name="ids" type="xsd:integer" minOccurs="0" maxOccurs="unbounded"/> </xsd:sequence></xsd:complexType>
<xsd:complexType name="Customer"> <xsd:sequence> <xsd:element name="id" type="xsd:string" minOccurs="1" maxOccurs="1"/> <xsd:element name="firstname" type="xsd:string" minOccurs="1" maxOccurs="1"/> <xsd:element name="lastname" type="xsd:string" minOccurs="1" maxOccurs="1"/> </xsd:sequence> </xsd:complexType> <xsd:complexType name="GetCustomers"> <xsd:sequence> <xsd:element name="selector" type="demo:Selector" minOccurs="1" maxOccurs="1"/> <xsd:element name="customers" type="demo:Customer" minOccurs="0" maxOccurs="unbounded"/> </xsd:sequence> </xsd:complexType> <xsd:element name="getCustomers" type="demo:GetCustomers"/> </xsd:schema>
We define one service:The customer search.
Setup Environment
. ./setenv.sh
Script Sources:IAS_LANG_SRC_DIRS=/home/ias/ZZZThreeSteps/langexport IAS_LANG_SRC_DIRS
Data Model:IAS_LANG_XSD=/home/ias/ZZZThreeSteps/xsd/customer.xsdexport IAS_LANG_XSD
Service Manager Configuration:IAS_SM_CFGDIR=/home/ias/ZZZThreeSteps/smexport IAS_SM_CFGDIR
Prepare First Call Input
. ./data/in/case01.json
{ "selector" : {
"ids" : [ 1 ] }, "_dmType" : “http://www.invenireaude.org/demo#GetCustomers"}
{ "selector" : {
"ids" : [ 1 ] }, "_dmType" : “http://www.invenireaude.org/demo#GetCustomers"}
Now we can check if setup is ok, let's have the processor read the file and convert it to XML first.
$ ias_qs_processor i file:data/in/case01.json o file:stdout?format=XML l fwd
<?xml version="1.0" encoding="UTF8"?><ns0:getCustomers xmlns:xsi="http://www.w3.org/2001/XMLSchemainstance" xmlns:ns0="http://www.invenireaude.org/demo"> <selector> <ids>1</ids> </selector></ns0:getCustomers>
Implement Service
. ./lang/src/getCustomers.y
IMPORT ds::fetchCustomers;
PROGRAM svc::getCustomers(VAR ctx AS Context: "http://www.invenireaude.org/qsystem/workers",
VAR msg AS GetCustomers : "http://www.invenireaude.org/demo")RETURNS GetCustomers : "http://www.invenireaude.org/demo" BEGIN msg.customers = ds::fetchCustomers(msg.selector); RETURN msg; END;
IMPORT ds::fetchCustomers;
PROGRAM svc::getCustomers(VAR ctx AS Context: "http://www.invenireaude.org/qsystem/workers",
VAR msg AS GetCustomers : "http://www.invenireaude.org/demo")RETURNS GetCustomers : "http://www.invenireaude.org/demo" BEGIN msg.customers = ds::fetchCustomers(msg.selector); RETURN msg; END;
This is the service wrapper. The top level programs always have two parameters: the context and the input data object.
There is not much logic here – only the database access procedure is called.
Implement SELECT
. ./lang/ds/fetchCustomers.y
PROGRAM ds::fetchCustomers(VAR selector AS Selector : "http://www.invenireaude.org/demo")
RETURNS ARRAY OF Customer : "http://www.invenireaude.org/demo"
EXTERNAL "libIASQSystemLib:ias_qs_lang_db_proxy:WrappedStatement"("db"," SELECT ARRAY INTO result ? firstname => firstname, ? lastname => lastname,id => id FROM CUSTOMER WHERE ? id IN selector.ids[*]");
PROGRAM ds::fetchCustomers(VAR selector AS Selector : "http://www.invenireaude.org/demo")
RETURNS ARRAY OF Customer : "http://www.invenireaude.org/demo"
EXTERNAL "libIASQSystemLib:ias_qs_lang_db_proxy:WrappedStatement"("db"," SELECT ARRAY INTO result ? firstname => firstname, ? lastname => lastname,id => id FROM CUSTOMER WHERE ? id IN selector.ids[*]");
This is an example of the external function call.In this case it's the database driver.
The “db” is an alias of the data source and the second parameter specifies the mappings.
Ready To Be Tested
ias_qs_processor i file:data/in/case02.json \ o file:stdout \
d sqlite:/var/tmp/customer.ias.db \ l match svc::getCustomers
{ "selector" : { "ids" : [ 2] }, "customers" : [ { "id" : "2", "firstname" : "Nick", "lastname" : "Pulovski" }],"_dmType" : "http://www.invenireaude.org/demo#GetCustomers"}
The -d option will definea data source named “db”.The program reads the file,
and executes our script.
Test Suite
One can easily build a test suite for those services.
Specify the whole directory as an input and compare directories data/out and data/work
ias_qs_processor i dir:data/in \ o dir:data/work/\${ID} \
d sqlite:/var/tmp/customer.ias.db \ l match svc::getCustomers
HTTP Service
Basic Synchronous Mode
ias_qs_processor \ i "srvhttp://localhost:50000/?mode=input&responderName=output" \ o dummy: \ d sqlite:/var/tmp/customer.ias.db \ l match svc::getCustomers m server
Testing with Curl (or any Web Client):
curl X POST d '{ "selector" : {}, "_dmType" : "http://www.invenireaude.org/demo#GetCustomers" }' H "ContentType: application/json" http://localhost:50000/
Your service is READY !
Asynchronous vs Synchronous Mode
Synchronous Mode: - Each call blocks one thread until the service is completed - pre-start threads with -T <num_threads> option - specify “-m server”
Asynchronous mode: - thread is allocated when whole message is read or written - not supported on every platform (epoll call) - handles 1000+ concurrent connections easily - less threads needed - the “asrvhttp” transport and no need for “-m server”
ias_qs_processor \ i "asrvhttp://localhost:50000/?mode=input&responderName=output" \ o dummy: \ d sqlite:/var/tmp/customer.ias.db \ l match svc::getCustomers
Efficient Web and Mobile Apps
Setup a simple (and efficient) asynchronousWeb server like lighttpd or nginx.
Provide some static HTML content along withJavaScript that calls your services using AJAX.
and you are ready to go ...
See the CIF and CEP tutorialson www.invenireaude.com for more details
See the CIF and CEP tutorialson www.invenireaude.com for more details
Enable Queue Based Service IBM WebSphere MQ
Just specify the queue based protocol and the data processor will handle the requests and send them according to the transport layer patterns. (Eg. ReplyToQueue, MID -> CID etc).
Assume the SERVCE.IN queue on the MQTEST queue manager.
ias_qs_processor \ i "mqm://MQTEST/SERVICE.IN?mode=input&timeout=1&responderName=output" \ o dummy: \ d sqlite:/var/tmp/customer.ias.db \ l match svc::getCustomers
Adjust options: T <no. of threads> C <commit count>
Protocol Gateway
DEF QL(GATEWAY.IN)DEF QL(SERVICE.IN)DEF QA(GATEWAY.OUT) TARGQ(SERVICE.IN)
Gateway Synchronous Proxy
ias_qs_processor \ i 'srvhttp://localhost:50000/mode=input&responderName=input' \ o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=20000' \ l proxy! m server
Note the srvhttp service and the proxy logic.
Gateway Asynchronous Proxy
ias_qs_processor \ i 'asrvhttp://localhost:50000/mode=input&responderName=input' \ o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=1' \ l aproxy!
Note the asrvhttp service and the aproxy logic.
Building the REAL system ....
Real Live Scenarios
Considerations:
● Advanced logic: - many inputs and outputs, data sources
● More services (let's say 500+)
● Configuration Options:- Database setups (IBM DB2, Oracle)- Transactions Control (XA)- Switching Transport Layers and/or Environments
● Fault Tolerant Deployment- service management
Advanced Rules
Script is a powerful and efficient language:
- strict typing, data model types exposed as defined by program,- procedures and functions, exceptions, - add your own extensions (e.g. scoring systems).
Database Extension provides mappings for:
SELECT ONCE,SELECT ARRAY,INSERT,UPDATE,DELETE,CALL,
and can handle various cases, see example on the next slide
Advanced Rules Example
PROGRAM mcif::ds::party::getParties(VAR selector AS PartiesSelector : "http://www.invenireaude.com/minicif/party/api" )RETURNS ARRAY OF Party : "http://www.invenireaude.com/minicif/party"EXTERNAL "libIASQSystemLib:ias_qs_lang_db_proxy:WrappedStatement"("ds.mcif"," SELECT ARRAY INTO result MAP type (
P => Person : 'http://www.invenireaude.com/minicif/party'( ? firstname => firstname, ? middlename => middlename, ? lastname => lastname, ? birthDate => birthDate, ? sex => sex
), O => Organization : 'http://www.invenireaude.com/minicif/party' ( ? shortname => shortname, ? fullname => fullname, ? established => established
) )
pid => pid FROM CIF_VW_PARTY WHERE ? pid IN selector.pids[*]");
PROGRAM mcif::ds::party::getParties(VAR selector AS PartiesSelector : "http://www.invenireaude.com/minicif/party/api" )RETURNS ARRAY OF Party : "http://www.invenireaude.com/minicif/party"EXTERNAL "libIASQSystemLib:ias_qs_lang_db_proxy:WrappedStatement"("ds.mcif"," SELECT ARRAY INTO result MAP type (
P => Person : 'http://www.invenireaude.com/minicif/party'( ? firstname => firstname, ? middlename => middlename, ? lastname => lastname, ? birthDate => birthDate, ? sex => sex
), O => Organization : 'http://www.invenireaude.com/minicif/party' ( ? shortname => shortname, ? fullname => fullname, ? established => established
) )
pid => pid FROM CIF_VW_PARTY WHERE ? pid IN selector.pids[*]");
Value of the 'type'column will indicate the created data object type.
Rule Number One
Use Configuration Files
Use configuration files instead of the command line options. Data Processor Configuration File:
- XML or JSON (same model)- Full specification of Inputs and Outputs:
- protocol and connection details,- operation pattern (pub/sub,client/server, producer/consumer),- transaction modes,- attributes, callbacks (script implemented exits).
- Data Sources - IAS Extra features (event counter, cache and more) - Program logic (can be more than one)
- Execution parameters (numbers of threads etc).
Can be split into two files/locations – configuration and so called registry with environment specific details (eg. QMgr names, switching transport from queues to files for testings).
Web Service Configuration
<?xml version="1.0" encoding="UTF-8"?><spec:specification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:spec="http://www.invenireaude.org/qsystem/workers/spec" xmlns:io="http://www.invenireaude.org/qsystem/workers/io" xmlns:logic="http://www.invenireaude.org/qsystem/workers/logic" xsi:type="spec:Specification">
<inputSpec> <inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output"> <connection alias="qs.web"/> <destination></destination> <txnMode>NONTXN</txnMode> <timeout>-1</timeout> </inputs> </inputSpec> <datasourceSpec> <datasources name="db"> <connection alias="ds.customer"/> <txnMode>NONTXN</txnMode> </datasources> </datasourceSpec> <logicSpec> <logics xsi:type="logic:MatchExecute" instances="2"> <load>svc::getCustomers</load> </logics> </logicSpec> <mode>processor</mode> </spec:specification>
Aliases are references to the registry.
svc/web/inquiry.xml
ESB (MQ) Service Configuration
<?xml version="1.0" encoding="UTF-8"?><spec:specification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:spec="http://www.invenireaude.org/qsystem/workers/spec" xmlns:io="http://www.invenireaude.org/qsystem/workers/io" xmlns:logic="http://www.invenireaude.org/qsystem/workers/logic" xsi:type="spec:Specification">
<inputSpec> <inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output"> <connection alias="qs.esb"/> <destination>SERVICE.IN</destination> <txnMode>NONTXN</txnMode> <timeout>-1</timeout> </inputs> </inputSpec> <datasourceSpec> <datasources name="db"> <connection alias="ds.customer"/> <txnMode>NONTXN</txnMode> </datasources> </datasourceSpec> <logicSpec> <logics xsi:type="logic:MatchExecute" instances="2"> <load>svc::getCustomers</load> </logics> </logicSpec> <mode>processor</mode> </spec:specification>
svc/esb/inquiry.xml
Registry Files
<?xml version="1.0" encoding="UTF-8"?><ns0:specification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns0="http://www.invenireaude.org/qsystem/workers/spec" xsi:type="ns0:Registry">
<io> <connections alias="qs.esb">
<protocol>mqm</protocol><host>TESTMQ</host>
</connections> <connections alias="qs.web">
<protocol>asrvhttp</protocol><host>127.0.0.1</host><port>50000</port>
</connections></io>
<ds><connections alias="ds.customer">
<protocol>sqlite</protocol><location>/var/tmp/customer.ias.db</location>
</connections></ds>
</ns0:specification>
env/registry.xml
Starting Data Processors
export IAS_QS_REGISTRY=${PWD}/env/registry.xml
Test with these commands:
ias_qs_processor f svc/esb/inquiry.xml
ias_qs_processor f svc/web/inquiry.xml
Rule Number Two
Use Service Manager
Use the Service Manager to: - start/stop services - monitor and automatically restart failed ones.
Service Configuration
<?xml version="1.0" encoding="ASCII"?><ns0:serviceConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:ns0="http://www.invenireaude.org/sm/cfg" xsi:type="ns0:ServiceConfig">
<services name="svc.web.inquiry"> <grpAttrs name="svc" value="inquiry" /> <grpAttrs name="type" value="web" /> <grpAttrs name="mon" value="yes" /><startCmd><exe>ias_qs_processor</exe><args>-f</args><args>svc/web/inquiry.qs.xml</args></startCmd><resGrp>rg.svc</resGrp></services>
<services name="svc.esb.inquiry"> <grpAttrs name="svc" value="inquiry" /> <grpAttrs name="type" value="esb" /> <grpAttrs name="mon" value="yes" /><startCmd><exe>ias_qs_processor</exe><args>-f</args><args>svc/esb/inquiry.qs.xml</args></startCmd><resGrp>rg.svc</resGrp></services> <services name="sm.monitor"><grpAttrs name="type" value="sm" /><startCmd><exe>sm_mon_service</exe><args>-m</args><args>mon=yes</args></startCmd><resGrp>rg.mon</resGrp></services>
</ns0:serviceConfig>
<?xml version="1.0" encoding="ASCII"?><ns0:serviceConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:ns0="http://www.invenireaude.org/sm/cfg" xsi:type="ns0:ServiceConfig">
<services name="svc.web.inquiry"> <grpAttrs name="svc" value="inquiry" /> <grpAttrs name="type" value="web" /> <grpAttrs name="mon" value="yes" /><startCmd><exe>ias_qs_processor</exe><args>-f</args><args>svc/web/inquiry.qs.xml</args></startCmd><resGrp>rg.svc</resGrp></services>
<services name="svc.esb.inquiry"> <grpAttrs name="svc" value="inquiry" /> <grpAttrs name="type" value="esb" /> <grpAttrs name="mon" value="yes" /><startCmd><exe>ias_qs_processor</exe><args>-f</args><args>svc/esb/inquiry.qs.xml</args></startCmd><resGrp>rg.svc</resGrp></services> <services name="sm.monitor"><grpAttrs name="type" value="sm" /><startCmd><exe>sm_mon_service</exe><args>-m</args><args>mon=yes</args></startCmd><resGrp>rg.mon</resGrp></services>
</ns0:serviceConfig>
This is a watch dog program which willmonitor all services matching
the expression mon=yes.
sm/SMServices.xml
Resource Group Name – see the next slide
<?xml version="1.0" encoding="ASCII"?><ns0:deploymentConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns0="http://www.invenireaude.org/sm/cfg"xsi:type="ns0:DeploymentConfig" xsi:schemaLocation="http://www.invenireaude.org/sm/cfg../../IAS-ServiceManagerLib/xsd/smcfg.xsd">
<lckDir>/var/tmp/ias/locks</lckDir>
<refreshMS>1000</refreshMS>
<resources name="rg.svc"> <logDir>/var/tmp/ias/logs</logDir> <count>1</count> <exe xsi:type="ns0:ResourceGroupExe"/> <env> <vars name="IAS_DBG_GLOBAL" value="-memorytrace,-info,+stacktrace,+errors,-data"/> </env></resources>
<resources name="rg.mon"> <logDir>/var/tmp/ias/logs</logDir> <count>1</count> <exe xsi:type="ns0:ResourceGroupExe"/></resources>
</ns0:deploymentConfig>
a
<?xml version="1.0" encoding="ASCII"?><ns0:deploymentConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ns0="http://www.invenireaude.org/sm/cfg"xsi:type="ns0:DeploymentConfig" xsi:schemaLocation="http://www.invenireaude.org/sm/cfg../../IAS-ServiceManagerLib/xsd/smcfg.xsd">
<lckDir>/var/tmp/ias/locks</lckDir>
<refreshMS>1000</refreshMS>
<resources name="rg.svc"> <logDir>/var/tmp/ias/logs</logDir> <count>1</count> <exe xsi:type="ns0:ResourceGroupExe"/> <env> <vars name="IAS_DBG_GLOBAL" value="-memorytrace,-info,+stacktrace,+errors,-data"/> </env></resources>
<resources name="rg.mon"> <logDir>/var/tmp/ias/logs</logDir> <count>1</count> <exe xsi:type="ns0:ResourceGroupExe"/></resources>
</ns0:deploymentConfig>
a
Service Deployment
Look for logs files here.
sm/SMDeployment.xml
Preparing Environment
Create locks and logs directories (see SMDeployment.xml)
mkdir p /var/tmp/ias/locksmkdir p /var/tmp/ias/logs
Set the configuration location
export IAS_SM_CFGDIR=${PWD}/sm
Managing Services
sm_start_service Starting: svc.web.inquiry[0] - starting.Starting: svc.web.inquiry[0] -- started.Starting: svc.esb.inquiry[0] - starting.Starting: svc.esb.inquiry[0] -- started.Starting: sm.monitor[0] - starting.Starting: sm.monitor[0] -- started.
sm_dsp_service svc.web.inquiry : X svc.esb.inquiry : X sm.monitor : X sm_stop_service -aStopping: svc.esb.inquiry[0] - will be terminated. Stopping: svc.web.inquiry[0] - will be terminated. Stopping: sm.monitor[0] - will be terminated.
Database Related Feature List
Drivers (native not ODBC):
SQLiteIBM DB2 (commercial)
Oracle (commercial)
Transacted or non-transacted sessions.
XA Support if applicable.
Multithreading.
Summary
Once the environment is setup test driven development is effortless.
Adding a new service may require as little as:- updating data model – usually done by the Service Registry- adding mapping procedure and some extra logic if needed.- deploying implementation with Version Control System of your
choice, no compilation at all.
The data processor provides necessary means needed to build demanding solutions:
- multithreading, asynchronous http(s)- transaction support- transaction grouping with commit counts etc.
Service Manager will provide a fault tolerant solution.
Learn More
Visit:
http://www.invenireaude.com/content/articles/index.html
Build your installation (see Ubuntu Quick Start) in less than 15 minutes.
Download and learn from these tutorials:Customer Information File:
- Database access with Queues and HTTP/REST- Event Driven ESB- Gateways Examples
Complex Event Processing- account transaction processing rules- system log screening