web developer make the most out of your database !

Web Developer, make the most out of your Database !

Leveraging Database Techniques to simplify and performance-optimize your data

intensive web applications

Jean-Marc DesvauxHead of Information Systems

GENERAL CONSTRUCTION CO.LTD

@jmdesvaux

How many times have you changed an Application underlying Database ?

Very rarelyyou always end-up relying on database specific behaviours and

on extensions from SQL standards (ANSI-SQL)

Never use Transact-SQL, PL/SQL, PLpgSQL, PL/php,… ?

Never sort out a performance problem using a database specific feature ?

Don’t you at least rely on constraints (FKs, PKs) or DB sequences/PK auto-increment ?

Commercial :Customer wants to choose

Database desupport protection

Database-agnostic App Advantages

Database-agnostic App Disadvantages

Commercial : Less options to optimize Data intensive app performance

In the real world you will at some point have to face database specific features (PKs,FKs,DB Sequences,ANSI-SQL non standard syntax)

Coding in the wrong place (Client/Middleware/Database)=Poor Architecture of your App.

Why don’t we leverage the Database development capabilities for our Apps ?

See the Database as an important piece of our Application Architecture, a place

where we can add value to our App

Let’s walk through of couple things we can do at the database(*) level

..other than DML statements

(*)We will take the example of an Oracle 12c Database (it’s the one we use) but you will find similar capabilities in other database too

DATABASE VIEWS

create view MYVIEW as select * from MYTABLE;

To hide complexity

Better reusability of SQL complex queries

Avoid modifying applications in case you need to refactor a tableview definition can be reviewed while the view name and attributes stay unchanged.

As a Security mechanismTo surface only the data needed by the application or the user.

As an example : we want to allow only specific account codes to specific user

Application will use an ACCOUNT_CODES_VIEW view to list account codes:-Create view ACCOUNT_CODES_VIEW as select acc_id,… from ACCOUNT_CODES AS T1 ,USERS_AUTHORISED_ACCOUNTS AS T2 where T1.acc_id=T2.acc_id and T2.user_id=T3.user_id and T2.user_ldap_account=<database session application user>

VIEW

TABLE1

SELECT .. FROM TABLE1,TABLE2,TABLE3 WHERE ….

TABLE2 TABLE3

APP

VIRTUAL COLUMNS

Calculated column without de-normalizationNo value stored (only in indexes when indexed)

Avoid modifying applications (if you need to refactor calculation)

TABLE

Col1 NumberCol2 NumberVCol Number as (col1*col2)

select remuneration from EMPLOYEES where remuneration = 10000;You can create an index on a virtual or add constraints to the virtual column.

alter table EMPLOYEES add (Remuneration number as salary+bonus);alter table EMPLOYEES add (Remuneration number as myfunction(salary,bonus));

select * from EMPLOYEES where salary + bonus = 10000;select * from EMPLOYEES where myfunction(salary,bonus) = 10000; where myfunction is a database function (deterministic, not selecting from EMPLOYEES)

DATABASE PACKAGESStored Procedures and Functions

Wrap business logic in a database stored procedure or function to mask complexity and centralize complex and data intensive processes inside the database, close to the data for better

performance.

DATABASE PACKAGESStored Procedures and Functions

create or replace PACKAGE BODY MYPACKAGE asfunction is myfunction1(param1 in varchar2,..) return varchar2 is

begin….. Do something and return a varchar2…..

end myfunction1;function is myfunction2(param2 in varchar2,..) return number isbegin

….. Do something and return a number…..end myfunction2;………..

end PACKAGE_MYPACKAGE;

Good practice to create packages (Package Header + Package Body)

create or replace PACKAGE MYPACKAGE asfunction myfunction1(param1 in varchar2,..) return varchar2;function myfunction2(param2 in varchar2,…) return number;…..

end PACKAGE_MYPACKAGE;

DATABASE PACKAGESCall the package function from :

SQL statements

SELECT column1, mypackage.myfunction1(column2) FROM mytableWHERE column3 = mypackage.myfunction2(column4);


From other procedures or functions :

create or replace PACKAGE BODY MYPACKAGE as

function is myfunction1(param1 in number,..) return varchar2 is

declare mychar number;begin

….. mychar:=myfunction2(param1);….. Do something and return a varchar2…..

end myfunction1;…end PACKAGE_MYPACKAGE;


Your Application (Java in this case) :

public void getEmployeeName(int empID) throws SQLException {CallableStatement stmt=null;Connection conn=null;

try {conn getConnection ();String myfunct = “begin ? := mypackage.myfunction1(?); end;”;stmt = conn.prepareCall(myfunct);stmt.setInt(1, empID); stmt.registerOutParameter(2,java.sql.Types.VARCHAR);stmt.execute;String empName = stmt.getString(2);stmt.close();….}

catch(SQLException se){….Finally { ..if stmt!=null) stmt.close();….…

SUBQUERY FACTORING

Example 2 (In-line function)

WITH

FUNCTION myfunction(p_id IN VARCHAR2)

RETURN VARCHAR2

IS

BEGIN

RETURN p_id||'- ok';

END;

SELECT myfunction(mytable.mycolummn) FROM mytable

Example 1 :

WITH T1AS

select T2.col xfrom T2,T3where …

SELECT x from T1

SUBQUERY FACTORING

WITH counter (r)AS(select 1 r from dualunion allselect r+1 from counterwhere r < 5 )

SELECT r from counter

WITH RECURSIVITY

Output :r---12345

Which is similar here to : select rownum r from dual connect by rownum <= 5

SUBQUERY FACTORING

with x( s, ind ) as( select sud, instr( sud, ' ' )from ( select '53 7 6 195 98 6 8 6 34 8 3 17 2 6 6 28 419 5 8 79' sud from dual

)union allselect substr( s, 1, ind - 1 ) || z || substr( s, ind + 1 )

, instr( s, ' ', ind + 1 )from x , ( select to_char( rownum ) z from dual connect by rownum <= 9 ) zwhere ind > 0and not exists ( select null

from ( select rownum lp from dual connect by rownum <= 9 )where z = substr( s, trunc( ( ind - 1 ) / 9 ) * 9 + lp, 1 )or z = substr( s, mod( ind - 1, 9 ) - 8 + lp * 9, 1 )or z = substr( s, mod( trunc( ( ind - 1 ) / 3 ), 3 ) * 3

+ trunc( ( ind - 1 ) / 27 ) * 27 + lp+ trunc( ( lp - 1 ) / 3 ) * 6

, 1 ) ) )

Select s from x where ind = 0;

WITH RECURSIVITY

PIPELINED FUNCTIONS

Process data and pipe rows as you go

PIPELINED FUNCTIONS

Create an Array of one dimension:create or replace TYPE " TYP_CHARS_REC" AS OBJECT ( v1 varchar2(200),…, vN varchar2(200))Create an Array of the one dimension array to define a table array :create or replace TYPE " TYP_CHARS_REC_TABLE " is table of TYP_CHARS_REC;

Create a pipelined function :function GET_MYPIPELINED_RECORDS(var1 type1,.., varP typeP) return TYP_CHARS_REC_TABLE pipelinedas …begin…

pipe row (TYP_CHARS_REC (value1,….,valueN));…end;

PIPELINED FUNCTIONS

Select from the pipelined function directly as if it was a table :select * from TABLE(GET_MYPIPELINED_RECORDS(Param1,,,,ParamP));

Wrap it in a database view where possible (no parameters required or parameters set via session context) :create view MYPIPELINED_RECORDS_VIEW as select * from TABLE(GET_MYPIPELINED_RECORDS());

Use the new complex view to select * from MYPIPELINED_RECORDS_VIEW

Use Case examples : • Stock item qty update when store transactions are committed.• Audit logs (prev values and new values audit by user and date)• To enforce complex consistency/compliance checks

Create trigger Store_Issue_Line_Insert_tiggerAfter inserton STORE_ISSUE_LINEFor each rowBegin

Update STOCK_ITEMSset Stock_item_qty= Stock_item_qty - :new.store_issue_line_qtywhere Stock_items.stock_item_id=:new.stock_item_id;

End Store_Issue_Line_Insert_tigger;

DATABASE TRIGGERS

INSTEAD OF TRIGGERSMasking CRUD operations complexity

VIEW1

TABLE1

INSERT INTO VIEW / UPDATE VIEW / DELETE FROM VIEW

TABLE3 TABLEx

APP

INSTEAD OF TRIGGERS

TABLEy

create trigger MY_VIEW1_INSTEAD_OF_TRIGGERinstead of insert or delete or updateon VIEW1for each rowBEGIN

If INSERTING then …do your inserts

Elseif UPDATING then …do your updates

Elseif DELETING then …do your deletes

End if;END;

URL_MATCHComparing Strings similarities

Select Match_strings from(Select TABLE.Column Matched_strings,Utl_Match.Jaro_Winkler_Similarity(TABLE.Column,

’String to match’) match_scoreFrom TABLE)Where match_score>=80;

Lightweight Directory Access Protocol (LDAP) Integration

Use Case :-Database is a good place to store security data (Employees or Customers records are already stored in the database)

Your Applications require a standard LDAP directory to work

Use LDAP functions to sync or manage LDAP entries in the database directly, you can use other data at the same time (check if employee is employed before granting access to some App for example).

DBMS_LDAPExample: Disabling a user in the LDAP directory

…retval PLS_INTEGER:=-1; l_session DBMS_LDAP.SESSION;emp_array DBMS_LDAP.MOD_ARRAY; emp_vals DBMS_LDAP.STRING_COLLECTION;BEGIN

DBMS_LDAP.USE_EXCEPTION := true;l_session := DBMS_LDAP.INIT ('yourldaphostnameIP', 389 ) ;retval :=DBMS_LDAP.SIMPLE_BIND_S (l_session, 'ldapaccount', 'ldapaccountpwd') ;

emp_array := DBMS_LDAP.CREATE_MOD_ARRAY(20);emp_vals(1) := ‘userlogin’;DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'uid',emp_vals);DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'cn',emp_vals);emp_vals(1) := 'DISABLED';DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'orclisenabled',emp_vals);retval := DBMS_LDAP.MODIFY_S(l_session, ‘cn=userlogin,cn=users,dc=yourdomain,dc=mu',emp_array);DBMS_LDAP.FREE_MOD_ARRAY(emp_array);

retval := DBMS_LDAP.UNBIND_S(l_session);Exception when others then

retval := DBMS_LDAP.UNBIND_S (l_session);return 'Error : '||sqlerrm;

END;

DECLARE

c utl_tcp.connection; -- TCP/IP connection to the Web server

ret_val pls_integer;

TCPresult varchar2(1000);BEGIN

c := utl_tcp.open_connection -- open TCP connection(remote_host => 'www.acme.com', remote_port => 80, charset => 'US7ASCII');

ret_val := utl_tcp.write_line(c, 'GET / HTTP/1.0'); -- send HTTP

request ret_val := utl_tcp.write_line(c); BEGIN LOOP

TCPresult:=utl_tcp.get_line(c, TRUE)); -- read result…. do something ….

END LOOP; EXCEPTION WHEN utl_tcp.end_of_input THEN NULL; -- end of input END; utl_tcp.close_connection(c); END;

UTL_TCPExample: requesting an URL and reading the content

UTL_SMTPDECLARE

c UTL_SMTP.CONNECTION;

PROCEDURE send_header(name IN VARCHAR2, header IN VARCHAR2) ASBEGIN

UTL_SMTP.WRITE_DATA(c, name || ': ' || header || UTL_TCP.CRLF);END;

BEGINc := UTL_SMTP.OPEN_CONNECTION(‘mysmtpserver.dom.com');UTL_SMTP.HELO(c, ‘dom.com');UTL_SMTP.MAIL(c, '[email protected]');UTL_SMTP.RCPT(c, '[email protected]');UTL_SMTP.OPEN_DATA(c);send_header('From', '"Sender" <[email protected]>');send_header('To', '"Recipient" <[email protected]>');send_header('Subject', 'Hello');UTL_SMTP.WRITE_DATA(c, UTL_TCP.CRLF || 'Hello, world!');UTL_SMTP.CLOSE_DATA(c);UTL_SMTP.QUIT(c);

EXCEPTIONWHEN UTL_SMTP.TRANSIENT_ERROR or

UTL_SMTP.PERMANENT_ERRORTHEN

BEGINUTL_SMTP.QUIT(c);

EXCEPTIONWHEN UTL_SMTP.TRANSIENT_ERROR or

UTL_SMTP.PERMANENT_ERRORTHEN null;END;

raise_application_error(-20000,'Failed to send mail due to the following error: ' ||

sqlerrm);END;

UTL_MAIL

UTL_MAIL.SEND (sender => ‘[email protected]’,recipients => ‘[email protected],cc => ‘[email protected]’,bcc => ‘[email protected]’,subject => ’My Subject’, message => ’My message text’, mime_type => ‘text/plain; charset=us-ascii’);

UTL_HTTPExample: getting map info from Google Maps API

l_url VARCHAR2(2000); l_response VARCHAR2(4000):=null; myString VARCHAR2(4000); resp_value VARCHAR2(4000);

BEGINl_url := 'http://maps.googleapis.com/maps/api/whatever_api_path_and_param';

beginreq := UTL_HTTP.BEGIN_REQUEST(l_url);

resp := UTL_HTTP.GET_RESPONSE(req);loop

UTL_HTTP.READ_LINE(resp, resp_value, TRUE);l_response := l_response || resp_value;

end loop;UTL_HTTP.END_RESPONSE(resp);exception

when UTL_HTTP.END_OF_BODY thenUTL_HTTP.END_RESPONSE(resp);

end;

myString:=XMLType(l_response).extract('xmlelement/xmlelementb/xmlelement3()').getStringVal();

END;

XMLType is a system-defined opaque type for handling XML data, with predefined member functions like extract to extract

XML nodes and fragments.

HTTPURITYPEAnother Way to easily retrieve HTTP content

select HTTPURITYPE('http://somesite.com/path').getxml() FROM dual;

Output :<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet href="http://somesite.com ...

SELECT HTTPURITYPE('http://google.com/').getClob() FROM dual;

Output :<html><head><meta http-equiv="content-type" content="text/html; charset=ISO-8859 ...

<!DOCTYPE html><html class="no-js" lang="en"><head>

<link rel="dns-prefetch" href="//ajax.googleapis.com"><meta name="viewport" content="width=device-width, initial-scale=1.0">

<title>Developers Conference 2015 - Mauritius</title><meta name="description" content="The Developers Conference 2015 is a 3-day event going to take place between the 23rd and the 25th of April 2015.

The Mauritius Software Craftsmanship Community (MSCC) is going to organise this event in corporation with local and international sponsors and partners."><meta name="keywords" content="devconmru, developers conference mauritius, gab2015, global azure bootcamp, user group, community, call for papers, internet

of things" />

<link rel="shortcut icon" href="favicon.ico" type="image/x-icon" />

<meta name="mobile-web-app-capable" content="yes"><link rel="icon" sizes="192x192" href="img/mscc-touch-icon-192x192.png"><link rel="manifest" href="manifest.json">



SELECT HTTPURITYPE('http://www.devconmru.org/').getClob() FROM dual;

JSON (JavaScript Object Notation) supportExample : a google translate response for

http://ajax.googleapis.com/ajax/services/language/translate?v=1.0....

SELECT HTTPURITYPE('http://ajax.googleapis.com/…').getClob() FROM dual;

You can store the JSON document returned in a table (here my_json_doc) :

CREATE TABLE json_doc ( id number, json_data CLOB, CONSTRAINT json_doc_json_data_chk CHECK (json_data IS JSON) );IS JSON will automatically check that the JSON document syntax is compliant.

SELECT T.json_data.FirstName, T.json_data.LastName, JSON_QUERY(T.json_data, '$.ContactDetails') FROM json_doc T ORDER BY T.json_data.FirstName, T.json_data.Last_name;Output :James Bond {"Email":"[email protected]","Phone ":""}Lorem Ipsum {"Email": "[email protected]","Phone" :"44 123 123456","Twitter":"@lorem"}

JSON (JavaScript Object Notation) supportAnother example of a web API call this time with a single SELECT

SELECT JSON_QUERY(T.json_data, '$.ContactDetails') from(SELECT HTTPURITYPE('http://ajax.googleapis.com/…').getClob() json_dataFROM mytable)

From dual;

Output :James Bond {"Email":"[email protected]","Phone ":""}Lorem Ipsum {"Email":« [email protected]","Phone" :"44 123 123456","Twitter":"@lorem"}

AND MANY MORE ….

TO CONCLUDE

Database can do much more than just storing data

Keep data near the database tier as much as possible

Leverage your database to mask complexity and to surface your data ready for consumption

Use Database Views and Packages with an API approach to encapsulate database functionality

Use Database functionality to reach out to external data

THANK YOUEnjoy the rest of your day

@ DEVCONMRU

web developer make the most out of your database !

Technology