web developer make the most out of your database !
Post on 23-Jul-2015
85 Views
Preview:
TRANSCRIPT
Web Developer, make the most out of your Database !
Leveraging Database Techniques to simplify and performance-optimize your data
intensive web applications
Very rarelyyou always end-up relying on database specific behaviours and
on extensions from SQL standards (ANSI-SQL)
Never use Transact-SQL, PL/SQL, PLpgSQL, PL/php,… ?
Never sort out a performance problem using a database specific feature ?
Don’t you at least rely on constraints (FKs, PKs) or DB sequences/PK auto-increment ?
Database-agnostic App Disadvantages
Commercial : Less options to optimize Data intensive app performance
In the real world you will at some point have to face database specific features (PKs,FKs,DB Sequences,ANSI-SQL non standard syntax)
Coding in the wrong place (Client/Middleware/Database)=Poor Architecture of your App.
See the Database as an important piece of our Application Architecture, a place
where we can add value to our App
Let’s walk through of couple things we can do at the database(*) level
..other than DML statements
(*)We will take the example of an Oracle 12c Database (it’s the one we use) but you will find similar capabilities in other database too
DATABASE VIEWS
create view MYVIEW as select * from MYTABLE;
To hide complexity
Better reusability of SQL complex queries
Avoid modifying applications in case you need to refactor a tableview definition can be reviewed while the view name and attributes stay unchanged.
As a Security mechanismTo surface only the data needed by the application or the user.
As an example : we want to allow only specific account codes to specific user
Application will use an ACCOUNT_CODES_VIEW view to list account codes:-Create view ACCOUNT_CODES_VIEW as select acc_id,… from ACCOUNT_CODES AS T1 ,USERS_AUTHORISED_ACCOUNTS AS T2 where T1.acc_id=T2.acc_id and T2.user_id=T3.user_id and T2.user_ldap_account=<database session application user>
VIEW
TABLE1
SELECT .. FROM TABLE1,TABLE2,TABLE3 WHERE ….
TABLE2 TABLE3
APP
VIRTUAL COLUMNS
Calculated column without de-normalizationNo value stored (only in indexes when indexed)
Avoid modifying applications (if you need to refactor calculation)
TABLE
Col1 NumberCol2 NumberVCol Number as (col1*col2)
select remuneration from EMPLOYEES where remuneration = 10000;You can create an index on a virtual or add constraints to the virtual column.
alter table EMPLOYEES add (Remuneration number as salary+bonus);alter table EMPLOYEES add (Remuneration number as myfunction(salary,bonus));
select * from EMPLOYEES where salary + bonus = 10000;select * from EMPLOYEES where myfunction(salary,bonus) = 10000; where myfunction is a database function (deterministic, not selecting from EMPLOYEES)
DATABASE PACKAGESStored Procedures and Functions
Wrap business logic in a database stored procedure or function to mask complexity and centralize complex and data intensive processes inside the database, close to the data for better
performance.
DATABASE PACKAGESStored Procedures and Functions
create or replace PACKAGE BODY MYPACKAGE asfunction is myfunction1(param1 in varchar2,..) return varchar2 is
begin….. Do something and return a varchar2…..
end myfunction1;function is myfunction2(param2 in varchar2,..) return number isbegin
….. Do something and return a number…..end myfunction2;………..
end PACKAGE_MYPACKAGE;
Good practice to create packages (Package Header + Package Body)
create or replace PACKAGE MYPACKAGE asfunction myfunction1(param1 in varchar2,..) return varchar2;function myfunction2(param2 in varchar2,…) return number;…..
end PACKAGE_MYPACKAGE;
DATABASE PACKAGESCall the package function from :
SQL statements
SELECT column1, mypackage.myfunction1(column2) FROM mytableWHERE column3 = mypackage.myfunction2(column4);
DATABASE PACKAGESCall the package function from :
From other procedures or functions :
create or replace PACKAGE BODY MYPACKAGE as
function is myfunction1(param1 in number,..) return varchar2 is
declare mychar number;begin
….. mychar:=myfunction2(param1);….. Do something and return a varchar2…..
end myfunction1;…end PACKAGE_MYPACKAGE;
DATABASE PACKAGESCall the package function from :
Your Application (Java in this case) :
public void getEmployeeName(int empID) throws SQLException {CallableStatement stmt=null;Connection conn=null;
try {conn getConnection ();String myfunct = “begin ? := mypackage.myfunction1(?); end;”;stmt = conn.prepareCall(myfunct);stmt.setInt(1, empID); stmt.registerOutParameter(2,java.sql.Types.VARCHAR);stmt.execute;String empName = stmt.getString(2);stmt.close();….}
catch(SQLException se){….Finally { ..if stmt!=null) stmt.close();….…
SUBQUERY FACTORING
Example 2 (In-line function)
WITH
FUNCTION myfunction(p_id IN VARCHAR2)
RETURN VARCHAR2
IS
BEGIN
RETURN p_id||'- ok';
END;
SELECT myfunction(mytable.mycolummn) FROM mytable
Example 1 :
WITH T1AS
select T2.col xfrom T2,T3where …
SELECT x from T1
SUBQUERY FACTORING
WITH counter (r)AS(select 1 r from dualunion allselect r+1 from counterwhere r < 5 )
SELECT r from counter
WITH RECURSIVITY
Output :r---12345
Which is similar here to : select rownum r from dual connect by rownum <= 5
SUBQUERY FACTORING
with x( s, ind ) as( select sud, instr( sud, ' ' )from ( select '53 7 6 195 98 6 8 6 34 8 3 17 2 6 6 28 419 5 8 79' sud from dual
)union allselect substr( s, 1, ind - 1 ) || z || substr( s, ind + 1 )
, instr( s, ' ', ind + 1 )from x , ( select to_char( rownum ) z from dual connect by rownum <= 9 ) zwhere ind > 0and not exists ( select null
from ( select rownum lp from dual connect by rownum <= 9 )where z = substr( s, trunc( ( ind - 1 ) / 9 ) * 9 + lp, 1 )or z = substr( s, mod( ind - 1, 9 ) - 8 + lp * 9, 1 )or z = substr( s, mod( trunc( ( ind - 1 ) / 3 ), 3 ) * 3
+ trunc( ( ind - 1 ) / 27 ) * 27 + lp+ trunc( ( lp - 1 ) / 3 ) * 6
, 1 ) ) )
Select s from x where ind = 0;
WITH RECURSIVITY
PIPELINED FUNCTIONS
Create an Array of one dimension:create or replace TYPE " TYP_CHARS_REC" AS OBJECT ( v1 varchar2(200),…, vN varchar2(200))Create an Array of the one dimension array to define a table array :create or replace TYPE " TYP_CHARS_REC_TABLE " is table of TYP_CHARS_REC;
Create a pipelined function :function GET_MYPIPELINED_RECORDS(var1 type1,.., varP typeP) return TYP_CHARS_REC_TABLE pipelinedas …begin…
pipe row (TYP_CHARS_REC (value1,….,valueN));…end;
PIPELINED FUNCTIONS
Select from the pipelined function directly as if it was a table :select * from TABLE(GET_MYPIPELINED_RECORDS(Param1,,,,ParamP));
Wrap it in a database view where possible (no parameters required or parameters set via session context) :create view MYPIPELINED_RECORDS_VIEW as select * from TABLE(GET_MYPIPELINED_RECORDS());
Use the new complex view to select * from MYPIPELINED_RECORDS_VIEW
Use Case examples : • Stock item qty update when store transactions are committed.• Audit logs (prev values and new values audit by user and date)• To enforce complex consistency/compliance checks
Create trigger Store_Issue_Line_Insert_tiggerAfter inserton STORE_ISSUE_LINEFor each rowBegin
Update STOCK_ITEMSset Stock_item_qty= Stock_item_qty - :new.store_issue_line_qtywhere Stock_items.stock_item_id=:new.stock_item_id;
End Store_Issue_Line_Insert_tigger;
DATABASE TRIGGERS
INSTEAD OF TRIGGERSMasking CRUD operations complexity
VIEW1
TABLE1
INSERT INTO VIEW / UPDATE VIEW / DELETE FROM VIEW
TABLE3 TABLEx
APP
INSTEAD OF TRIGGERS
TABLEy
create trigger MY_VIEW1_INSTEAD_OF_TRIGGERinstead of insert or delete or updateon VIEW1for each rowBEGIN
If INSERTING then …do your inserts
Elseif UPDATING then …do your updates
Elseif DELETING then …do your deletes
End if;END;
URL_MATCHComparing Strings similarities
Select Match_strings from(Select TABLE.Column Matched_strings,Utl_Match.Jaro_Winkler_Similarity(TABLE.Column,
’String to match’) match_scoreFrom TABLE)Where match_score>=80;
Lightweight Directory Access Protocol (LDAP) Integration
Use Case :-Database is a good place to store security data (Employees or Customers records are already stored in the database)
Your Applications require a standard LDAP directory to work
Use LDAP functions to sync or manage LDAP entries in the database directly, you can use other data at the same time (check if employee is employed before granting access to some App for example).
DBMS_LDAPExample: Disabling a user in the LDAP directory
…retval PLS_INTEGER:=-1; l_session DBMS_LDAP.SESSION;emp_array DBMS_LDAP.MOD_ARRAY; emp_vals DBMS_LDAP.STRING_COLLECTION;BEGIN
DBMS_LDAP.USE_EXCEPTION := true;l_session := DBMS_LDAP.INIT ('yourldaphostnameIP', 389 ) ;retval :=DBMS_LDAP.SIMPLE_BIND_S (l_session, 'ldapaccount', 'ldapaccountpwd') ;
emp_array := DBMS_LDAP.CREATE_MOD_ARRAY(20);emp_vals(1) := ‘userlogin’;DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'uid',emp_vals);DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'cn',emp_vals);emp_vals(1) := 'DISABLED';DBMS_LDAP.POPULATE_MOD_ARRAY(emp_array,DBMS_LDAP.MOD_REPLACE,'orclisenabled',emp_vals);retval := DBMS_LDAP.MODIFY_S(l_session, ‘cn=userlogin,cn=users,dc=yourdomain,dc=mu',emp_array);DBMS_LDAP.FREE_MOD_ARRAY(emp_array);
retval := DBMS_LDAP.UNBIND_S(l_session);Exception when others then
retval := DBMS_LDAP.UNBIND_S (l_session);return 'Error : '||sqlerrm;
END;
DECLARE
c utl_tcp.connection; -- TCP/IP connection to the Web server
ret_val pls_integer;
TCPresult varchar2(1000);BEGIN
c := utl_tcp.open_connection -- open TCP connection(remote_host => 'www.acme.com', remote_port => 80, charset => 'US7ASCII');
ret_val := utl_tcp.write_line(c, 'GET / HTTP/1.0'); -- send HTTP
request ret_val := utl_tcp.write_line(c); BEGIN LOOP
TCPresult:=utl_tcp.get_line(c, TRUE)); -- read result…. do something ….
END LOOP; EXCEPTION WHEN utl_tcp.end_of_input THEN NULL; -- end of input END; utl_tcp.close_connection(c); END;
UTL_TCPExample: requesting an URL and reading the content
UTL_SMTPDECLARE
c UTL_SMTP.CONNECTION;
PROCEDURE send_header(name IN VARCHAR2, header IN VARCHAR2) ASBEGIN
UTL_SMTP.WRITE_DATA(c, name || ': ' || header || UTL_TCP.CRLF);END;
BEGINc := UTL_SMTP.OPEN_CONNECTION(‘mysmtpserver.dom.com');UTL_SMTP.HELO(c, ‘dom.com');UTL_SMTP.MAIL(c, 'sender@dom.com');UTL_SMTP.RCPT(c, 'recipient@some.com');UTL_SMTP.OPEN_DATA(c);send_header('From', '"Sender" <sender@xyz.com>');send_header('To', '"Recipient" <recipient@abc.com>');send_header('Subject', 'Hello');UTL_SMTP.WRITE_DATA(c, UTL_TCP.CRLF || 'Hello, world!');UTL_SMTP.CLOSE_DATA(c);UTL_SMTP.QUIT(c);
EXCEPTIONWHEN UTL_SMTP.TRANSIENT_ERROR or
UTL_SMTP.PERMANENT_ERRORTHEN
BEGINUTL_SMTP.QUIT(c);
EXCEPTIONWHEN UTL_SMTP.TRANSIENT_ERROR or
UTL_SMTP.PERMANENT_ERRORTHEN null;END;
raise_application_error(-20000,'Failed to send mail due to the following error: ' ||
sqlerrm);END;
UTL_MAIL
UTL_MAIL.SEND (sender => ‘mymail@mail1.com’,recipients => ‘yourmail@mail2.com,cc => ‘hismail1@mail3.com’,bcc => ‘hismail2@mail4.com’,subject => ’My Subject’, message => ’My message text’, mime_type => ‘text/plain; charset=us-ascii’);
UTL_HTTPExample: getting map info from Google Maps API
l_url VARCHAR2(2000); l_response VARCHAR2(4000):=null; myString VARCHAR2(4000); resp_value VARCHAR2(4000);
BEGINl_url := 'http://maps.googleapis.com/maps/api/whatever_api_path_and_param';
beginreq := UTL_HTTP.BEGIN_REQUEST(l_url);
resp := UTL_HTTP.GET_RESPONSE(req);loop
UTL_HTTP.READ_LINE(resp, resp_value, TRUE);l_response := l_response || resp_value;
end loop;UTL_HTTP.END_RESPONSE(resp);exception
when UTL_HTTP.END_OF_BODY thenUTL_HTTP.END_RESPONSE(resp);
end;
myString:=XMLType(l_response).extract('xmlelement/xmlelementb/xmlelement3()').getStringVal();
END;
XMLType is a system-defined opaque type for handling XML data, with predefined member functions like extract to extract
XML nodes and fragments.
HTTPURITYPEAnother Way to easily retrieve HTTP content
select HTTPURITYPE('http://somesite.com/path').getxml() FROM dual;
Output :<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet href="http://somesite.com ...
SELECT HTTPURITYPE('http://google.com/').getClob() FROM dual;
Output :<html><head><meta http-equiv="content-type" content="text/html; charset=ISO-8859 ...
<!DOCTYPE html><!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ --><!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]--><!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]--><!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]--><!-- Consider adding a manifest.appcache: h5bp.com/d/Offline --><!--[if gt IE 8]><!--><html class="no-js" lang="en"><!--<![endif]--><head>
<link rel="dns-prefetch" href="//ajax.googleapis.com"><!-- Mobile viewport optimized: h5bp.com/viewport --><meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Developers Conference 2015 - Mauritius</title><meta name="description" content="The Developers Conference 2015 is a 3-day event going to take place between the 23rd and the 25th of April 2015.
The Mauritius Software Craftsmanship Community (MSCC) is going to organise this event in corporation with local and international sponsors and partners."><meta name="keywords" content="devconmru, developers conference mauritius, gab2015, global azure bootcamp, user group, community, call for papers, internet
of things" />
<!-- Place favicon.ico and mscc-touch-icon.png in the root directory: mathiasbynens.be/notes/touch-icons --><link rel="shortcut icon" href="favicon.ico" type="image/x-icon" />
<!-- Chrome Touch Icons --><meta name="mobile-web-app-capable" content="yes"><link rel="icon" sizes="192x192" href="img/mscc-touch-icon-192x192.png"><link rel="manifest" href="manifest.json">
<!-- Apple Touch Icons -->
SELECT HTTPURITYPE('http://www.devconmru.org/').getClob() FROM dual;
JSON (JavaScript Object Notation) supportExample : a google translate response for
http://ajax.googleapis.com/ajax/services/language/translate?v=1.0....
SELECT HTTPURITYPE('http://ajax.googleapis.com/…').getClob() FROM dual;
You can store the JSON document returned in a table (here my_json_doc) :
CREATE TABLE json_doc ( id number, json_data CLOB, CONSTRAINT json_doc_json_data_chk CHECK (json_data IS JSON) );IS JSON will automatically check that the JSON document syntax is compliant.
SELECT T.json_data.FirstName, T.json_data.LastName, JSON_QUERY(T.json_data, '$.ContactDetails') FROM json_doc T ORDER BY T.json_data.FirstName, T.json_data.Last_name;Output :James Bond {"Email":"james.bond@dom.com","Phone ":""}Lorem Ipsum {"Email": "lorem.ipsum@dom2.com","Phone" :"44 123 123456","Twitter":"@lorem"}
JSON (JavaScript Object Notation) supportAnother example of a web API call this time with a single SELECT
SELECT JSON_QUERY(T.json_data, '$.ContactDetails') from(SELECT HTTPURITYPE('http://ajax.googleapis.com/…').getClob() json_dataFROM mytable)
From dual;
Output :James Bond {"Email":"james.bond@dom.com","Phone ":""}Lorem Ipsum {"Email":« lorem.ipsum@dom2.com","Phone" :"44 123 123456","Twitter":"@lorem"}
TO CONCLUDE
Database can do much more than just storing data
Keep data near the database tier as much as possible
Leverage your database to mask complexity and to surface your data ready for consumption
Use Database Views and Packages with an API approach to encapsulate database functionality
Use Database functionality to reach out to external data
top related