june 2003 - cbdi

June 2003Eventually we will move to a simpler,

higher level platform that is designed

specifically for SOA.

CBDIJournal3 Editorial

4 Roadmap ReportBuilding Web Services – The FutureIt may come as a shock to read in this journal that

components and objects may not necessarily be the

best future development platform for services.

In this report we think the unthinkable and challenge

conventional wisdom.

By Jonathan Stephenson

10 Best Practice Report Services Oriented Architecture -Part 4 – The PlatformThis report deals with the technology platform that

supports business services, and, more importantly, its

separation from those services. Such separation is

crucial in enabling business services to evolve at the

speed of the business, rather than being constrained

by technology change.

By Oliver Sims

18 Best Practice ReportWeb Services To Improve BusinessIntelligenceThe core attributes of Web Services enable some really

exciting and revolutionary ways to get more, better

business intelligence. In this report we set some

expectations and identify some vendors that seem to

understand this perspective.

By Richard Veryard

inside...

Insight for Web Service & Software Component Practice

Moving to Web Services?Learn from CBDI with our Consultingand Workshops

CBDI provide a variety of educational and consulting services based on our deep and narrow

focused knowledge acquired in the analytical process. The new services are designed to help

you work smart and make the right decisions, as you move into Web Services. There are

products applicable to enterprises and software organizations.

Product Area CBDI Education Consulting ApplicableSpecialist Workshop Products Enterprise

Areas Or Industry

1. Web Service Understanding Y 1 Day Enterprise

2. Web Services Strategy and Roadmap Y YEnterprise

And Industry

3. Service Based Business Analysis and Design Y 1 or 2 Day Y Enterprise

4. Web Service Architecture Y 1 or 2 Day Y Enterprise

5. Business Process Design for Web Services Y Y Enterprise

6. Web Service Design YEnterprise

And Industry

7. Web Services Security Interop Y 1 Day YEnterprise

And Industry

8. Web Services Management Y 1 Day Enterprise

9. Information Modeling for Service OrientedY 3 Day Y Enterprise

Architectures

10. Web Services Integration Platforms Y Enterprise

11. Selecting a Web Services Toolkit Y Y Enterprise

12. Process Design for Service and ComponentY Enterprise

Delivery and Consumption

13. Software Vendor Product Management Review Y Y Industry

14. Software Vendor Marketing Support Y Y Industry

15. Venture Capital Support Services Y Industry

To find out more see our product catalog at:http://www.cbdiforum.com/public/services_bro.php3

To discuss how CBDI can assist call us on: +353 28 38071 or 73

Web Services bring dramatic simplification to the consumer

of a service. Essentially this simplicity is delivered by hidingthe complexity behind the service interface. But under the

covers we still have all the old complexities of the service

implementation. In fact for many organizations the service

simply wraps the monolithic morass of existing, packaged

and legacy applications that remain largely unchanged and

in many cases untouchable.

Editorial

CBDI JOURNAL © CBDI Forum Limited, June 2003 3

www.cbdiforum.com

What we are doing is analogous tosweeping our dirt under the carpet –out of sight, out of mind.Most enterprises have applicationportfolios that reflect years of constantchange in the IT world. The outdatedprogramming languages, compliers,middleware, models etc are all there,hopefully hidden from view behind nicenew wrappers. If we dig too deep, wesee them in all their diversity, in the sameway we can sometimes see eons ofgeological rock strata exposed by glacialactivity or erosion on rock faces and cliffs.

The notion of layers or strata provides apowerful reminder that what we have isnot just legacy, but potentially many,many mappings between the layers, withhuge potential for inefficiency. Now let’simagine that we ruled the world, and thatwe had the power to change things areclearly in great need of improvement. Ofcourse we wouldn’t start from wherewe are today; we would envisage anenvironment where the implementationwas optimized specifically for SOA.There are some clear indications thatsuggest some of the software platformproviders in our industry, large and small,are thinking along these lines right now,for both the development and run-timeenvironments.

In this months CBDI Journal our leadreport by Jonathan Stephenson examines

this assertion, and concludes that this isalready happening. We can already seepractical implementations of a worldthat is moving beyond objects andcomponents. In this report we think theunthinkable and make a detailed casethat the current environments aresuboptimal for delivering Web Servicesand SOA. Further we examine a realworld alternative that shows how end -to - end XML can deliver massivesimplification of the implementationenvironment.

Also this month we publish the fourthpart of Oliver Sims detailed expositionon SOA - The Platform, in which headdresses the issues of separation. Hestresses the criticality of moving all yourmany glue efforts into the provisionof a single integrated ESOA blueprintand virtual service-oriented platform,precisely to protect, or to take advantageof changing implementation layers.

On the one hand, replacing today’s C#and Java environments may seem along way off, but with today’s priorityon efficiency and cost control, manyorganizations are likely to view thisapproach as having the strategicpotential to radically alter the costequation of software delivery. Rememberyou heard it here first!

Regards, David [email protected]

Thinking the unthinkable -

eventually we will move to

a simpler, higher level

platform that is designed

specifically for SOA.

4 CBDI JOURNAL © CBDI Forum Limited, June 2003

By Jonathan Stephenson IntroductionXML has become the lingua franca of

business integration and application

development. Web services is a recent

example of the inexorable march of XML

into all aspects of corporate IT but

despite the fact that we may describe

data using XML schema, and transport it

as SOAP, developers insist of converting

it back to binary object format when ever

they need to process it. This is neither

logical nor efficient.

It may come as a shock to read in the

CBDI Journal that components and

objects may not necessarily be the best

future development platform for services.

Today we have little choice but to build

components using Java and .NET and

then to expose their services through

SOAP URIs. In this report we will think

the unthinkable and challenge this

assumption. We start by looking at

how end-to-end XML could work, the

problems and inconsistencies inherent

with the current deployment platform

and then investigate the alternative vision

heralded by the Water platform and the

rumored X# language.

The Corporate XML StackThe XML standards bodies have ensuredby their work that XML can be used todescribe and store data objects, anddescribe processes, architectures andtransformations. The following table listssome of relevant standards/proposedstandards:

Data schema XML Schema

Process XLANG, BPEL4WS

Object messages SOAP

Procedure calls SOAP

Interfaces WSDL

UML designs UMI

Object meta data MOF

XMLTransformation

XSLT

Data queries XPATH, XQUERY

In most areas XML has been used tosimplify the description of objectsbecause it is readable and readilyextensible. In the B2B domain we canmodel a procedure that involves twocompanies with completely different ITinfrastructures and build a common

It may come as a shock to read in this journal that

components and objects may not necessarily be the best

future development platform for services. In this report we

think the unthinkable and challenge conventional wisdom.

Building Web Services –The FutureLooking Beyond Components and Objects

Roadmap Report

Components and

objects may not neces-

sarily be the best future

development platform for

services.


continues...

▼▼

String strQty = request.getParameter(“Qty”);

if strQty !=null && strQty.length()>0{

try {

boOrder.Qty =Integer.parseInt(strQty);

bOK=true;

} catch (NumberFormatException e {

}

}

Listing 1. Sample Java for Form processor for 1 integer

▼▼

▼▼

Platform de jureLet’s start by following a typical piece ofdata as it travels the tortuous journeyfrom internet user to corporate database.To build the systems we use today youneed skills in HTML, Java Script, Javaand SQL. As the data entered into theweb browser winds its way to thepurchasing table of the relationaldatabase it passes through manytranslations, we start on the web page:

Web TierThe user is presented with a form, aset of entry fields where his order iscomposed; then the form is HTTPPosted to the web server where it isprocessed by a server page (JSP/ASP).We focus on the integer where thequantity of goods is typed.

Conversion 1The text is tested for consistency withthe data type and then converted

to a Java/.NET type. Allthe parts of the salesorder are extracted fromthe form in this way andcomposed as an Orderobject.

TEXT Java int

TEXT .NET int

The sample code in listing 1 shows thetext is checked and converted to aninteger – this must be repeated for everyinput field on the form.

process model and data schema usingXML that both systems can implement.The finished system exchanges data usinga HTTP connection and to the uneducatedeye the system seems to demonstrateend-to-end XML processing. Scratch thesurface however and it’s a differentstory. Each step in the processing pathinvolves conversion of the XML to andfrom other binary standards.

For all the important steps in the process,moving data around, transforming data,describing data, etc. XML has beenadopted, but for the programming of thelogic and storing the data in a relationaldatabase we are stuck with oldtechnology such as 3GLs (Java/C#) andSQL. The relational database vendorshave made a start along the road toend-to-end XML and most can map nXML schema to their relational tablerepresentation. So the area we will focuson will be the 3GL.

What Is An Object Anyway?Most of us are happy to treat objectsas conceptual entities. Analysts anddesigners use them to model a systemand hand them over to developers to turninto 3GL classes. The classes are thetemplates from which objects are createdwithin the binary address space of thelanguage interpreter. Unfortunatelywhenever we need to store the object inthe corporate (relational) database orsend one across the wire we have totranslate the object into anotherrepresentation. Like electrons, we haveevidence that they existed but we neversaw them. Does this matter? Yes,because as we split our applicationarchitecture into more layers the numberof object-XML-object translations growsand this is both inefficient and a potentialsource of errors. The WS-I compatibilitytests testify to the potential for error whendealing with hierarchical structures.

Conversion 2The number is now held in an attribute of

a programming object in the web tier.

Next the web service call is made to

create the order and to do this a SOAP

call is made. This involves

the Web Services utility

classes in conversion

from native type to XML.

Our integer is serialised

back to text again so that

the data can be sent via

an HTTP Post to another

web server.

Java int TEXT XML SOAP Schema

.NET int TEXT XML SOAP Schema

Conversion 3The SOAP listener processes the XML

message, locates the implementation for

the service (.NET or Java class) and after

converting the Text back into the program

language type, passes the parameter to

the component. The input parameters

to your service implementation may be

designed as an object type, a set of

separate parameters or

even an XML document –

either way this step

converts the XML type

back to a programming

language type.

XML TEXT Java int

XML TEXT .NET int


Building Web Services – The Future continued...

occur? Or a more topical example couldbe the mapping of a hierarchical objectstructure to an XML schema.

Components and Services– Do They Make Sense?Apart from the numerous datatransformations there are other issues toconsider when we use a component builtin .NET/Java to provide a service. Aservice is a function that is alwaysavailable; it doesn’t need to be initializedor explicitly created unlike an objectwhere the programmer has to instantiatethe object before calling it, for example:

clsOrder MyOrder = New clsOrder();

So a service is implemented by an objectwhich has to be created and destroyed,while the service remains accessible24x7. When a service is called a proxyobject is first created, the service iscalled and the proxy destroyed. When anapplication server uses a pool of objectsto implement a service there is still oneservice. The service is a façade thathides the component implementation; ithas a threading model governed by theweb server, not the application server.This discussion I hope highlights the factthat services and components are notnecessarily a well-matched couple –more of a shotgun wedding than a matchmade in heaven.

One of the issues here of course isthat all the commonly used businesslanguages were invented way beforeXML and e-commerce were around. Thesaying, if you have only a hammereverything looks like a nail, means thateverything has to be an object even if itdoesn’t really behave like one. XMLdocuments are force fitted into objectsand services are built from methods onobjects. Objects are easier to work withbecause the notation used to accessproperties and methods is so intuitive.

An XML document in contrast needs tobe parsed to get to its nodes (nodes areequivalent to properties) and it has nomethods. So we load the XML into anobject and manipulate it from there.

Where to from here?The current response to these issues bythe main players in development tools isto make the XML support packagesmore powerful. Gradually the code thatwe write becomes filled with thefunctionality provided by these utilitiesand less of the native languageconstructs, until someone decides tobuild a platform in which the XMLdocument and its data types becomesthe built-in data type.

So, instead of writing in Java tomanipulate XML, to access data usingSQL for storage in tables, we could useXML to represent programming objects.If we designed it correctly, data couldremain in XML from the point of entry, beprocessed by services written in XMLand be stored as XML documents.

Let’s begin by building our wish list:

● On the web tier, forms create wellformed XML documents andentered data is automaticallychecked against an XML schema

● Server pages are written in an XMLbased language that works directlyon XML content and can makecalls to Web Services which wouldbe treated as ordinary functions,except they start with http://

● Web services and web resourcesare part of the language systemand do not need a utility layer.

● Services accept XML and canstore them in a database in theirnative XML. The database layertakes care of normalization andindexing.

Conversion 4The last stage of the journey is to store

the data, and to do this it is likely that we

need an SQL Insert statement for the

relational database. The order object

now has to become a

normalized table structure

of header and items. But

before we can insert the

record we must choose

either dynamic SQL or a

parameterized command

– let’s make it easy and

build a SQL string – that’s

right, another conversion

back to text:-

Java int SQL TEXT

.NET int SQL TEXT

Conversion 5Finally one last conversion

from SQL to native

database tables.

SQL TEXT RDBMS Record

SQL TEXT RDBMS Record

Is this as bad as it gets?You might be thinking that we have tried

to make this a worst case scenario, but

far from it. It could be yet more tortuous

if we start passing XML documents to

our service layer and then processing

these with a DOM (Document Object

Model) parser. This would require the

population of an in-memory hierarchical

object to represent the XML document.

Or the service would be an older CICS or

similar COBOL system with data types

different from those used by the web tier.

At each conversion there is possibility of

error or incompatibility, simple ones

like date formats have always been a

problem - is the string “2/3/67” a date in

February or March to take but a trivial

example? Does the XML Integer map to

the database integer or can an overflow

▼▼

▼▼


continues...

The next section looks at Water in moredetail, but we shouldn’t continue withoutmentioning the rumour that Microsoft istoo working on the same idea, and wehear it was called X# until someonelet the cat out of the bag and it washushed up.

Concise XMLThe trouble with any proposal to useXML for functional programming is itsverbose syntax. There are two immediateanswers to this, either hide the full listingof the text using an IDE or shorten thesyntax. The second option has beenadopted by the Water language and it is

● The same language is used for alltiers of the application, which isstored as XML and uses XML data asthe native data type.

If you ever worked with the Lispprogramming language you might seesome interesting parallels here. Theunification of the data representationand the language opens up manypossibilities. If the language and the datais all XML then a business object issimply some data and some code in apackage. Unlike the business objecttoday which is some memory residentconstruct that can’t exist outside thelanguage interpreter, a business objectbecomes a document that can be listed,printed and checked by a human. Whatis even more obscure today is that thedata represented by the object neverexists in the database; it is split intonormalized tables and the RDBMSdoesn’t even ‘know’ that the data wasonce an object with any meaning. Therelational database vendors have allstarted to address the need to handleXML and we have included a brief surveyof three standard databases and theirXML capability, see Table 1.

OK, so we have thought the unthinkableand you will not be surprised to learn thatothers are ahead of us and have built it,or at least some of it. If you look atwww.waterlang.org you will find a welldocumented example of an XML-basedlanguage that uses Concise XML dataas the native type. Concise XML isdescribed later. The working interpreteris there for you to try and an integrateddevelopment environment called Steamexists as a commercial tool fromwww.clearmethods.com. In addition tothis important language there are alsosome XML standards or proposedstandards that relate to our problemspace.

Vendor

MSFT, SQLServer

Oracle

DB2

Type of support

XML to table mapping

XPATH

FOR XML Query

XML Storage

Internet repository

XML Extender

Validation

Indexing

XPATH

Version

2000 on

9i Rel 2

Description

XML mapping schema allow XML to be mapped totables for storage and retrieval

XPATH queries are supported through ADO

SQL Select extended to return XML documents

New column type XML can create XML documentrepository or map nodes to relational data structures

Allows XML content to be made available to internetusing JDBC, JNDI, FTP, WebDAV or servlet APIs

XML to relational mapping

Validate with DTD prior to storing

Indexed searches on XML elements

XPATH queries supported in UDFs

Table 1: XML to Relational Support in 3 Leading Databases

XQUERY

XFORMS

XDOC

XSLT

XPATH

XQuery is a powerful and convenient language designed forprocessing XML data

An alternative to HTTP forms that knows about XML schemaand can create user interfaces for data entry with output in XMLdata. See [email protected] list

Microsoft’s new XML forms package

An XML transformation language which includes XPATH expressions

A language for returning parts of an XML document orexpressions based on the nodes of the document

Table 2: Related XML standards or proposed standards

Building Web Services – The Future continued...

Item which has three fields (properties):

<Item id=”145”, Size=’large’,

Colour=”red”/>

I now assign the object to a named fieldso I can reference it from my code:

<set my_item=<Item id=”145”,

Size=’large’, Colour=”red”/> />

The Water ImplementationRather than repeating the Water tutorialhere, we refer you to www.waterlang.orgfor the full story and take a higher levelview of how Water fits into our vision ofend-to-end XML.

Water is implemented using Java; thismeans that is runs on all platforms in theweb tier, client and server. If you have aJava VM and/or a servlet engine you canuse Water. As it sits on top of the JavaVM it has database access throughJDBC drivers but also implements ahigher level database programmingmodel where Water objects are storedand retrieved from a table.

There is no reason why Water could notbe implemented on an alternativeplatform, .NET C# or unmanaged C++for example, but Java is the obvioussubstrate for maximum platformcoverage. It will be interesting, if itemerges, to see what Microsoft havedone in this area. .NET is designed on aSOAP-based object bus but follows themore traditional approach of loadingXML into a binary object for processing.

End-To-End Concise XMLArchitectureThe goal of never having to write a line ofcode to convert between binary objectsand XML representation can beachieved, albeit by using Concise XML in the programming language. The solutiondrawn out in the Figure 1 just about runson any platform, today, and you candownload everything you need free ofcharge from the web.

the way entities and attributes are used.

C. Fry’s discussion of this issue can be

found at http://www.waterlang.org/doc/trouble_with_xml.htm. Water’s

Concise XML (cXML) is more consistent

and this is used to represent both objects

and logic.

The WATER LanguageWater is a general purpose programming

language that uses concise XML as its

basic syntax. It represents objects using

XML and treats web services just like a

local function call. So for example, this

code snippet sets up a service to get the

current temperature and then calls it to

return the temperature of a given US town:

<set current_temperature=

www.<web “http://weather.com/

current/temperature”/>

/>

<current_temperature city=”new

york”/> ? “50”

To build an XML object there is no

special utility to learn, everything is

treated as an object. The following code

creates an instance of an object called

called ConciseXML. ConciseXML has asimple correspondence with the full XMLrepresentation. The important changesare illustrated in Table 3.

These simple changes allow you to writecode in XML without getting writer’scramp, here is an example of somefunctional XML code

<defmethod field_widths_to_

start_end_pairs

positions=required=other

_unkeyed_args>

<set P=0/> 

<set R=<vector/> /> 

positions.<for_each>

R.<insert P

<do> <set P=P.<plus

value/> />

P

</do>

/>

</for_each>

R

</defmethod>

When XML 1.0 is used to represent

objects it leads to ambiguities due to

Description

End tags

Positionalparameters

Dot notationfor paths

Attributesrepresentation

Full XML

<MyTag>

Conent of tag

</MyTag>

<person

first=”Mike”

last=”Plusch”/>

<execute_path>

“abe”

<foo

color=”blue”/>

bar

</execute_path>

<list <item

name=”bread”/>

<item

name=”milk”/> />

ConciseXML

<MyTag>

Conent of tag

</>

<person “Mike” “Plusch”/>

“abe”.<foo color=”blue”/>.bar

<list>

<attributes><item

name=”bread”/><item

name=”milk”/></attributes>

</list>

Table 3: Concise XML vs Full XML

There’s no smoke

without fire...


technologies and processes have beengetting more and more convoluted.However, today’s priorities and focusare efficiency and cost control andmany organizations are going to viewthis approach as having the strategicpotential to radically alter the costequation of software delivery.

This whole aspect of service developmentcame to our attention when the MicrosoftX# rumor made it into the IT press. Sincethen a closer look at Water and a morequestioning look at the programmingmodel used across the tiers of anenterprise architecture has thrown upsome interesting answers for where wemay be headed, one day. It has takenmuch effort for industry to narrow themainstream development platformsdown to Java and C# so we can’t seethis being overturned in a hurry but theWater language has shown how natural itis to create native XML capability in alanguage interpreter. If we believe the oldadage about smoke and fire we mustsurely conclude that there are peopleinside the largest software company inthe world working on a project to makeend-to-end XML a mainstream technology.

Jonathan [email protected]

We have shown both the WaterDBapproach and the XML mappingsupported by the RDBMS vendors. In thelatter approach it would be necessary toconvert between the Water object format(cXML) and XML 1.0; this is built-infunction of the Water runtime.

So far we have put this forward andresisted playing devil’s advocate. Sowhat are the down sides to this? Here’sour list:

1. Water is implemented by a relativelyunknown software vendor.

2. Few programmers have heard of it(although we hope we havecontributed here) and support islimited to one company.

3. Concise XML is still not as conciseas Java, although there will be muchless of it when dealing with XML.

4. Water is an interpreter running on topof an interpreter and which will haveperformance implications; but asbefore if you can eliminate many linesof Java this could possibly run faster.

ConclusionsWeb Services bring massive simplicity tothe process of consuming services; butunder the covers the implementation

Figure 1: The Water Solution


By Oliver Sims The ESOA that positions you not onlyfor the present but also for the futurerests on a firm architectural basis thatdelivers:

● Separation of the technologyplatform from the businessservices offered, and from thebusiness logic that implementsthose services

● Flexibility and responsiveness sothat the services offered bothwithin and outside the enterprisecan respond fast to the evolutionof the business

Previous reports in this series haveillustrated some of the main conceptsand approaches involved in aneffective ESOA. This report dealswith the technology platform thatsupports business services, and, moreimportantly, its separation from thoseservices. Such separation is crucial inenabling business services to evolveat the speed of the business, ratherthan being constrained by technologychange.

First we outline the rationale forseparation of business servicesfrom their underlying technologyinfrastructure. Then we consider thenature of the ESOA – what kind of“architecture” it is. Third, wesummarize some key aspects of theinfrastructure products available onthe market today, and discover thereare some holes! Finally, we present anapproach to “filling the holes”.

Of course, filling holes with “glue” iswhat IT developers have been doingfor years now. “Glue” is thecolloquial name for the designsand implementations of softwaretechnology that all IT departmentsproduce in order to add local functionto COTS1 products. What counts,however, is not so much the filling ofthese holes, but the way they’re filled.

Finally, we show how ESOA makes avirtue of the necessary glue to providean effective “virtual platform” that canprovide for the rigorous separationbetween business and technologyfunction that is required.

Building an Enterprise Service-Oriented Architecture (ESOA)

is not easy. It’s true that some impressive immediate gains

can be made, but the reason we recommend moving to an

ESOA is precisely to ensure that those early gains become

the foundation of a truly productive and responsive service-

oriented system. Without such a foundation, you may find,

after a couple of years, that you have created rigid and

inflexible stovepipes - service-oriented legacy!

Services Oriented Architecture -Part 4 - The Platform

Best Practice Report

1. COTS means “Commercial Off-The-Shelf”, and is an oft-used acronym for products that can be bought.Here we also include open source products that can be downloaded without charge.


continues...

technology, and vice-versa. Figure 1illustrates the effect: not only do bothbusiness and technology changes resultin modifications to the IT systems, butalso each change is one of both businessand technology, thus doubling the effectof each change! By the way, this appliesto new function too, not just changes.And what if we need to move to a newplatform, or we’re developing servicesfor two or more platforms? It isimperative that we shield our serviceimplementations as much as possiblefrom the impact of underlying softwaretechnology, whether from changes tothat technology, or from the need to runservices on several different platforms.Mature ESOA does just this, and thisreport describes how it does so.

In this report, we take a top-down view ofthe technology “platform” and describean approach that protects, as much aspossible, the business services and theirimplementations from the evolution andchange that must occur as platformsmature or change. By “platform” we meanthe set of DBMSs, application servers,GUI infrastructures, communicationsstacks, system services such as logging,configuration, naming, etc., that underlie

IntroductionService orientation is quintessentiallyabout satisfying enterprise businessrequirements in a responsive andefficient way. The major inhibitor to this isthat the business changes and evolvesat a quite different rate than the rate ofchange in underlying software andhardware technology. Why is this aninhibitor? Because most currentapproaches to application softwaredevelopment fail to separate businesslogic needed to implement services fromthe underlying software technologysuch as application servers. Look at anybusiness-level application code today,and the chances are you’ll find all sorts oftechnology code buried inextricablywith the business logic, from GUI-drivingcode to thread and transactionmanagement code. This means thatchanges to business function drag intechnology concerns, and changes totechnology impact business logic. Bothare inhibited. And this is not some minortechie problem. It means is that everytime the business needs something tochange, or needs some new function,the business logic developers areimmediately immersed in software

the business services and upon whichthey run.

The ESOA “Blueprint”How do we get a handle on separatingsoftware technology function fromsoftware business services? Well, the“A” in ESOA stands for “architecture”.This is not there just to pad out asnappy acronym. It means that service-orientation at the enterprise level must beclearly designed, and that design mustbe visible. In other words it must have anarchitecture. But “architecture” canmean many things. Here, we use it in thesense of a blueprint that applies to arange of different applications, all builtaccording to the same structuralprinciples. When a number of differentapplications can all share the samestructure, and where the relationshipsbetween the parts of the structure are thesame, then we have what might be calledan “architectural style” [Hubert] (thesame concept is also called a “productline” [Clements], or part of an “approach”[Herzum]). An architectural style can beexpressed through what we call a“blueprint” (also sometimes called a“metamodel”).

Such a blueprint becomes the expressionof the “architecture” in ESOA. It is adetailed design for:

● The structure of the modules to bebuilt by business developers, howthey relate to each other, wherethe services are offered, and towhom (internal, external, etc.)

● The transparencies enjoyed bybusiness developers – that is, theextent to which software technologyis hidden from them

● A specification of how extra-functional challenges such asscalability, buildability, performance,etc. will be addressed.

Figure 1


Services Oriented Architecture - Part 4 - The Platform continued...

We saw a part of such a blueprint inpart 1 of this series, where we discussedwhat a “component” is. A furtherexample of what the blueprint addressesis the “distribution” part of a distributedsystem – that is, the logical distributiontiers. These are areas of responsibility ofbusiness logic as seen by the businessdeveloper. They are separate from, butmappable to, the physical distribution.For reasons beyond the scope of thisreport, we prefer a four-tier model to theoften-used three-tier model. Briefly, thetiers are:

● User – the specification of theuser interaction with some device(including screen layouts)

● Workspace – the business logicinvolved with managing a user’ssession. This tier provides servicesto the user tier.

● Enterprise – the business logicinvolved in providing enterprise-level services to authorized

● A concept of development thatrigorously separates businesslogic from the technologyinfrastructure, and also explicitlyaddresses mapping of platformindependent designs to thenecessary platform-specificartifacts. This is exactly theprovince of the OMG’s newstrategy, “Model-DrivenArchitecture” or MDA [OMG].Space prohibits further discussionof MDA in this report.

● A concept and implementation of“platform” that enables theseparation to be as complete aspossible. Of course, it can neverbe completely separate: if it were,then business logic could not run!But we can push the boundary ofthe platform upwards so thatseparation becomes as clean andas complete as possible.2

The key to separation is to define a“virtual platform” for business developersthat is deterministically mappable to anumber of real platforms. Figure 3illustrates this. Note the additionalblueprints (designs) for the virtualplatform and for the real platforms. Forexample, components built according tothe business service blueprint could bemapped or transformed to J2EE or aCCM implementation or ,NET, all of whichsupport the concept of component. Themapping or transformation (or partialtransformation) would produce businessservice components that are of thespecific form supported by the targetplatform.

Now this all seems like a huge task. Butit isn’t. Most IT departments have skilledpeople who produce bits of code, oradd-ons to development tools, that makethings easier for application developers.

requestors. This tier providesservices to the workspace tier, andalso to other areas such as B2Bcollaborations

● Resource – the business logicrequired to access resources suchas data (or printing). This tierprovides resource services to theenterprise tier.

Figure 2 illustrates these four logicaltiers, and shows how they can bemapped to a variety of physical systems.

Figure 2 is fairly high-level, and illustratesthe “distribution” concept that is part ofthe set of core concepts within an ESOA.However, the objective for the ESOAblueprint is to define things, based onthese core concepts, at a sufficiently lowlevel of detail that the same business-level code can be mapped to severaldifferent physical system structures,and to differing software technologyplatforms. This requires two things:

Figure 2

2. MDA defines “platform” as follows: “A platform in general is a set of subsystems/technologies that provide a coherent set of functionality through interfaces andspecified usage patterns that any subsystem that depends on the platform can use without concern for the details of how the functionality provided by the platformis implemented.” [OMG] page 2-3.


continues...

available today, implementing an ESOAis difficult. There are two reasons for this:

● COTS products are point solutions,not ESOA solutions

● COTS products are general-purpose

● IT development is (typically)project-based

Point SolutionsIT development organizations today arefaced with a huge choice of middlewareand development tools. Somewhere onthe market, you can find everythingneeded to implement an ESOA. The mainconstituent parts of EOSA run-time plat-form include such things as an applica-tion server (component container mid-dleware), Web Service support (see[Sims]), DBMS, and a user interface infra-structure. The function required includes:

● Web Service definition andpublishing

● Optimization mechanisms so thatweb services can be used whererequired within a single addressspace

● XML handling

● Transaction processing andConcurrency support

● Event management through anotification service

● Workflow and B2B support

● User Interface infrastructure

● DBMS

● Caching mechanisms

And for each of these, there are a numberof sub-functions needed – for example,effective transaction processing needsthread and connection pooling, andevent management needs queuing.

The development platform includes suchthings as web service definition tools,compilers, repositories, GUI designtools, and modeling tools. Again,sub-functions are needed, such as theability to interchange artifacts amongthe various tools.

The overall problem is that, althougheverything required is available, it’s notavailable in a single integrated product.This results in high levels of complicationacross the whole developmentenvironment. From the CIO/ChiefArchitect’s view, all you can buy fromvendors today are big construction kits,where you often have to make upyour own assembly and operationinstructions. There are lots of specs andinstructions for the individual parts, andmuch advice on sub-assemblies -assuming you understand all the parts.The main areas of complexity are:

● Technical complications throughoutthe development environment.Addressing these requires highlevels of scarce skill for manydevelopment projects. The result islow effective skill levels applied tomany developments, with theinevitable poor quality and re-work.

● Rapid change in and evolution ofsoftware technologies. This resultsin high levels of technology churn,and a disinclination to install new

A run-time example of “glue” is a loggingservice (it’s surprising how many realplatforms do not provide this, or provideone whose function needs to beexpanded in some way). A development-time example of glue is a script that takesan analysis-level model and transforms itinto the beginnings of a design-levelmodel. In each case, what’s really beingdone is to produce a virtual platform.However, the blueprints generally onlyexist in the minds for the glue providers,and are often lost when the project ends.Figure 3 suggests that this process, socommon in so many projects across theindustry, should be formalized andapplied explicitly within the enterprise sothat they are not lost and re-inventedproject by project.

Current Status for ESOAPlatformsA mature ESOA platform actuallycomprises two important “platforms”:the run-time platform, and the“development platform” used bybusiness developers to build servicecomponents that will run on the run-timeplatform. However, despite the widerange of COTS products that are

Figure 3


or upgraded products. Whentechnology is changed, this has asevere knock-on effect on thebusiness function portfolio. This isexacerbated by business evolutionand change (naturally) being out ofsync with technology change

● Lack of focus on an ESOAarchitectural style (see next section).

Thus we have a situation where a greatdeal of work is needed to turn thecollection of products into an effectiveESOA platform.

General-purpose COTS productsThe reason there are holes in the COTSproducts available today is thatproducers have not appreciated theconcept of architectural styles. Hencethey have had to produce very general-purpose platforms, suitable for manyarchitectural styles. However, it is theconcept of architectural style thatenables a platform to be very high level.Implementing services in a distributedenterprise system can done with aparticular architectural style, and hencethe platform for that style could be muchhigher-level than that provided by thegeneral-purpose COTS products. Thereare some indications that this may -perhaps may - be changing. But don’thold your breath for the next two years.

Project-Based DevelopmentThe second impediment to effectiveESOA implementation is a strong“project orientation” within thedevelopment organization. Of course,this does not mean that projects per seare an impediment. However, whereeach development project has significantauthority to choose (some of) its owntechnology code or products, and also tobuild its own glue, then it can beextremely difficult to achieve a sensibleseparation between software technologyand business function. This is because insuch an environment:


And this when much current applicationdevelopment and design thinking isoptimized for yesterday’s environment,characterized by OO and fat clientdesign plus rapid developmentprocesses such as RAD and XP. To thishas been added distributed systemthinking characterized by thin or “rich”clients and app server technologies.Finally, the underlying software technologyis often researched and adopted on aper-project basis, with little other thanoperating systems and major middlewareproducts being used as a commoninfrastructure across projects.

Changing to more productive developmentenvironments while facing such pressuresis difficult, and implies organizational aswell as design paradigm changes. Themajor inhibitor is the lack of an end-to-endcohesive product that directly aims atlifting productivity - an “enterpriseproductivity platform” (EPP). While feworganizations are able to meet thechallenge of building an EPP as amarketable product, many can move inthe right direction using existing productscombined with a tight focus on harvesting

● There is little funding for effortsoutside the project, and so:

● There is limited re-use of commonartifacts, whether softwaretechnology or application modules,across projects; rather, there isoften a great deal of re-invention.

Such organization means that expertiseand artifacts are not shared acrossprojects, and so valuable learning andknowledge is dissipated at the end ofeach project. This adds to the cost ofeach project, but such costs are notvisible because the cost basis is project-oriented, not service-oriented. Figure 4illustrates this.

Now one of the striking things about ITorganizations is that they are oftenrelatively impervious to the businesspressure for dramatic time-to-marketimprovements. This is because ITorganizations are fighting on two majorfronts:

● Time-to-market pressures frombusiness stakeholders

● Dealing with rapid and wide-ranging technology “churn”

Figure 4

While few organizations

are able to meet the

challenge of building an

enterprise productivity

platform as a marketable

product, many can move

in the right direction using

existing products.


continues...

software technology can be hiddenmuch more effectively.

In other words, there is an urgent need toview the glue as creating a virtualplatform for service developers. Space prevents detailed discussion, but twoexamples can illustrate the kinds ofconcerns involved. Both look atproviding simplicity and transparency tobusiness service developers in ways thatare specific to the ESOA architecturalstyle, but independent of underlyingplatform technology. The two examplesinvolve invoking services from someclient, and building a front-end.

Invoking a business serviceFirst, we look at an example of the kindof glue code often provided for businessdevelopers on an ad hoc basis byfriendly colleagues working in the sameproject. You may wish to re-read thesection “Creating and accessing a WebService” from part 1 of this series to getan overview of what the developer isdoing. The example is one where abusiness developer needs to invoke a“Place_Order” service (say from a PC orfrom a web server). The information he orshe has is the order data (in a variableorder_data ), and a key - the string

their skills, and the assets that thoseskills have produced, to provide an in-house equivalent. The objective is toshare assets across projects, and have asingle way of filling the holes in theexisting product providing the the run-time platform and the developmentenvironment. Without such a move with-in IT, business stakeholders are likely tocontinue to be disappointed.

Filling the HolesWe talked about the prevalence of “glue”code, and the need to take an enterpriseview of the effort in producing it. Tosummarize this, Figure 5 shows therequirement at the top, what’s availablein the middle (showing the holes), andthe way that glue fills the gaps at thebottom. From the business servicedeveloper’s point of view, where muchingenuity and time must be applied toproviding the business logic (processes,procedures, algorithms, data handling,etc.), the less software technology thebetter. However, much glue function isbuilt from the bottom up, with importantaspects left to the business developer “incase they’re needed”. With a clearblueprint, the things that are neededcan be pre-defined, and a great deal of

Figure 5



// Invoke the Place_Order service:

Error = bus_service.invoke

(“Place_Order”,”create”,order_data);

Support for either of the above twoapproaches is more complex to write,even if it’s much more helpful to thebusiness developer. However, this isthe kind of thing that’s defined in theblueprint, written once, and provided asa standard aspect of the virtual platformfor all business developers. It clearlyseparates business logic from softwaretechnology since this form can bemapped within the virtual platform toa number of different technologies –not only different web service COTSproducts, but also, other non-webtechnologies such as CORBA.

User InterfaceThe user interface is an area oftenassigned to junior programmers toimplement. This is a mistake. The userinterface is where the system actuallydelivers benefit to the enterprise. Theless usable it is, the less benefit. Inaddition, UI code is particularly difficultto write well, requiring much low-leveltechnical detail that is technology-specific. For example, an HTML-definedUI is quite different from one usingSwing. However, it is possible toconstruct a UI framework - part of thevirtual platform - that not only simplifiesthings a great deal, but can also shieldthe business developer from technologydifferences. Such an infrastructureincludes such capabilities as:

● Navigation schemes

● Widget standards

● Internationalization concerns

● Performance framework (so thatthe users do not experience the“sticky mouse” syndrome, nor dothey wait for server access whenperforming trivial operations suchas tabbing off a field).

Today, XML is starting to be applied to

“Place_Order” - to the URL that identifiesthe WSDL that is the effective interface to the service implementation. The code the developer might write to place the orderis as follows, where the code is actuallysimple pseudocode, and lines beginning“//” are comments:

// Get the URL of the web service

url = wsdl_location_service.find

(“Place_Order”);

// Bind to web service whose

WSDL is at the specified URL

web_service =

wsdl_system.bind(url);

// Create a proxy to use to

access the web service;

order_proxy =

proxy_service.makeproxy

(web_service)

// Now invoke Create Order:

error = order_proxy.create

(order_data);

Notice the three functions used bythe developer – wsdl_location_service,wsdl_system, and proxy_service. At leastone, probably two, and perhaps all ofthese are “helper” functions writtenlocally to assist the business developer,and not provided by the web serviceCOTS product being used. The friendlycolleagues have, on an ad-hoc basis,picked out the parts that they believedcould do with some help, and haveprovided it in a way that reducessoftware technology complexity tosome extent.

Now let’s take a top-down view, and ask,if I’m a business developer, what might Iprefer to code here. A good answer is:

// Find the “Place_Order” service:

bus_service =

system_service.find(“Place_Order”);

// Invoke the service:

error =

bus_service.create(order_data);

Experience suggests that thissimplification is not only eminently do-able, but also better than the followingeven briefer and still viable approach:

We recommend that

you seriously consider

moving all your many glue

efforts into the provision

of a single integrated

ESOA blueprint and

virtual service-oriented

platform.


the UI. In the mid-90s, I helped build acomponent container middlewareproduct which used a pre-cursor to XML as the only way to define GUIs. A layouttool produced XML-like definitions,which were interpreted dynamically. Thisworked well on 16MHz processorsrunning Windows 3.1, and on slightlymore powerful PCs running Windows95.Performance was not a problem. Neitherwas UI building (see [Eeles]). Today, thisapproach is being developed using XML,and XUL (XML User Interface definitionLanguage - see [XUL]) is an interestingapproach to simplification. Here is anexample of a simple XUL panel definition:

<xul>

<window title="Nexaweb Example" >

<button text="Button 1" />

<button text="Button 2"/>

</window>

</xul>

Given an appropriate run-time container,this can dynamically produce a windowcontaining two push-buttons. We canexpand this to make the push-buttons dosomething:

<xul>

<window title="Simple Example

Window" >

<button text="Button 1" oncommand=

”/myapp/button1click.jsp”/>

<button text="Button 2" oncommand=

”/myapp/button2click.jsp”/>

</window>

</xul>

There is already work being done toproduce a layout tool, so that you don’thave to write raw XML. However, muchof the work visible on the web is still apoint solution. Our experience in thisarea has taught us that an holisticsystem-wide approach is required tointegrate this effectively into an overallsystem where data shown on userinterfaces is obtained from a large shared

database. For example, consider auser getting a partial list of entities(e.g. Customers) from the enterprisedatabase, and double-clicking on oneitem in the list to open a window showingthat customer’s details. The functioninvolved in this can be generalized tocover any entity, so that all the developerhas to do for a given entity type is tospecify the display format of the list, andof the panel where details will be shown.

ConclusionCOTS platform products tend to be pointsolutions, and are not integrated withother products. It is a significant job tointegrate them, and to provide somesupport for business service developers.Where that support is not provided,then the integration task is placed onthe business developers, and thisoverhead is sometimes repeated in eachdevelopment project. When support isprovided, it is often ad hoc, and usefulglue is often not captured for re-use else-where.

Mature ESOA suggests doing this workonce, capturing it, and re-applying itacross service provisioning projects. Thisinvolves consciously providing andmaintaining an ESOA virtual platform.While this is unlikely to increase overallcosts over a period, and should reducethem - perhaps significantly, the costprofile within a development organizationwill certainly change.

Some initial indications suggest thatperhaps, in the longer-term plans ofsome of the big software providers inour industry, there may be an intention toaddress a new higher-level platformfocused on the architectural stylerequired by ESOA. This would not onlyaim at the run-time, but also at thedevelopment-time environment. Further

good news is that the OMG’s MDA islikely to move tool developers in thesame direction. It may turn out to be arace to be the first software vendorto put an ESOA virtual platform, includ-ing a high-productivity developmentenvironment, onto the market.

Meanwhile, we recommend that youseriously consider moving all your manyglue efforts into the provision of a singleintegrated ESOA blueprint and virtualservice-oriented platform.

This does not mean a big bangapproach. Proven techniques exist toevolve towards an EPP for ESOA. Themajor inhibitor is the effort andmanagement focus needed to plan andtake the first steps towards the cohesiveholistic environment required. Thismeans that in-house efforts to create an

EPP cannot provide “instant gratification”- indeed, it may be six months from thestart before benefit is seen. The solutionto this is a focused transition program tomove from IT’s current state to the ESOAvirtual platform and EPP.

The next report (the last in this series)will address just such a process fortransitioning to an ESOA virtual platform,and the organizational implications ofsuch a move.

ReferencesClements – Clements & Northrop, Software ProductLines, Addison Wesley 2001

Eeles – Eeles & Sims, Building Business Objects,Wiley 1998

Herzum - Herzum & Sims, Business ComponentFactory, Wiley 2000

Hubert – Richard Hubert, Convergent Architecture,Wiley 2002

OMG – MDA Guide version 1.0, OMG documentomg/2003-05-01

Sims – CBDi Journal, March 2003, “Part 1 –Foundation”.

XUL – The Open XUL Alliance -http://xul.sourceforge.net/links.html

3. The book Business Component Factory (Herzum & Sims, Wiley, 2000) was based on an early in-house Productivity Platform that was highly successful in the late90s, but which has since disappeared for various reasons.


Best Practice Report

By Richard Veryard

With many new concepts there is a tendency for everyone

to jump on the bandwagon. Thus we often see vendors

rebranding their products and refashioning their marketing

messages – sometimes regardless of genuine fit, relevance

and readiness. But in the area of BI, there is strong

justification for vendors to be getting excited about Web

Services. The core attributes of Web Services potentially

enable some really exciting and revolutionary ways to get

more, better business intelligence. In this report we set

some expectations and identify some vendors that seem

to understand this perspective.

Web Services To ImproveBusiness IntelligenceHow business intelligence fits into aservice-oriented world

IntroductionWeb services are being widely discussedfor integration of operational businessprocesses. Many enterprises are startingto deploy web service technology forconnecting applications internally. Thereis significant interest in deploying thesame technologies externally, forconnecting applications betweenmultiple organizations, although this iscurrently inhibited by concerns aboutsecurity and the immaturity of adequatestandards.

Meanwhile, the use of the Internet as aplatform for business intelligence (BI) isbecoming more mature and sophisticated.There is an important role for webservices in the business intelligence

space, and some of the specialist BI

vendors are starting to support web

services.

Business intelligence is an important

management function, and can

undoubtedly benefit from the range of

technological innovations with which

CBDI members will be familiar, including

web services and grid computing. We

expect the deployment of these

technologies, and the construction of a

services-oriented architecture for the

enterprise, to embrace management

systems including BI, and not be

restricted to operational systems.

In this report, we review the moves that

the BI vendors have already taken to


continues...

Web services can be used to make thisfunctionality available remotely over the internet, and a number of BI vendorsoffer web service front-ends to theirbasic OLAP products, including CrystalDecisions, Cognos and Microstrategy.

Some BI vendors offer a reduced versionof this, in which the remote servicesmerely provide access to a set ofpredefined reports, or generatepredefined management alerts, with noability to perform analysis dynamically.This is the case for many so-calledportals or dashboards, in which theservice can be understood simply as aone-way information flow from the datastore to the manager. For example,Microsoft’s new product SQL ServerReporting Services (SSRS)1 is presentedas a Business Intelligence tool, but isessentially a server-based enterprisereporting engine, providing a useful .NETdata hub but with no on-line analyticfunctionality.

Some BI vendors appear to think that theBI process terminates with informationdissemination. And if the aim is solely todisplay information on remote users’screens, then it isn’t obvious whydelivering the information using webservices is any better than simplydelivering the information in HTMLform to a standard browser – theso-called zero footprint solution thathas been commonplace for some time.Fortunately, there are some BI vendorsthat take a broader view.

Closed Loop BIEven with full functionality for query,reporting and analysis, the traditionalview of business intelligence only givesus a partial picture of the process,lacking a broader system purpose andcontext of management control andaction. Figure 1 encourages a misleadingview of business intelligence as a

support web services, and offer a

framework for integrating BI more

comprehensively into the service-

oriented world.

Business IntelligenceFrameworkThe Traditional View: Query,Reporting and AnalysisAt the core of most BI systems are

software products providing query,

reporting and analysis functionality,

sometimes referred to as OLAP.

Traditionally, these functions have been

based either directly on operational

systems, or more commonly on a

data warehouse or data mart, which

assembles and restructures data from

one or more operational data stores.

This is shown in Figure 1.

Figure 1: Traditional View ofBusiness Intelligence

The result of the data combination can

be stored in another data store, or may

remain virtual. In many cases, the data

are obtained from both relational and

multidimensional OLAP schemas, and

are combined into a hybrid schema

(Hybrid OLAP or HOLAP) which is then

usually virtual.

passive activity by individual managers.

It is a known pitfall for business

intelligence to be taken over by clever

number-crunchers, identifying fascinating

statistical patterns with no practical

relevance for management action.

Figure 2 Closed-Loop BusinessIntelligence

In contrast, Figure 2 shows how business

intelligence can be understood as a

closed control loop. Managers use tools

to process and interpret information; they

then act upon this information and

monitor the effects of their actions. If the

actions have the expected effect on

business performance, this helps to

confirm the original interpretation; if

management intervention doesn’t work

in the expected way, then this should

trigger further analysis. This management

feedback and learning loop is a key

element of true business intelligence.

Closed loop business intelligence also

includes the possibility of management

actions whose primary purpose is to gain

more information/intelligence – to learn

something or to test a hypothesis. A

major retailer offers a special price for a

specified product in selected stores,

and then watches the effect on sales

volumes. The FBI may arrest a few minor

1. Not to be confused with SQL Server Resolution Services, also known as SSRS.


members of a criminal gang in order

to provoke the gang leaders into a

detectable reaction.

MicroStrategy’s Transactor product is

designed to support such closed-loop

business intelligence, combining

monitoring and control into a single

on-line transaction. Users view business

intelligence information on-line and can

directly initiate actions against an

operational system, such as database,

ERP or legacy. Typical applications

include: inventory control & reordering,

authorization, and budgeting.

Real-Time EnterpriseAs well as web services for business

intelligence, it is also worth mentioning

the converse: business intelligence for

web services. We have previously referred

to the growing complexity of SOA

solutions, and the need for increasingly

sophisticated management tools to

remain on top of this complexity. Some

specialized intelligence is already found

in some system management platforms,

such as IBM’s Tivoli, especially to monitor

security threats. However, there is clearly

an opportunity to develop this kind of

functionality further to achieve a broader

range of autonomic system behavior2.

Thus in the Real-Time Enterprise, as

shown in Figure 3, transactions and

events can be intercepted and analyzed

as they happen, without waiting for

them to reach a data store, and this

leads to the possibility of a real-time

or near-real-time system response. A

good example of this is the Marks &

Spencer Profit Protection System

described in Box 2.

However, while web service and XML

technologies generate many new and

exciting opportunities for the Real-Time

Enterprise, this usually needs to be

placed within a broader business

intelligence context. This is especially

important where security is concerned,

since a system that relies on a fixed set

of rules and policies is usually vulnerable

to intelligent attack. Further off-line

analysis will allow the policies to be

continually reviewed and refined.

It is also important to provide a context

for making sense of events and trends.

Regular readers will recall our analysis of

the Kodak case3, in which the online

retailer found itself obliged to supply a

large number of digital cameras at an

incorrect price. Real-time business

intelligence can help identify a sudden

increase in demand for a given product,

and may place some constraints on

automatic supply until the increase can

be explained. But we need some context

for this. It is only when we can link a

sudden increase EITHER to a marketing

campaign by Kodak OR to some hostile

activity on an internet newsgroup that we

know what to make of it - and therefore

how to respond.

Thus there is still a role for stored data for

business intelligence in the Real-Time

Web Services To Improve Business Intelligence continued...

Enterprise. Among other things, we

usually need to be able to analyze data

for the current period, and this requires

reviewing transactions for the past 24

hours, or the past 7 days or whatever.

Business intelligence is often about

detecting changes, trends and patterns,

finding “the difference that makes a

difference”4, and is not just about the

absolute numbers prevailing at the

present. If we want to perform

sophisticated statistics, we need to have

enough data to produce statistically

significant results. We need to be able

to distinguish meaningful events from

random noise.

There is also a role for off-line processing.

Although we can expect a progressive

shift towards autonomic systems in

which some self-correction and self-

protection functions can be coded as

policies and executed automatically, at

least for the foreseeable future there is

always going to be a management/

learning loop that goes through human

brains and therefore requires some

thinking time.

One of the features of Tivoli is the ability

to reduce large volumes of security data

to rather smaller volumes of security

information, which is better for human

inspection. Business intelligence software

may be used for filtering and selecting

things for human attention - but these

filtering rules will themselves need some

a management/learning loop.

It may well be that human intervention

now operates at a higher level of

abstraction - perhaps by refining policies.

But any intervention should still have a

detectable effect on WIGO (what is

going on). Sometimes there will be a

considerable delay before this effect is

discernable. This is yet another reason

2. CBDI Journal January 2003 http://www.cbdiforum.com/secure/interact/2003-01/auto.php3

3. CBDI COMMENTARY - WHERE DID THEY GO, I JUST DON'T KNOW... February 2002.

4. This was the definition of “Information” proposed by Gregory Bateson, one of the founding fathers of systems thinking.

Figure 3: The Real-Time Enterprise


continues...

Provide query & reporting servicesto users. Provide dynamic analysisservices to users.

Obtaining real-time data via web services.Intercepting XML documents in transit.

Presenting a virtual data store as aweb service.

Exercising management action throughweb services.

Disseminating management actionthrough web services.

Using web services to supportpeer-to-peer communications betweenknowledge workers.

Figure 4: Broader View of BusinessIntelligence

Figure 5: Place of Web Services inBusiness Intelligence

why the BI system as a whole needs to

have a memory.

A Broader Framework for BIPutting the closed-loop business

intelligence shown in Figure 2 together

with the real-time enterprise shown in

Figure 3, we have produced the general

framework for business intelligence

shown in Figure 4.

With this broader view of business

intelligence, we can now see several

additional ways in which web services

Business Intelligence for Real-Time Fraud DetectionThe UK retailer Marks & Spencer had an offline analysis to detect fraud with a6-week delay, using a traditional data warehousing approach. It replaced this withan on-line profit protection system that can alert the security officer (now retitled“profit protection officer”) in under 30 seconds. Among other things, the systemdetects suspicious or physically impossible sets of transactions (such as using thesame credit card in different stores at the same time). Yet the system still handlesevery transaction in every store.

The transactions are still recorded in the data warehouse for other analyticalactivates that are not so time sensitive, whilst the profit protection systemexamines the same transaction data before it gets to the data warehouse. Thebusiness rules are defined dynamically by the profit protection officers.

Box 2: Example: Real-Time Fraud Detection

Business Intelligence for Real-Time Credit ControlBanks have traditionally set overdraft limits and other credit limits through a slowbatch process. These limits are set when an account is opened, and raisedperiodically as a consequence of a “good” transaction history. (What counts as a“good” transaction history often varies over time.)

But the obvious problem with a batch process is that it is slow to respond tochanging circumstances. When a customer gets into financial difficulties, it cansometimes take months before the bank can detect the warning signs andwithdraw credit facilities.

Some banks have been experimenting with the concept of dynamic delinquency,where simple business intelligence is applied to the transaction record on acontinual basis, to provide a much faster recognition and response.

Credit limit is now a derived attribute.

Box 1: Example: Real-Time Credit Control

can play a useful role and help integrate

the BI process.

Some of these steps may also involve

grid technology. (See our report in the

CBDI Journal for January.) Grid concepts

may be used to present a heterogeneous

array of data stores as a single virtual

data store. Grid concepts may also be

used to link an array of managers and

other knowledge workers, collectively

making sense of large amounts of

complex information.

Examples


Web Services To Improve Business Intelligence continued...

Design and Process IssuesExternalizationWhile there remains considerableresistance to using external web servicesfor “business critical” transactionsystems, the situation is rather differentin the field of business intelligence.Indeed, business intelligence applicationsalready use external data to great extent– stock trades, statistical sources etc.We can expect the use of web servicesto result in more sources, more traffic,easier integration. As binding and billingsystems improve, we can expect greaterflexibility and wider usage.

Design derived attributes as servicesIn Box 1, we discuss the growingdevelopment of real-time or near-real-time financial controls, such as theconcept of dynamic delinquency.

In operational terms, there is an attributecredit limit that serves as part of thecontext for one or more differentiatedservices – it helps to determine whethera transaction will be approved, and whatactions will be taken on a missedpayment. We can use web services andreal-time business intelligence to supportdynamic delinquency; and this requirescredit limit to be represented, not as asimple attribute on the customerdatabase record, but as a web service. Acontextual attribute that would previouslyhave been regarded as basic (becauseentered by an authorized user on ascreen) can now be regarded as derived.

Does the derivation always need tobe calculated in real-time? In manysituations, real-time calculation will beneither technically feasible, norcommercially necessary. There are manydifferent ways that the credit limit webservice can be implemented – forexample, using an observer pattern.Indeed, some implementations of thederivation may still include a manual step– perhaps for borderline or exceptional

Business Intelligence for Real-Time InvestmentA UK investor can access a page on the London Stock Exchange website that

shows (anonymously) the last five transactions for a given stock – including the

number of shares traded and the price.

An active investor can interpret a pattern of transactions in a familiar stock as a

sign that the share price is about to move, and immediately place a trade.

Alternatively, an investor might spot a flurry of activity in an unfamiliar stock, and

place the stock on a watchlist, while initiating some further research.

Furthermore, a professional investor can place a series of small trades in order

to observe the effect on the shareprice. A company planning a hostile take-over

may quietly purchase a certain proportion of the shares before announcing its

intentions. Other investors will be looking to detect and interpret these actions.

Many investors set automatic trades, which are triggered by simple price

movements. But it is now feasible to use web services to build real-time

investment systems where trades (including derivatives) are driven by much more

complex conditions. Typically, investment decisions are based on a complex

combination of information from different sources. There is a clear advantage for

investors who can quickly assimilate new and relevant sources of information –

and this generates a strong competitive edge for web-service-enabled providers

and consumers of financial information.

For example, there is growing pressure on large companies to provide faster

external access to financial data. A company might publish the latest sales figures

or net asset values in XML on its website, to be processed automatically by

institutional investors. (Continuous information reduces surprise, reduces risk,

therefore increases the investment return for the investor and reduces the cost of

capital for the company.) Ultimately, such published figures are driven by the

company’s internal data, and could in principle be released in near real-time,

subject to appropriate (automated) policies for filtering and delay. Thus the

Real-Time Enterprise extends real-time business intelligence to its shareholders

and other stakeholders.

Box 3: Example: Real-Time Investment

Business Intelligence for Real-Time Price AdjustmentA major retailer continuously checks the prices at its main competitors. When a

competitor introduces a special price or promotion, we can rapidly detect the

effect on our sales of that item. This effect varies with the proximity of the

competitor’s store to our store. We can then instantly introduce a matching price/

promotion only in those stores that are affected by the competitor.

Box 4: Example: Real-Time Price Adjustment


cases. Part of the overall architecture forthe business solution will determine therequired service level of this derivedattribute in terms of how much latency istolerable in order to meet the businessobjectives. Provided this service level isachieved, the choice between differentimplementation strategies is not visibleto the consumer of the service.

Random VariationDifferentiated services can also be basedon random policy variation. For example,a random selection of customers is offereda special discount. Business intelligencetools may then be used to analyze theresponse to this policy variation, tocalculate the effectiveness of differentpolicies.

PerformanceOne of the traditional reasons forseparating operational data systemsfrom business intelligence systems wasthat the performance characteristics ofOLAP applications were different – evenincompatible.

SummaryThe Business Intelligence space is beingradically challenged by new forms ofcomputing, including web services.Some of the BI vendors are now offeringweb service solutions that allow BIcontent to be integrated with otherinformation, or fed into other applications.This creates exciting opportunities forbuilding BI into the service-orientedarchitecture, and conversely for using theservice-oriented architecture to short-circuit the labor of traditional BI solutionssuch as data warehousing.

Business Intelligence already uses manyexternal sources of information, and maytherefore be more receptive to the use ofexternal web services where this provideseconomic or technical advantages. Wetherefore expect the convergence of BIand web services to become mainstreamin a fairly short timeframe – possiblyearlier than the use of web services forpure transaction processing.

Richard Veryard [email protected]

Vendor

Actuate

BusinessObjects

CrystalDecisions

Cognos

Microsoft

Microstrategy

SPSS

Product

Adaptive Reports

Data Integrator

XML Report WebServices

Cognos WebServices SDK

SQL ServerReporting Services(SSRS)

Transactor

SmartViewerweb server

Description/Comment

Allows customization of analytic content.

Claims to be the industry’s first real-timeand batch ETL (extract, transform, load)platform.

Creates standalone BI web service.Or can link with .NET applicationsdeveloped using Visual Studio

Allows Cognos application content tobe integrated into portal or otherenterprise applications.

Offers hub based on .NET.

Supports closed-loop businessintelligence

Allows SPSS content to be viewedthrough a browser.

URL

http://www.actuate.com/

http://www.crystaldecisions.com

http://www.cognos.com/products/web_services/

http://www.microsoft.com/solutions/bi/

http://www.microstrategy.com/solutions/Bicomponents/Web_Services.asp

http://www.spss.com/

This issue has not entirely disappeared,although there are now more ways ofreconciling the conflicting demands ofthe different applications.

Security and PrivacyWhen business intelligence is combinedwith web services, a complex array ofsecurity issues emerges.

1 The security of the underlying data –for example customer transactiondata. A web service solution mayinvolve intercepting and analyzing XMLdocuments. It may be appropriate tosegment the XML document, so thatcertain analytic applications (togetherwith BI users) can access andanalyze customer transaction data,but without having access to thecustomer’s identity.

2 The confidentiality of the analysis. Acompany may wish to analyse datafrom many sources, without itscompetitors knowing what analyses itis carrying out.

Vendor Summary

CBDI Raison d’etreWe aim to provide unique insight on component and web service technologies andprocesses for the software industry and its customers. To provide high quality analysisand other information resources on best practice in business software creation, reuseand management. To maintain the highest level of independence.

Modus OperandiCBDI has three channels:

- Subscription services - provision of continuous commentary and information.

- Workshops and seminars - providing indepth briefing and guidance on advancedarchitectures, processes and practices.

- Consulting - including related product management and marketing advice,application audit and guidance, technical and business evaluation for investment

How we compare with othersWe are not a mass market, media oriented organization. All material published by theforum is unique. The majority of material is published by our own analysts, orcommissioned from others, and under strict editorial control to ensure accuracy. Werigorously exclude spurious marketing.

We aim to provide depth insight which covers a narrow topic footprint in a deeper waythan the other analysts, and in particular cover not just the technology, but also thearchitectures, usage, practices and processes.

Also we are unusual as analysts we do not simply echo what the vendors say, we area think tank, identifying new ideas, opportunities and providing stimulus for thinking.We try to be thought leaders providing ideas generation and a rich source ofconceptual thinking based on practical, real world feedback.

Who reads CBDI Journal?Technical and Application Architects, Business analysts, Consultants, CTO’s,Designers, Product strategists, Senior Developers, Project Managers etc . Subscriptionis split 40% USA, 50% Europe.

Contact us: For further information on any of our services contact us at: [email protected] or+353 2838073.

Insight for Web Service &Software Component Practice

IMPORTANT NOTICE: The information available in this publication is given in good faith and is

believed to be reliable. CBDI Forum Limited expressly excludes any representation or warranty

(express or implied ) about the suitability of materials in this publication for your purposes and

excludes to the fullest extent possible any liability in contract, tort or howsoever for

implementation of, or reliance upon, the information contained in this publication. All trademarks

and copyrights are recognised and acknowledged.

Subscribe to theCBDI JournalThe CBDI Journal is

published monthly. In

addition to the CBDI

Journal, subscription

includes electronic access

to all back numbers that

now represent a significant

resource library. There are

individual and corporate

subscriptions schemes.

Corporate subscription

includes access to our

workshop materials.

For more details and to

subscribe see:

www.cbdiforum.com


june 2003 - cbdi

Documents