Operational Business Intelligence for Agile Enterprises

By Kishore Jethanandani

VENDOR STRATEGIES AND PRODUCT OFFERINGS

Agile enterprises need business intelligence to improve their sensory perceptions; the ability to

gauge reality, to view their resource flows and alacrity to remain on top of events. Business

intelligence vendors are increasingly conscious that customers hate to be hobbled by their sunk

costs in Information Technology. Instead, customers want to be able to rejig their existing

technologies to adapt in fluid situations. Information needs to flow unimpeded by clunky

technologies. Technology inadvertently has often, in the past, become a millstone instead of a

lubricant of change. Customers are increasingly looking for technologies that read the pulse of

their business activity and funnel information to all their employees who can communicate and

collaborate and act in time to respond to events.

The concerted effort that customers are making to lower latencies in data collection and decision

making is best illustrated by the effort investment banks are making to speed up the processes to

refresh their data that influences their decisions on portfolio management. They are looking to

receive information directly from the exchanges, option trading exchanges and ECNs so that they

can weigh the impact of events on any of the securities that they hold in their portfolios. Automatic

trading tools enable traders to complete the calculus of risk and return when they buy or sell

securities and need to evaluate the impact of major changes such as prices and interest rates.

Their information architecture has to be so constructed that it can tap market data from a variety

of servers, with their own data formats, and convert them into a single format and move the data,

aided by middleware, to the enterprise data infrastructure.

In the past, information systems were tenuously linked to the levers that companies could use to

act as situations changed. Increasingly, enterprises are looking to integrate decision making

processes and business processes so that the lag time between the receipt of information and

the response is minimized. They want to build in the rules for predictable responses to known

problems so that human resources can be reallocated to attend to more knotty problems. Where

human intervention is required, companies want to be able to quickly visualize a situation and

size up a problem before they act.

Above all, the best decisions happen when all the related information is brought to bear on a

course of action. In addition, decision-makers want to simulate alternative scenarios, visualize

them before they take their decisions.

The business intelligence industry is still evolving and the jury is still out on who is eventually

going to win. Clearly, established players with experience in implementing large size deals in the

enterprise software industry have the best chance to integrate several technologies required to

gather information, analyze data and communicate decisions. Business intelligence projects also

require consulting services that are so necessary for successful implementations of especially the

operational applications.

INTELLIGENCE ON TAP

Business managers, in operational situations, get the “missed the bus” feeling when they are

unable to take decisions at the right time; their calculations can go haywire as the ground shifts

underneath them. Well timed moves help them to grab opportunities and get ahead of the

competitors. Companies need to also take decisions before a problem snowballs into a crisis.

Delays in decision making can cause grievous losses to businesses. In the pharmaceutical

industry, for example, counterfeiting, medical errors and poor quality of products can undermine

confidence in companies and medical groups. The actual manufacturers could be blamed for the

harm done by counterfeit or an odd batch of poorly manufactured products causes harm to

patients. The healthcare industry is now well equipped with laser vision, RFID and other

technologies to eliminate errors in data entry and data gathering technologies which aggregate

data so that causes of any damage done can be traced back to specific deliveries and preemptive

action is taken.

Edge Dynamics Inc is one of the companies with applications for real time decision making for the

pharmaceutical industry. Typically, a complex set of contracts, deals and regulatory policy bind

the stakeholders in the supply chain consisting of the manufacturers, wholesalers and retailers.

Edge Dynamic has the software which is able to capture transaction order stream data originating

from an EDI or other such sources such as a Web-service B2B network. The data is analyzed for

discrepancies from forecasted numbers or any deviations from agreements initiated at the outset.

As data is received and analyzed, the partners in the supply chain will discover flaws in its design

and work towards managing inventories better. They will find better ways to optimize and

reevaluate their partners and their logistical planning.

The technology that goes into gaining visibility into the operations of businesses is illustrated by

the implementation of Siebel Analytics at Jostens which sells class rings, graduation

announcements, and yearbooks to schools across the country. Siebel Analytics is able to

aggregate data from the Oracle data warehouse, a Microsoft SQL Web/e-commerce application,

and Microsoft Access for use by Jostens sales staff which can view the results on role-based,

interactive dashboards. The Siebel Analytics software enables Jostens to keep track of the sales

performance data by each segment of the business. With real time data feeds, Jostens’ sales

staff can spot opportunities for cross-selling, up-selling, etc.

Retail stores present a familiar scenario where markdowns happen almost everyday when

inventories pile up unexpectedly. All too often, retail store managements are taken by surprise as

preferences change, the media influences attitudes, new promotions are announced by

competitors, seasons change or events affect purchases by consumers. At the local level,

consumer behavior can be quirky and the inventory in stock may not excite them. Retail stores

have to learn to stock an assortment of products that are in tune with the tastes of customers for

each of their stores spread around the country, ensure that they will be profitable and manage the

supply chain so that the products will available in time for the season.

When supply closely matches demand, companies can not only pass the benefits of lower losses

from stock outs to consumers in the form of lower prices but also offer products that closely

match their needs. Zara, a Spanish clothing company, takes less time than its competitors to

respond to market need. The managers at its stores send information about customer

preferences by handheld devices and it is all aggregated rapidly so that the most relevant

products are displayed in their stores. The dyeing and printing is done only after the customer

information is available.

The management of demand and supply has gotten more difficult as the product life cycles get

shorter and the supply chains get longer as goods are sourced from more distant places.

Increasingly, companies are looking at software that can aggregate point-of-sale data from

multiple sources, analyze it to predict demand for individual categories of products, optimize the

supply chain and help in pricing.

Several different pieces of software are used in the management of demand and supply. One of

them is revenue optimization software which takes into account information on demand, the costs

and determines the best price to offer based on elasticity of demand. Conversely, it can take the

prices offered by competitors as given and throw up the numbers for the desired demand and

supply. Such software can also pinpoint customer segments most likely to respond to a particular

offer. Manugistics Inc. is one company that leads in this segment of the market. However,

revenue optimization software does takes into account existing demand before it cranks out

figures on prices and potential segments to target.

The suppliers would rather that they could forecast demand accurately and produce as much so

that they can receive better price deals. This is best achieved by using demand forecasting

software. The successful implementation of demand forecasting tools presupposes the collection

of point-of-sales data and the willingness of retailers to share such information with their vendors.

Besides supply chain management software providers such as i2 and Manugistics, Business

Intelligence vendors such as NCR Teradata, Business Objects, Cognos, and Prescient are the

players in the segment.

An additional piece of software for collaborating in real time is the software for supply chain

management to collaborate with vendors, manage logistics and to share information. Oracle's 11i

E-Business Suite, for example, includes iSupplier and Collaborative Planning portal to

communicate with offshore contract manufacturers and suppliers with web based tools.

MACHINE LEARNING FOR INSIGHTS

The growing size of data sets has changed the analytical paradigm. Well known techniques, such

as statistical techniques, are overwhelmed by the colossal volumes of data. Typically, statistical

techniques begin with a hypothesis and a model, based on domain knowledge such as

psychology, which they seek to validate. The involvement of human beings and uncertain

processes preclude the use of insights in real time.

When data sets are large and chaotic, it is much harder to decide on the methodology for

verification. The dimensions overwhelm human cognition’s ability to see the connections and be

able to this quickly enough to make decisions. Increasingly, machine learning methods are

required to reduce raw data into patterns before humans can look for the story that is relevant for

decision making purposes. These automated methods of finding patterns look for correlations of

data over periods of time (time series), find clusters in activities such as crime or classify data

such as in decision trees. Market basket analysis, for example, looks for combinations of products

customers tend to buy.

The kind of situations in which machine learning has a compelling value is searches on the web.

Intelligence agencies have to look for terrorist activity, competitive intelligence analysts look for

information on rivals or content creators have to look for intellectual property rights violations.

Companies, such as FAST, have created tools that are able to extract insights from such a

labyrinth. Reuters, for example, uses FAST’s search tool to zero down on content that looks

suspiciously like its own.

Machine learning plays an important role in functions such as fraud detection, stock trading, and

customer segmentation to extract intelligence that cannot wait for an analyst to extract

intelligence. Neural Network software systems, for example, have reduced fraud in UK banks by

as much as 30%.

The vendors in the space include SAS, STATISTICA Data Miner, S-Plus, Fair Isaac, SPSS

Clementine, IBM Intelligent Miner Affinium Model, Insightful Miner and KXEN, IBM Intelligent

Miner and Genelytics.

Tools that understand fuzzy concepts

Companies are best able to extract insights when they can search across all their data and

classify and correlate it. For decision support knowledge, companies have to be able to conduct

searches on both structured and unstructured information.

The urgency to search for unstructured information is more urgent now as it has wide range of

applications such as especially law enforcement, customer service, drug discovery, knowledge

management, etc. Companies are beginning to discover the enormous benefits of mining text and

other unstructured information. The pharmaceutical industry, for example, is discovering that it

can reduce the time required to commercialize new drugs if only it could search and analyze

information pouring in from clinical trials for all drugs. When data on safety is available for all

clinical trials, the regulatory bodies can look for patterns that will help them to come to decisions

about accepting drugs for human use faster than is the case now. XML is the bedrock for linking

related databases and to search them with text mining tools.

Customers need a common language and search tools to parse all the relevant data and to

analyze it. Natural language is best able to express the nuances in human thought processes.

Inevitably, natural language can have a variety of meanings, synonyms, connotations and usage.

New tools are required to see words in their context before any meaning can be drawn from

them. This is best achieved when search engines have semantics capability.

The traditional and the most widely used method of searching databases, the Structured Query

Language, is inadequate for heterogeneous environments where data descriptions vary between

databases. This form of querying is relevant only for structured data and it presumes knowledge

of the specific information a person is searching. In most cases, people have a knowledge of the

theme they are interested in exploring. In heterogeneous environments, searching by using SQL

would be impossible since the number of series, as well its heterogeneity, overwhelms its ability

to extract meaningful information and knowledge.

With the advent of XML technologies it is now possible to classify unstructured information as

well. Individual elements of unstructured information can be described by tags or the metadata

that describes the information content in there. The detailed description of the content helps to

search repositories with large volumes of content much like SQL queries can extract information

from relational databases. XQuery can search both content repositories and databases and

extracted related quantitative and qualitative information. Microsoft’s SQL Server 2000 is one

product which supports XQuery and is able to use both structured and unstructured data for

analytical purposes.

Another approach to searching unstructured data is to use search engines. However, a search

conducted on unstructured data all too often yields a jumble of results which is an all too familiar

experience of users of the World Wide Web. Similar searches on corporate intranets are worse

since the information in not even linked as is the case with the World Wide Web.

Search technologies for corporate databases seek to look for the significance in a mass of words.

For example, someone maybe looking for information on a crime committed by a suspect named

John Lear of San Francisco will be able to find meaningful information when inter-related

information about the background of the person, the time, location, previous associations with the

victim is presented. Databases of unstructured information can have variables such as time,

location, biographical data, etc., as the dimensions of a data warehouse and they can store

related facts associated with each of them. A search conducted on such databases is more likely

to find related results instead of a jumble.

At the center of semantic search technologies is ontology or the knowledge base that helps to

define the “being” or the personae that are pivotal to understanding the universe under

consideration. For example, students and professors form the axis of the universe of an

educational institution. Semantic search tools create taxonomy to describe the entities in a

universe and their relationships with the world around them. The information about the university

is classified by entities which helps to create the links between the available records.

Search engines for unstructured data are now able to find information in an organized way by

using tokenization, linking and taxonomies. In essence, these methods look for patterns in the

unstructured data. The tools are so designed that they look for associated text; a word like crime

is related to gang membership, academic performance of the person, incidents of drug or alcohol

abuse, etc., and the information is presented in its relevant context.

The impact of correlating structured and unstructured data can be easily visualized if we look at

the decision analysis that is required for store location analysis. Typically, retail companies will

need structured data such as the demographics of the neighborhood. They will need also map

information in the form of satellite imagery. Also, they would like to have unstructured information,

such as crime, to gauge the attractiveness of the location besides lifestyle trends in the region.

One example of the use of intelligent search engines is the case of ISYS search engine from

Odyssey Development. The Ventura County in California searches through its numerous

repositories to find related information. It could, for example, use data of blood examination, from

a structured database, and find related information on several burglaries committed by the same

individual from several other repositories.

One of the several semantic search tools in the market has been created by Semagix for

searching media sources. The ontology is a hierarchy of categories beginning with general

classifications like News, Business and Entertainment and then more specific terms like cricket,

soccer, etc. The searches could be done by themes such as cricket tournaments which exclude

the possibility of tangential information, such as tournaments of all sports, appearing.

IBM is one company bringing a great deal of intellectual property to the table for searching

unstructured data. IBM WebSphere Information Integrator OmniFind Edition has pushed the

envelope by launching its Unstructured Information Management Architecture (UIMA), a platform

for integrating structured data and unstructured information. The platform supports a variety of

functions such as linking analytics software and enterprise applications, tools for developers to

conveniently create new or reusable text analytics components. With this architecture,

unstructured data in a host of formats or languages, whether it is located in databases, e-mail

files, audio recordings, pictures or video images can be searched.

The searches are unlike the familiar keyword searches; they use concepts to look for related

pieces of information. Text analytic components, supported by UIMA, can use WebSphere

Information Integrator OmniFind Edition to define the ontology, look for relationships in data, mine

text to find hidden knowledge and extract useful business information. An example of how these

kinds of search engines can look for inter-related information would be the case of customer

satisfaction; it would be possible to search for data in maintenance records, market research

studies, call center records and warranty claims to find the products that customers find most

satisfactory or vice versa.

Altogether a total of fifteen companies plan to use this architecture and they include Attensity,

SPSS, Endeca, Factiva, Kana ClearForest, Cognos, and SAS. Factiva and QL2 will provide data

for analysis.

They get it with visuals

Decision-makers are constantly intimidated by information clutter and are looking for tools to help

them digest information rapidly. There is a great deal of noise in large volumes of information

while the noteworthy nugget could well elude the decision makers. In industries such as the

securities industries, the value of information decays quickly unless the substance is absorbed

quickly.

Visualization is an indispensable tool for real time assimilation of relationships in large volumes of

data and their implications for decision making. One instance of this is American Water which has

to monitor the threat of a hostile intrusion on its IT network. It receives thousands of alerts and the

large majority of them are false alarms. Visualization tools help it to map the source address of a

packet and its destination to help isolate any suspicious activity.

Decision makers prefer interactive visualization tools to help them test their hypothesis visually.

Excel type of static graphics have been the staple for visualization in enterprises. Decision

makers need to be able to examine alternative scenarios and they like to have visuals that are

three-dimensional, pliable enough for impromptu reconfiguration to respond quickly to queries

and they like to flip them to view a problem from a variety of angles. The visuals are made lifelike

by the use of artifacts, colors and animations to convey the meaning of the information displayed.

All of these attributes are meant to contribute to effective communication of a message. None of

the widely available visuals available with spreadsheets have the capability to achieve this.

Visual queries are one of the means to isolate relevant data from a clutter and portray it visually.

Much like the structured query language, a visual query extracts specific pieces of information

from a mass of relational database and displays it on a graph. An alternative way to zero down on

selected information is by the choice of dimensions; an analyst might want to compare the bad

debt losses by regions, such as mid-west and the west coast, which is possible when a cube is

created. Cognos Visualizer, which works in combination with Cognos Powerplay for aggregation

of data from multiple sources, is one product that enables users to choose their dimensions and

the corresponding numbers they want to display graphically.

The vendors in this field include the Business Intelligence vendors and another group is

specialists with a focus on visualization. Among the leading BI vendors are Cognos, Business

Object and SAS. On the other hand, the specialists are companies like Vizible Corporation, Visual

Mining and visualization platform providers such as Antarctica System’s Visual Net and Spotfire’s

DecisionSite. The platform providers are the most versatile as they are designed to use data from

any source and they can customize analytical tools to conduct the desired kind of visualization.

Typically, the platform providers focus on industries that generate enormous quantities such as

the pharmaceutical industry or the natural resources industry.

Spreadsheets are forever

An aspect of real time access to data for decision making is also the ability of users to have the

option to continue to use familiar tools. Spreadsheets have been the most widely used for

analytics required for decision making purposes. Integration of spreadsheets with business

intelligence software is critical to their widespread adoption in the enterprise.

Excel spreadsheets are ubiquitous in enterprises despite the fact that they inexorably fragment

the data sources. The flexibility of Excel allows users to create their own data marts and they can

add formulas of their own choice. On the other hand, they contribute to fragmentation of data

sources and perceptions which conflicts with the objectives of gaining a consistent view of the

enterprise. Excel spreadsheets are also not scalable and are not the tool of choice when it large

teams have to work together. In addition, Excel spreadsheets do not have the ability to manage

business processes based on the analysis conducted. Above all, Excel spreadsheets do not have

the ability to pull together information from a diverse set of corporate databases.

Business Intelligence vendors have, in the past, provided partial integration with Excel

spreadsheets by adding features for exporting data to Excel sheets which can be loaded on to a

server. However, Excel users routinely create their own formulas to create additional data series

which were not included in the process of integration. One of the exceptions was SRC Software,

recently acquired by Business Objects, which had closely integrated a variety of databases with

an Excel interface.

Lately, business intelligence vendors have changed course and have offered Excel as the

interface to their business performance or business intelligence software. The software provides

a consistent view of the data and uploads the formulas together with information on changes

made by any of the workers in the enterprise so that they all have access to the same

information.

Beyond a patchwork of integration

The issue of integration has gained urgency as companies seek to manage their sprawling supply

chains, collaborate in product development, offer self-service options to customers and outsource

business processes. A common denominator in these applications is that they span several

systems and applications; enterprises need conduits for information to flow across all of them.

In the past, middleware was the accepted way to integrate applications; it was a step forward

from arduous custom coding that was the norm. While custom coding is the recommended

method for joining two applications, middleware simplifies matters when an application has to be

connected to several others. The middleware is the intermediate junction which allows

information to flow to several different directions. On its way, the middleware prepares the data,

the reformatting, merging, etc., which allows it to be accepted in another database or application.

Data transfers, in a message-oriented architecture, are akin to e-mail which remains in a server

queue till it can move through networks and eventually downloaded onto a desktop when desired.

Middleware created a patchwork of joins which grew in numbers and complexity over a period of

time and was increasingly difficult to manage. At this point of time, a need was felt for a platform

that would manage the gamut of middleware and associated software such as business process

management, be managed from a single platform. Enterprise Application Integration (EAI) suites

meet these needs. An EAI network has subscribing applications which replicate data received by

any one of them into another application.

Integration of applications is not limited to specialized EAI vendors such as Tibco, WebMethods

and SeeBeyond. These vendors have strengths in integrating several applications into a

continuous process but are less likely to be able to combine a broad range of complex business

processes. Another group of vendors are the large enterprise software companies offering

platforms such as IBM, Microsoft, BEA. Finally, the application vendors are also offering a series

of integrated applications such as SAP’s Netweaver, Siebel’s Universal Application Network

(UAN).

In the early stages of integration, message-oriented EAI software was most widely used for

transportation of data from one end to another. In a typical message oriented integration

technology, data can be transferred in an asynchronous manner so that current applications don’t

have to interrupt their functions to receive new data feeds. An intermediate middleware received

the data feeds and transmits to another application without the need to alter the two applications

in any way. The EAI network facilitates the navigation of data over a variety of networks,

programming languages and applications. When two applications are integrated, a point-to-point

integration will suffice. On the other hand, a publish/subscribe methodology will work better when

data transfers have to be undertaken to multiple applications.

One case study of the implementation of the EAI is Cincinnati-based HealthBridge which acts as

the junction for flows of information from hospitals and providers in the metropolitan area of

Cincinnati. The company navigates the data streams from 28 hospital clinical applications and

delivers more than 940,000 clinical results per month to 2,900 physicians. HealthBridge network

uses clinical messaging software to transmit data from one application in a hospital to another in

a physician’s office.

EAI makes a departure from piecemeal integration of some of the applications of an enterprise by

tying them with middleware. While integration by means of middleware is much less expensive

than an EAI implementation, the benefits too are far more limited. By integrating the entire

enterprise, an EAI also enables companies to view their business processes and applications in

their entirety. The process parameters are monitored so that companies can monitor their

performance. They can optimize their business processes by modeling them, simulating the

impact of alternative designs which could help to lower costs and redesigning their business

processes.

The IBM MQ series is the market leader in the space for message oriented integration

technologies with an estimated 65% of the market share. The leading position of IBM is

accounted for by open architecture which means that applications in diverse range of platforms

can be integrated without any special programming. On the other hand, Microsoft, the other major

player in the market, does not support any other platform other than its own without custom code

to make it happen. IBM’s WebSphere MQ connects applications and passes messages between

them. It has a library of connectors to Oracle, SAP, and Siebel Systems applications, as well as

mainframe systems such as CICS and IMS. BEA Systems offers WebLogic Integration and uses

XML based application adaptors to inter-link applications.

The other method of integration is to execute it at the level of data. This method relies on

database technologies like gateways, metadata, queries, data set transfers, and bulk data loading

tools such as ETL or data grids. The most commonly used methods are ETL tools, traditionally

used to feed data into Data Warehouses, are products like Data Junction's Integration Studio and

Engine, Informatica's PowerCenter. These tools extract data from operational data stores,

transform the data and load them into data warehouses.

Increasingly, the former specialists in ETL have morphed into data integration specialists.

Informatica, for example, has incorporated PowerAnalyzer into its data integration product

PowerCenter Advanced Edition and is working closely with Composite Software for EII type of

integration. IBM extended its data integration capability by its acquisition of Ascential and has

incorporated its ETL engine, renamed DataStage TX, which also includes EAI capabilities that

came with the acquisition of Mercator Software Inc. The IBM WebSphere Information Integrator

plays the role of data integration.

Integration of data sources is crucial to not only consolidate data but also to check for its quality.

This is illustrated by the case of The Scotts Company, Marysville, Ohio, the lawn and garden

products company, which needed to find a way to forecast consumer demand with greater

accuracy. In the past, it had to depend on its own shipment data instead of the point of sales data

which was scattered and could only be accessed from EDI systems that were routinely used by

retail chains. The data integration major, Ascential, designed a solution that allows the company

to access consolidated data from its POS of sales services in a way that could be read by its SAP

applications. A comparison of the actual shipment, production and inventory data with demand

helps to determine the trends in consumer demand and to make relevant adjustments.

Enterprise Information Integration (EII) is a set of tools which integrate a federation of data

sources besides applications. EII provides a single point access to all the data in the enterprise,

whatever its format, with metadata to describe all the data. The better known products in this

space include BEA's Liquid Data and IBM's DB2 Integration Integrator. Among the new

companies are Attunity, Avaki, Composite, and MetaMatrix. Composite has an alliance with

Cognos to integrate business intelligence software and its product Composite Information Server

3.0 starts at a price of $100,000. Avaki’s first product Avaki 6.0 was launched at $50,000 while

the total costs of deployment averages $175,000 to $250,000.

These systems have two important components; they need a data model to aid the process of

conversion of data from one source to another. These tools also provide graphical tools that show

the configuration of the network of applications and data sources and a directory of terms to

access methods and fields in the data sources.

Among the key players are start-ups such as MetaMatrix which has a partnership with Business

Objects and Hyperion while Composite Software has alliances with Informatica and Cognos. Both

these vendors come from a background in relational databases and their products afford SQL.

Another category of vendors, such as Ipedo, provide XML based query techniques. Composite

Software enables the aggregation of data into a portal view while any kind of manipulation of this

data has to be done manually by the user.

A much desired integration method is to orchestrate business processes in order to enhance the

ability to control the levers that will help to respond quickly to changes in the business

environment. The ability to manipulate business processes puts business executives in control of

IT and they can direct enterprise resources independent of the IT department. The need to

automate business processes had been alluded to by vendors in the content management and

workflow management as well as in the EAI space. However, the management of business

processes has been piecemeal so far and has not progressed to a level where all them can be

managed from a single platform of its own independent of other divisions.

A single platform, for business process management, provides the means to adapt business

processes to changing requirements rather than be set in an application stone. One of the key

barriers to configuring a series of business processes is that they have always been embedded in

applications. Once they are decoupled from applications, business processes can be broken up

into their components, reused for a variety of tasks and connected to complete a series of tasks

to complete a job at hand. The Business Process Modeling Language (BPML) provides the

means to create a path for the flow of business processes. For execution purposes, the Business

Process Execution Language (BPEL) plays a complementary role in that it manages the flow of

business processes. The conceptual bedrock of an independent management of business

processes is Pi Calculus which provides a method for unifying them and reuniting them for

another purpose. Just like properties define an object, Pi Calculus is like metadata which spells

out the tasks an individual unit of a business process can complete and the roles it expects

related processes to complete. In such a world, business processes are akin to packets in a

network which can be made to follow different routes depending on the addresses where they are

directed to move towards.

Intalio is one of the pioneers in the design of platforms for the management of business

processes in their own right rather than as a component of an application. Other notable products

in the same space are Microsoft BizTalk server and Holosofx which was acquired by IBM was

renamed as Business Integration Modeler and incorporated into its Websphere platform.

Siebel’s Universal Application Network (UAN) creates a process centric environment for the

integration of applications and data in an enterprise. At the heart of this strategy is a library of

business processes such as quote to cash, campaign to lead, order to pay, etc. which can be

used to compose solutions. An overarching SOA architecture enables other vendors to tie their

applications in the overall solution. UAN incorporates vendor-neutral interfaces based on SOAP

(simple object access protocol), WSDL (Web service definition language) and XML (extensible

markup language) to create an environment for a diverse range of vendors to plug in their

applications. The UAN has used the syntax of the BPEL4WS (Business Process Execution

Language for Web Services) standard to define all business processes which implies that

individual vendors can hook in their own servers.

Lately, another variety of data integration software for interlinking data sources across a grid has

appeared. This kind of software, such as the Avid Data Grid, spans a wide area network and

allows access to the resources available across enterprises without going through the

cumbersome process of using multiple passwords to access them. The data access is made

possible by a universal directory which provides the access to the data source. With just this one

access point, users are able to extract data from numerous sources and are delivered for specific

users.

Some of the key players in the grid computing industry are HP’s Adaptive Enterprise, IBM’s On

Demand and Oracle’s 10G and Sun’s NI. IBM’s On Demand program uses its WebSphere and

Tivoli products for policy-based management. HP has a similar product called OpenView Platform

for the management of the grid and integrates Talking Blocks Web services into it.

The key advantage of Grids is the possibility of lowering latencies afforded by a cluster of servers

and storage devices which are not clogged when traffic spikes unexpectedly which would tend to

happen when numerous applications have to be operated simultaneously. An array of storage

and servers helps to spread the load over several devices which also are better utilized because

they are not dedicated to specific applications. Charles Schwab was able to lower to lower query

response from four minutes to as low as fifteen seconds.

One of the earliest applications of Grid computing is the case of Hewitt Associates, a global HR

company, which has deployed an IBM WebSphere-based grid to operate software to calculate

pensions.

Corporate radars

The payoff of integration of information assets in an enterprise is the ability to monitor any sign of

a threat or an opportunity so that enterprises can take timely actions. Business Activity Monitoring

servers play the role of radar which can spot any exceptional events, such as missed schedules

in transportation, followed by the analysis of the impact of untoward events on related activities

such as communicating to a truck which could use its spare capacity to take on a load that the

assigned truck missed. Business Activity Monitoring involves a series of related activities of

observing business processes, comparing them with metrics, analyzing and visualizing its

implications and communicating for corrective action.

One of the applications of real time activity monitoring is the case of Brocade communications

which needed to keep track of the performance of its contract manufacturers to make decisions

about its outsourcing decisions. It acquired a business activity monitoring tool to be able to do this

in real time.

Some of the leading players in the industry are Informatica which has incorporated its Business

Activity Platform, developed jointly with WebMethods, in its PowerCenter RT, its integration

platform. WebMethods Application Integration capabilities and Informatica’s information

integration have been combined while they have, at best, lightweight business process

management capabilities. Ascential Software’s DataStage, now integrated with IBM WebSphere

Datastage has a Real Time Integration (RTI) Services component.

There are also companies from the business process domain, such as Microsoft BizTalk Server

2004, which has added a Business Activity Monitoring engine. BizTalk Server 2004 allows users

of Microsoft Office 2003 to monitor business processes from a desktop. Middleware specialist

TIBCO offers BusinessFactor, the technology it acquired from Praja. Celequest is among the

more prominent pure-play companies in the domain with its ActivityServer suite which has the

ability to stream data from operational systems and compares business rules with metrics to

determine whether an alert needs to be sent. Among the database companies, Teradata with its

Active Data warehousing product is focused on business activity monitoring.

A nose for bad data

In its early stages of growth, the data quality industry was largely populated by independent

players. In more recent years, the independent players have been bought over by the large

business intelligence companies. A prominent example of this is the acquisition of DataFlux by

SAS, Ascential bought Vality and was in turn bought by IBM. FirstLogic, an innovator in the

space, was bought by Pitney Bowes. Group 1, a data quality vendor, took a different course and

acquired an ETL vendor, Sagent, before Pitney Bowes purchased it. The process of extracting,

transforming and loading is expensive and companies hope to lower their costs by merging the

associated routines of validation, transformation, filtering or standardization. These processes are

optimal way to improving data quality when companies want to do data mining in a data

warehouse environment while much more needs to be achieved in a real time environment.

An example of the functionality available with such products that offer both data consolidation and

data quality services is IBM’s WebSphere Data Integration Suite. Its ProfileStage component

automates the process of matching the data structures and data formats, from different

databases, so that the data from the source and the target are consistent. Similarly, the

QualityStage component standardizes data for individual entities, such as a customer, and

ensures that disparate conventions in storing data don’t contribute to inconsistencies. The data

quality components are provided with the DataStage, an ETL tool.

When data is updated in a real time environment, using technologies such as messaging queues,

there is a need to execute data quality functions early on in the transactional databases or by

other means such as metadata. The automation of data quality functions presupposes a

comprehensive solution to provide universal definitions of data and a means to convert from one

definition applicable to a specific application to another, removing duplications and correcting

errors in addresses, names, etc. Master data management is a means by which a database of

metadata is created and rules for converting from one format to another.

A key problem with current methodologies is that the data cleansing is done after the data has

already been extracted so that the source of the error is not detected. When data reconciliation

takes place with the help of master data management systems, the source of the error is also

identified.

It is now possible to buy rudimentary master data management products from a variety of

vendors. Among the platform vendors, the leaders are IBM and HP, PeopleSoft and SAP are the

players in the applications vendor category and Ascential, Infomatica among integration vendors

besides system integrators and Hyperion from the BI category.

An example of the implementation of a master data management system is the case of Unilever

which needed a centralized way of managing its data to pursue a global policy for brand

management and supply chain management. Historically, Unilever followed a decentralized policy

for the management of its subsidiaries which meant that its IT system was fragmented. It has now

implemented a master data repository which helps to align its transactional systems for a

consistent view of its data.

Sensors everywhere

Automation of data collection is one of the means to collect data free from errors. Sensors also

help companies gain visibility into their environment and to expand the universe of problems they

are able to address. The volumes grow with the use of sensors and present new challenges in

data processing. Increasingly, RFIDs and other type of sensors are available for commercial

application. For mass adoption, on the other hand, the costs of RFIDs would have to be

substantially lower before they will be accepted in applications such as supply chain

management.

General Electric has been one of the early pioneers in the use of sensors for analytical purposes.

These sensors gather data on the state of health of its jet engines. The data is gathered at one

place where analytical software looks for signs of trouble and provide early warning to their

customers. The military has often been a leader in adoption in new technologies and its

willingness to accept the RFID technology is a pointer to wider diffusion in industry at large. Cost

factors are less binding in situations where high value activities, such as the manufacture of

aircraft engines, is involved or in the military where security is an overriding consideration.

The use of RFIDs in mass applications such as supply chain management is going to bring it

closer to ubiquity. In the early days, the industry had to improve the technology, the ability to read

information, so that it would work in commercial environments. Wal-mart is working with a

hundred of its partners to expand the use of RFIDs for supply chain management. NEC of Japan

has reported early successes in the use of RFID in its PC assembling plants. Unlike bar codes,

RFIDs do not require manual scanning of bar codes; the productivity at its Yonezawa Plant, as a

result, increased by 10% besides the benefits of just-in-time replenishment of inventories.

It will take middleware or some other form of integration technologies for companies to be able to

receive information from a variety of sources and funnel it to a central database. One of the

significant initiatives, to popularize the use of RFIDs, is the partnership between Oracle and Intel

and Xpaseo to supply tools for the management of information received from sensors. These

tools will mediate the flow of information from sensors and integrate with existing products from

Oracle and Intel, including Oracle Application Server 10g, Oracle Database 10g and Oracle E-

Business Suite. In addition, the partnership will extend the scope of pervasive computing by

linking data from other devices such as hand held devices, PC, mobile devices using Intel

communication and server platforms.

Other companies who have RFID offerings include SAP, IBM and SUN. There are also smaller

specialist companies who offer middleware to integrate RFID technology with the rest of the

enterprise software.

THE BIG PICTURE

Real time enterprise has to put in place several moving parts before the entire vehicle for rapid

response takes shape. Vendors have been able to offer most of the components of the solutions

of an adaptive enterprise and are acquiring companies to consolidate their products to provide a

complete package. Some remarkable breakthroughs have already been achieved especially in

the analysis of data. Real time interpretation of data, aided by machine learning techniques, and

predictive analytics capabilities has equipped enterprises to grasp the dimensions of the

problems they encounter and to anticipate the outcomes and consequences. The imminent

prospect of automated data gathering and business process design will trigger as much

excitement as dashboards have done in the recent past. Larger vendors such as Oracle, IBM,

Microsoft, SAS, SAP, Business Objects and Hyperion are emerging stronger than before and are

best equipped to emerge as leaders. In all probability, infrastructure providers from among the

former ERP companies will have the strongest foundations to supply the products and the

consulting skills to win over customers in the future.