THE USES OF PERVASIVE INTELLIGENCE

Business Intelligence is an evolving industry with growing market potential which has attracted a

swarm of players who are serving their customers in several different novel ways. It has grown

from a technology focused on decision support and performance management in some

departments to increasingly ubiquitous tool that spans operations management across the

enterprise. As it incorporates a gamut of functions from business activity monitoring to

performance management and business planning, business intelligence attracts a growing

number of companies who earlier specialized in individual functions. Customers have a daunting

task of choosing from a rich crop of innovative packages; they will have to make a judgment

about the products that will become the standard for the industry in the future.

The growth of the business intelligence industry has been propelled by convergence of several

factors which will continue to fuel rapid expansion and innovation. Regulatory compliance is

inescapable and the need to monitor operations risks is the baseline for additional applications to

reduce costs, fraud detection, accelerated responses to market changes, better targeting of

customers, detection of latent opportunities for product development and a personalized

approach to selling among several other applications.

Keeping an eye on all activities

The process of selection from several different offerings will be influenced by enterprises’

preference for tactical or strategic goals or a combination of them. While decision-support

analysis or performance management in selected functional areas are possible in isolation with

business intelligence tools alone, companies will need a comprehensive strategy, including

business process management, content management and performance management, when they

weave business intelligence into the fabric of their daily business activity for rapid responses to

market changes. Dell, for example, has embedded intelligence into its daily business and is able

to set in motion its business processes all along the supply chain, once a configuration for a

computer is spelt out by a customer, including orders for its contract manufacturers and shipping

companies.

In the past, spreadsheets, OLAP tools, management report generation and query tools were the

preferred tools as they catered to decision-support and performance management in selective

functional areas. Enterprises will need more of dashboards and performance management tools

as they look to monitor and evaluate business activity across the enterprise. They also need to

able to collaborate and communicate rapidly for a coordinated response to contingencies facing

the enterprise based on the information received from performance metrics. In general, they need

to aggregate, analyze and act in near real time.

For their tactical or their operational needs, companies need information on all the metrics that

govern their performance metrics and the parameters that form the co-ordinates of their

decisions. In the past, companies could barely integrate information from a few of their

operational data stores and create standard reports to make decisions quickly; they looked for

tools where they could drill down and view from different angles. Increasingly, enterprises will

need to inter-connect all their sources of data to take actions that are consistent with their

enterprise strategy.

One instance of the use of performance measures is La Suisse Insurance which set targets for

its sales force based on a multi-dimensional view of data. The company was losing up to $50,000

per salesperson each year by paying monthly allowances to salespeople who were

underperforming. An OLAP tool helped in gaining multiple views of sales performance — by

salesperson, branch, and region— which uncovered opportunities for raising productivity. Data

warehouses can extend the capability and include the analysis of impact of prices, advertising,

etc. on the results achieved.

Alternatively, companies can focus on decision-support tools which involve ad hoc queries and

make considerably greater demands on the analytical and data management capabilities of

business intelligence software. This kind of query requires much larger data sets for cross-

referencing across several dimensions of data as well as over time. Companies need to be able

to integrate data from several transaction data bases, usually in a data warehouse environment,

and need the IT horsepower to conduct such complex queries. The typical applications of ad hoc

queries are customer segmentation and response modeling.

Strategic decisions, unlike operational decisions, take a longer time as companies need to be

able to parse current and historical data before they can come to decisions. They involve the use

of statistical and data mining tools to consider alternative scenarios, predict future financial

performance, conduct customer segmentation for product positioning and make decisions about

the choice of their channels. This is typically done with data stored in data warehouses and

updated periodically, typically overnight or on the weekends, in order not to interrupt analysis

during the day.

A growing number of companies favor using business intelligence tools to align their strategy and

tactics. They want to be able to make course corrections to ensure that their tactical initiatives are

in line with their strategic direction including coping with contingencies which could roil their best

laid plans. With the advent of active data warehousing and rule based decision engines, they are

able to compare actual figures with benchmarks to come to decisions about routine decisions

such as inventory replenishment, yield management or pricing decisions. This kind software

allows companies to embed rules that prompt decisions when an expected event happens.

Integration of decision and business process management for linking strategic and tactical

management is rare (not impossible) but immensely beneficial for those who succeed. PSS World

Medical based in Jacksonville, Fla. has a sprawling network of 44 distribution centers selling

medical equipment and supplies. Each of the centers is tied, by a dashboard, to its own P&L

statement and to the performance goals of each individual who are rewarded according to the

achievements of their metrics. An integrated network business processes and information flows

ensures that the drivers of performance are transparent to the senior corporate management. In

the past, performance measures relied on one measure such as productivity often at the expense

of quality. In a business intelligence context, balanced scorecards help to assemble a groups of

metrics which best represent the correlation between individual performance and the company

overall.

Teradata’s Active Data Warehousing is one instance of a data warehouse concept which updates

data, in real time, for operational and strategic decision making. Companies are then able to

respond to significant events affecting their financial performance. In the past, batch processes in

data warehouses prevented quick responses to events. An alternative method of achieving the

same objective is the standards based Service-Oriented Architecture (SOA) which pieces

together components of business processes, data services and applications and each of them

can be invoked by a web service.

An instance of the use of event-based decision making is Land’s End which progressed from

using dashboards for reporting to more event-based decision making. In the first stage of

implementation of a BI solution, it began to provide figures of inventory as well as supplies in the

pipeline and compared them with incoming customer demand to make decisions about orders to

be placed with vendors. Now it has embedded triggers in its systems so that alerts are

automatically sent out whenever inventory runs low.

Event based monitoring does not stop at single or discrete situations but extends to responses to

a sequence of events. Thus, the monitoring of the supply chain would involve the movement of

goods from the vendor to the transportation company and then to the buyer. The events along the

way can have consequences for inventory holdings and financial payments. Delays can occur on

the way due to weather changes or congestion on the roadways or the vehicle carrying the goods

could meet with an accident. Based on preset rules, the BI software can make simultaneous

decisions for alternative means to replenish inventories as well as financial payments to

transportation companies.

An example of responses to complex events management is the experience of American Electric

Power Company which often made duplicate payments to its numerous suppliers who submitted

invoices under different numbers and in several different formats such as e-mail, snail mail, etc.

The complex event management software helps to detect duplicate payments by looking at

addresses, dates, amounts submitted and names of vendors to check for overpayments.

Business Activity Monitoring provides the infrastructure to monitor events, compare metrics to

standards and sends out alerts when action is required. In the daily routine of business,

companies need to keep track of their supply position. Often, supply will either exceed or run

short of requirements. BAM tools enable companies to compare the actual situation with

thresholds and send out alerts if the situation warrants action by decision makers.

BAM has been widely adopted in the financial services, logistics and the telecommunications

industry and its acceptance in the mainstream is expected to be completed by 2008. In the

financial services industry, it has been readily accepted as a large majority of trading decisions is

triggered by news feeds about financial events. In the logistics industry, a large number of

scheduling decisions are prompted by information on progress of shipments. Similarly, the

telecommunications industry uses BAM to monitor the observance of service level agreements.

One example of a product that servers such needs is Celequest's ActivityServer suite which

processes data flowing continuously from transaction systems. It compares standards of

performance and the actual metrics to determine when production yields are lower than the norm

or alerts store manager when inventory is running low. The standards of performance are

determined by comparing historical data in data warehouse with the actual achievements.

Brocade Communications uses Celequest’s Activity Server to monitor product yields achieved by

its contract manufacturers.

Enterprises would rather be naked

In order to test their strategic assumptions and to be able to see how they play out in practice,

company managements need greater visibility into their business processes before they can

evaluate the impact of their actions. A recent survey of 300 business-technology executives found

that close to 60% of them want greater visibility into their business processes while nearly 80% of

them are interested in data on performance metrics. The implication of an interest in visibility of

their business processes is that companies are looking for ways to monitor the ebb and flow of

business activity and take preventive measures if adverse situations are encountered.

For a closer integration of operational management and analytical capabilities for strategic

management, companies need to lower the information and decision latencies to be able to

respond to situations in real time. Business Activity Monitoring helps to lower information

latencies as it monitors current event data and uses BI to compare it to expected performance.

Business Process Management (BPM) software has the tools to make corrections rapidly. For

example, customer satisfaction will be affected by call wait time before customer representatives

can respond to customer queries. Business Process Management has to be able to use this

information to reroute calls to another call center or representative. This would involve the tasks

of finding the most efficient alternative route (modeling), ensuring that the traffic flow through

alternative routes to proportionate to capacity and the pathways are interconnected (integration).

(BPM), which is designed to work across departments, enables organizations to alter automated

and coded processes, without major re-investments in IT, prompted by analysis of event

information. At its efficient best, companies will be able to automate marketing by defining

customer data in a manner that is consistent with business decision rules and any turn in events

will change the approach to marketing campaigns. Software such as Fair Isaac's Blaze Advisor,

Ilog's JRules or Pegasystems' PegaRules are designed to read customer data and change the

offers made based on the customer information received.

BPM is intelligible to business users and helps to manage workflows and the process design. The

centerpiece of a BPM implementation is the process engine which monitors processes as well as

operational and business metrics, watching for exceptions that may require human intervention.

One of the benefits of rapid feedback from tactical decisions is business activity monitoring (BAM)

or the ability to take impromptu actions to remain on course to reach strategic objectives. Portfolio

managers, for example, have to be able to makes adjustments in their holdings based on news

feeds received. Delta Air Lines monitors the impact of unexpected events, such as weather

conditions and gate changes, to alert its employees and to update its gate display systems. This

would not be possible unless the analytical and operational systems are not integrated.

Increasingly, business process management software provides a graphical view of the design of

workflows as well as progress achieved all on a web site where it is transparent to the entire

company. Blue Rhino, a gas distribution company, used BPM to optimize the flow of shipments of

full gas tanks, inventories and empty tank returns among the company's 200 distributor locations.

An interlinked inventory tracking system continuously updates data on stocks and obviates the

need to manually reconcile the numbers and to issue paper bills of lading. The more advanced

BPM software also allows event management and triggers responses as exceptions are recorded

such as the actions asset managers need to take following interest rate changes.

IBM’s WebSphere Business Integration Monitor and TIBCO’s BusinessFactor have the

capabilities to use transaction data and conduct analytics to find out whether they meet standards

and they assess the ability of business processes to facilitate the expected performance metrics.

These products can also retrieve historical data from a data warehouse to provide the

benchmarks. The more recent acquisitions of these two companies, Alphablox product by IBM,

and TIBCO’s OpsFactor, have capabilities to monitor the impact of current events. This would

enable companies to analyze the impact of events such as delayed shipments on their financial

health.

Business intelligence software needs to be seamlessly integrated with operational data

management, business process management and visualization to meet operational objectives. In

one survey of users of BI software, it was found that 32% of the respondents consider pre-

packaged integration of BI software with existing enterprise applications as one of the most

sought after attribute. One instance of such an approach to product offerings is the case of Siebel

Business Analytics platform which has web services interfaces that enable it to insert analytical

results into Java or .Net and display the results in any application. Furthermore, its analytics are

closely intertwined with its CRM packages and directs the workflows for real time information

flows.

Visualization is essential for rapid collaborative decision making as illustrated by the case of Sun

National Bank, a regional bank, which needed to compete with much larger banks. It created a

single data warehouse for all its branches and provided analytical software to monitor

performance metrics such as the fastest growing products, regional variations in revenue, all

visualized as charts and graphs, on dashboards for rapid decision making.

A larger number of business and operational users need analytical tools that cater to their specific

needs. Consequently, analytical software needs to be scalable and designed for role-based

customization. A case where a customer switched from reporting tools to more scalable

enterprise software is the Ministry of Tourism in Bahamas which had used an OLAP tool, within

its offices, for its processing power and analytical capabilities. Later, the ministry needed to

communicate with 400 local hotels and regional tourist boards and switched to an enterprise

reporting tool with server-based computing environments. Lately, OLAP providers have upgraded

their scalability.

As information is drawn from a growing number of functional departments, business intelligence

software should be able to draw data from a variety of repositories. Information Builders product,

WebFocus 7, for example, provides access to more than 200 data sources and data formats,

including relational and legacy data.

An all-encompassing use of business intelligence is that companies find themselves looking at

both structured and unstructured data. Regulatory compliance, as required by Sarbanes Oxley,

requires detailed monitoring of controls which often means finding a record among million others.

Similarly, fraud detection in the corporate sector often requires sifting through thousands of e-

mails which would be impossible with manual methods. It would be hard for companies to

uncover pain points in customer experience unless they are able to mine call center

conversations. Similarly, companies invariably experience delays in understanding the causes of

warranty claims while shipments of the same product continue and later add to the costs of

refunds or product recalls. It is important to be able to correlate structured and unstructured

information to complete the analysis. Attensity is one company which has products that can read

as well as store structured and unstructured information in a relational form. Whirlpool uses its

Relational Extraction Server to extract information and develop insight about warranty claims,

customer feedback and service records.

One of the pioneers of an integrated use of structured and unstructured data is EDS, a systems

integrator for over 9,000 corporate customers. Its large client base implies that it buys countless

servers, PCs, networking gear, software and services from numerous suppliers who sell to its

offices spread in 65 countries. The contracts with its suppliers are a valuable source of

information to find ways to lower prices, as well as to evaluate suppliers based on their response

times and other metrics.

The benefits of integrating unstructured data can extend to streamlining the business processes

including increasing the efficiencies in the supply chain. Motorola has used EII to integrate its

information flow to be able to see the order status at every stage of its supply chain network.

Modern text mining technology has progressed beyond keyword searches of familiar search

engines and increasingly looks for items that fit a pattern. Searching by keywords often throws up

irrelevant results because the meaning of words can change with context. This is achieved by

using taxonomies, while mining textual information, which can ferret out results that are inter-

related. The significant difference in searching by taxonomies is illustrated by the experience of

Chelsea & Scott Ltd., a retailer of children's products under the OneStepAhead and Leaps and

Bounds brands. Customers could not often find the relevant products because they were not

aware of their existence. They could, however, describe the needs of their babies such as their

need for comfort. Installation of a natural language search engine change matters and a search

by comfort yields results of all the products that meet this need.

An all encompassing view of the enterprise

The integration of analytical and business processes requires additional choices involving

technologies that help to integrate them. Applications, content and data repositories, analytical

software and workflows management software have existed independently as discrete processes.

Increasingly, they will operate as a seamlessly integrated, continuous process that can be

monitored from an all embracing GUI or an enterprise portal. The analytical tools are integrated

by “portlets” or web services. These portals are available from companies like BEA Systems

which earlier specialized in application servers or players like Vignette which earlier specialized in

enterprise content management or from companies like Plumtree, the only remaining company

from the former ‘pure-play’ portal providers.

Alternatively, companies can also decide to integrate their analytical software with their

operational applications such as Customer Relationship Management Software. In such a

situation, they expose their Application Programming Interface to inter-link BI software or

components.

Similarly, relational data base companies have increasingly integrated analytical applications on

their databases. Microsoft, for example, now supports analytical applications on its ‘Yukon’ SQL

Server which includes support for OLAP, statistical and data mining functions besides database

queries.

The integration of business and analytical applications has to extend to linking content

repositories; this was not possible with the available integration technologies. With the advent of

web services, XML and a gamut of object-oriented, component based technologies enables

integration of structured and unstructured data. IBM has bolstered the prospects of the

technology by acquiring Venetica and uses its content integration middleware into its DB2

Information Integrator suite.

Depending on their business needs, companies can decide on the kind of package they want to

buy. If they would rather focus on specific departments, they can decide to buy packaged

analytical applications such as especially the finance department. On the other hand, they could

opt for integrated packages which include data management and integration, analytical tools and

applications and collaboration software. Companies can also decide to customize their business

intelligence software and elect to buy analytical development environment which allows them to

use components to build applications for their needs.

The overlapping functions of business intelligence and enterprise management have also

attracted many different types of vendors from the enterprise resource planning, performance

management and business process management space besides best-of-breed innovators who

excel in some segments of the process. Buyers, therefore, have to weigh the benefits of staying

with their familiar ERP vendors who increasingly have the ability to package business intelligence

packages versus choosing the richer functionality of ‘pure-play’ business intelligence vendors or

the niche ‘best-of-breed’ innovators. While the increasing availability of integration tools, with the

advent of web services, XML and Java, afford the ability to conveniently extend ERP packages,

management of a diversity of vendors does increase costs.

The sum of the parts is higher

Buyers of business intelligence software have to weigh the often conflicting needs of

standardizing the source of their software to realize cost economies as against acquiring

innovative software from ‘best-of-breed’ companies who often do not have the ability to provide a

package of transaction data software, ETL and business intelligence tools. The quality of

middleware has vastly improved in recent years with the advent of web services but companies

still prefer a solution from a single vendor as we will discuss later. While business transaction

database companies are increasingly incorporating business intelligence software, they are still

not functionally as rich in business intelligence functions.

At this point of time, the industry is divided between vendors who excel in implementation of

business intelligence software, as assessed by Gartner, and many of them are cash-rich

companies from the mature segments of the enterprise software sector. On the other hand, the

pure-play business intelligence software companies have stood out for their visions; they are

emerging companies which are relatively weak in their capabilities to develop the infrastructure

for business intelligence. None of the companies have both the capabilities for users to be able to

make a clear choice of a vendor.

The players from the traditional segments of the enterprise software industry, i.e., Enterprise

Resource Planning, RDMS, Customer Relationship Management and Supply Chain Management

companies have added Business Intelligence functions in their offerings. They stress their proven

track record and familiarity while the ‘pure-play’ BI companies berate this ability as commonplace

in an environment where web services facilitate convenient integration with existing applications.

According to estimates of Gartner, the users who decide to opt for ERP companies, who are also

able to meet the application requirements of customers including BI, will incur costs that are 25%

lower than those who decide on ‘best-of-breed’ or more specialized innovative companies by

2005. The best of breed companies are more innovative and find new ways to compete but the

fact remains that the older companies are either buying them or are able to incorporate or embed

their software within their package and lower costs of servers and license fees.

The merits of these contradictory claims can be judged from a recent survey of BI and DW

professionals who are members of TDWI. The data from 552 BI and DW professionals indicates

that 56% of them use BI solutions from their transaction database providers either exclusively or

jointly with third-party solutions. On the other hand, 61% of them are using either third-party

solutions exclusively or in combination with the transaction database providers. Users are more

likely to prefer transaction database providers when they need to access information from their

software and when the data is used for routine analytical functions like fraud detection and

profiles of customers for contact management. Also, they are more likely to use software from

transaction database providers when they also provide the ETL solution as well saving them the

job of extracting the data. On the other hand, third-party solutions are likely to be preferred for

advanced analytics like predictive analysis or ad hoc queries. Also, companies have a less

compelling need to stay with their transaction database providers when they already have legacy

software installed and need to use integration technologies. In all, companies are more likely to

use transaction database providers as they increasingly use BI for improving operational

management as is the case today.

The leap towards enterprise wide intelligence

Businesses have to often choose between data marts implemented in some departments in

contrast to the bolder alternative of investing in a data warehouse or an enterprise wide real time

intelligence infrastructure from the very beginning. A smaller data mart can be a test case and is

more likely to be accepted by a minority of technology buffs in an organization while a data

warehouse or enterprise intelligence is more efficient. If companies opt for a data warehouse,

they will encourage standardization of data definitions and planning for business process

management across the enterprise which is not a pressing concern when data marts are

implemented in some divisions. Furthermore, data marts require a different kind of technology

which is rendered obsolete later when enterprises decide to accept data warehouses or

enterprise wide real time intelligence. Also, data marts, duplicate data which has to be cleaned

later when companies switch to data warehouses.

One case where a company had to abandon its investments in a data mart is GE Real Estate

which has investments in over 8,000 properties around the world. The management of such a

portfolio required information that was often available in the individual countries where the

operating units were located. GE decided against the ideal solution of a web based data

warehouse solution primarily because reconciliation of a variety of data definitions used by its

departments seemed much too daunting. After the modest success of its data marts, GE had to

entirely abandon its Microsoft server which could not scale for a data warehouse and had to

instead change to a UNIX server with an Oracle database.

A more comprehensive survey of 150 technology executives, who have implemented business

performance management (BPM) across the enterprise, shows that a step-wise process has

been more successful for most companies. Twenty percent of the respondents have implemented

BPM and 60% of them implemented them for particular problems before expanding their scope to

the entire enterprise.

The decision to buy software for a specific department or the company as a whole is not simply a

technical decision. While the more skilled or “power users” are avid consumers who relish new

software, the risk of provoking a more political response from other employees is much greater

especially if they are either wedded to the tools they have used in the past (typically, Excel) or are

not sophisticated users of some arcane business intelligence software. The choice to use more

integrated software should be preceded by finding a sponsor with the clout to implement new

business intelligence software. Also, companies have to find the means to customize the software

according to the roles of each individual in the enterprise.

On the other hand, the benefits from enterprise business intelligence yield the greatest benefit.

Companies can more effectively align their strategies with the resources and tactics of their

company. They can also monitor performance in real time to ensure that their actions are able to

achieve their goals.

Intelligence for all decisions

Analytical software is not any longer a stand-alone application that is the preserve of power users

who have special skills to use arcane tools. Instead, analytical applications permeate the

enterprise and they have to be adapted for users who are pre-occupied with solving business

problems or just daily operations and do not want to bog down in mastering the technical nuances

of sophisticated tools. While power users require data exporting, cube and data modeling options,

business users need the ability to manipulate data, at a granular level, locally in Excel. Casual

users prefer dashboards and canned reports.

The business users are also interested in sharing information with their team members. Tools like

Crystal Reports and Actuate Corp.'s e-reporting disseminate information to many casual users and high-

level decision makers.

Finally, analytical tools have to be seamlessly integrated with business process management as

well as tools for evaluation of results of actions taken based on the initial analysis. An example of

this kind of software is Customer Power 5.0, a tool developed by NCR, which is targeted at

customers in the retail, financial and catalog marketers. They are able to use customer

information and analytics for improving the effectiveness of marketing campaigns. One clothing

company is able to use transaction information to create catalogs specially designed for its best

customers.

Enterprise users look for tools that have an intuitive interface and are customized for their

industry and functional area of expertise; generic analytical tools are less likely to meet their

needs. The need to expand the functionality of analytical tools stretches the technical capability of

vendors. In the past, analytical software companies specialized in a limited range of functional

expertise. Some companies like Hyperion excelled in performance management while others like

SAS and SPSS distinguished themselves in statistics. In their efforts to meet a more diverse set

of needs of customers, analytical software vendors are now entering a risky terrain where they

have to acquire companies to provide a portfolio of products including integration with business

process management software. Furthermore, they have to be able to gain domain knowledge of a

variety of industries where they have customers. Finally, customer needs are evolving as

knowledge workers in enterprises find ingenious ways to use information. It is unlikely that all

customers will be able to find the entire range of functionality they need.

An alternative possibility is that customers can buy a package of tools and have the ability to build

new applications using development tools and a platform. Custom applications, developed by

customers themselves, are more likely to take advantage of the domain knowledge than vendors.

Customers would prefer component based application development with graphical tools so that

they don’t have to invest inordinate amount of resources in programming. A typical product of this

nature which bundles a package of software tools and a development environment is Oracle’s

10g which combines business analytic and reporting tools, Oracle Discoverer for querying

analysis and dashboard features; Oracle Spreadsheet Add-In, which allows direct access to

Oracle's online analytical processing (OLAP) from within Microsoft Excel spreadsheets; Oracle

Warehouse Builder with extract, transform and load (ETL) capabilities; and Oracle BI Beans, a

set of custom developing tools.

The advantage of custom applications is that they readily adapt to the existing infrastructure of

the company including the data warehouse. In addition, companies have already implemented

their data categories, dimensions and the data model and their customized applications will adapt

better to this ensemble while packaged applications would require considerably greater

adjustment within the enterprise including changes in the data warehouse.

Weaving the information fabric

The latest research, based on feedback from users, indicates that ease of integration is the single

most important criterion for selection of Business Intelligence software. More than 75% of the

users consider this to be the most important factor in the selection of BI software. The high level

of significance of integration is illustrated by the case of Alaska Airlines which preferred Siebel

Analytics for the ease of integration of the software in its heterogeneous environment especially

the data warehouse. The metadata that came with Siebel Business Analytics is especially useful

in the integration process.

Data warehouses have been the lynchpin of strategic applications of business intelligence; they

enabled companies to coalesce information from disparate sources. In addition, the data

warehouse stores data only after it has been cleaned for inconsistencies and when it conforms to

standard definitions. Data is prepared for use in data warehouses by ETL (extraction,

transformation and loading) tools which extract data from operational data stores, transform the

data so that data definitions are consistent and duplicated data is removed and the output is

loaded into data warehouses in line with its metadata. The data warehouse updates the same

information. The downside of the data warehouse is that it can update information only when

analytical functions are interrupted while data can be updated usually on weekends or beyond

work hours. As the volume of data grows, the time spent on loading increases to an extent where

it would conflict with data analysis functions. Also, real time analytics is really not feasible with the

latencies that are inescapable with the data warehouse.

The alternatives to ETL technologies are the EAI (Enterprise Application Integration) and EII

(Enterprise Information Integration) technologies which can access data from a variety of

sources. The EAI integrates applications and helps to access data from them. One example of

how a company leverages such a technology is Virgin Mobile which needed to rapidly expand its

services in the USA without sinking investments in an elaborate IT infrastructure. It struck an

agreement to use Sprint’s operating infrastructure, for the phone service, which was integrated

with its CRM and financial software using process integration software.

The problem with simple integration of applications is that their data definitions or the metadata

may not be the same so that it would be hard to extract data without standardizing their

definitions. On the other hand, EII helps to both integrate the applications and uses XML to match

the taxonomy of the data extracted.

For real time applications, BI software will have to work with both historical and current

information. Enterprise Information Integration software is able to draw information from both the

data warehouse as well as operational data stores in real time as transactions happen. When the

data is drawn directly from operational data stores, it is not cleansed of inconsistencies in data

definitions or duplication. A BI tool, such as Cognos ReportNet, uses Composite Software’s

Composite Information Server, to extract information from data repositories throughout an

enterprise and create a single, more comprehensive data source.

One application of the EII technology is the case of Owens Corning which combined its EII

software and the BI software to generate reports on gross margins earned every day. The data

was extracted from its numerous ERP databases. In the past, the same exercise took about a

month. The reduction of the lag time in reporting has helped Owens to make mid-course

corrections.

An increasingly preferred means of integrating software is the Services-Oriented Architecture

(SOA) which reduces the monolithic applications of the past into components or services which

construct the larger applications. These services or applications are mounted on a single user

web based user interface. Users can choose the services they need instead of buying a package

which often has components that are not relevant for them. Each of these components are inter-

connected by web services which use XML tags to build an interface which can be used to

access services using web technology. Web services can be used to provide access to business

processes, applications, BI cubes, reports, queries and data integration functions, databases etc.

The web services are indexed in a UDDI registry. In order to connect to a web service, an

application or portal simply queries the UDDI registry, finds the service and then dynamically

connects to it by sending it an XML message known as a SOAP (Simple Object Access Protocol)

message. The Service Oriented architecture is a more cost effective since it does not require IT

people to inter-connect applications.

An example of a SOA implementation is Siebel 7.5.3, released in mid-2003, which is expected to

crystallize with the Siebel 8.0, due in 2006. The complete Web-services-oriented software will

include workflow management tools.

The cross-point of all data

At the heart of efforts at integration is the task of creating taxonomy and metadata or descriptions

of data which helps to describe data across any application, repository or database. Most

metadata is sui generis and hard to relate to other descriptions of data associated with another

application.

Data is fragmented for a variety of reasons. Structured and unstructured data is hard to correlate

because the former is stored in a relational data base and the latter in a content repository.

Content itself is divided between document, web and digital asset management repositories. It is

hard to find related structured and unstructured information in a relational database and a content

repository because the former uses SQL to extract data and the latter uses a search engine. The

descriptions of the data vary depending on the function of the person who is using it. A marketing

person is concerned about customer acquisition while a sales person is interested in prospects,

orders and closures. The information systems they use will differ; a marketing person will use a

CRM system while a sales person will typically use Salesforce.com. Both will enter customer

reference data often with different conventions about recording first names and second names,

address entries, etc. When the units of a company are spread across the world, each national

subsidiary uses a different language which often means they also use another information

system.

Increasingly, companies need to be able to correlate information. They need to be able to see

inter-related information about customer purchases and their behavioral traits which are best

described in text. Similarly, they need to correlate information about customer acquired and the

sales earned from them. They also want to share content created in one subsidiary with another

in another country and translate and use it rapidly.

A real time enterprise needs not only gain access to all this information in a cohesive manner but

also be able to do this quickly. The advent of XML has considerably simplified the process of

correlating disparate information by described its components at a more granular level with the

help of tags. Once content descriptions are available at this level of detail, some extracts can be

reused rapidly. With the use of XML, it is now possible to search simultaneously in relational

databases as well as content repositories. Companies are now able to translate content in a

repository in one country for use by another country without requiring the intervention of skilled

translators.

MacDonald is a classic instance of a company that operates across national boundaries and has

to work with repositories located in several different countries. In addition, it has numerous

franchises working independently and loose time when they recreate the same content. In order

to collate information from its subsidiaries, MacDonald has to be able to translate the content

from several different languages and draw them from several different information systems. It

also benefits from centralizing content creation and redistributes it to local units where it can be

customized. All the content can be accessed from a portal.

Golden Gate Software’s Data Synchronization solution is an example which helps to use data

from disparate sources in real time. It has been used in the health care industry by hospitals to

collate information from individual hospitals and patient care centers in a single repository. With

synchronized clinical and administrative data, one of its customers is able to produce detailed

analytic reports to track patient metrics, length of stay, and other care management data.

CONSIDERATIONS ABOUT THE FUTURE

RFID

Radio Frequency Identification is used to identify, track and sort objects. A tag is inserted in an

object for transmitting information about the product identification; the data is transmitted by radio

waves and received in a digital form for storing in computers. The tag is a tiny antenna with the

ability to send radio signals to a reader or a mechanism that is able to receive radio signals. RFID

is presumed to have potential applications in supply chain management which is expected to

generate enormous data for BI analysis purpose even though applications development is

uncertain.

Wal-Mart’s declared its plans to use RFID with much fanfare but the effort petered off after the

suppliers balked at the idea. The aggregation of the RFID data will be a complex task of

integration as the data will be received from a plethora of often small companies involved in

logistics management.

Yet, some companies have found ways to use RFID data to their advantage. Graniterock, a

Watsonville, California based company, has installed a BI system to monitor time spent by trucks

in its quarries. The clerical tasks of identifying incoming and outgoing trucks don’t need to be

done any more, as their movements are recorded electronically, so that Graniterock saves time.

The customers benefit as they are able to monitor the costs they incur on the time spent by

individual trucks and the relative efficiency of each company.

RFID has potentially profitable applications in several different industries. Tracking baggage in

the airlines industry is an enormous cost and pioneers like Delta Airlines is already using RFID to

trace lost baggage. The drug industry also has an intractable problem of controlling the diversion

of drugs for intoxication.

Complex Business Event Monitoring

Investment in Information Technology infrastructure creates new possibilities. CRM paved the

way for business analytics. In turn, business analytics afforded opportunities for event based

business activity monitoring. Even as companies are implementing simple event based business

activity monitoring technology, conditions have been created for more complex event based

business activity monitoring which enables decision making including pre-emptive actions, based

on predictions, which help to avert a crisis or an adverse situation from happening.

An example of more advanced event based monitoring is the action that can be taken in

response to news feeds in the financial services industry. A simple event based business activity

monitoring would be a stop-loss routine which triggers a sale whenever the price of a stock falls

by a given percentage amount. A relatively more complex business activity monitoring would be

to monitor trading activity to determine whether the patterns in bids by traders in an auction

suggest that they are colluding and are violating SEC regulations. This could go further and rules

could be written which help to predict the future performance of the stock and investment

decisions are triggered when expected events actually occur.

The use of complex event monitoring is expected to rise with the use of RFIDs when a lot more

data is expected to be gathered. Several different applications are possible with the data

gathered from the scanning of the codes embedded into products. The obvious application of

RFIDs is to monitor trends in inventory depletion. The more important applications would be to

monitor the time of delivery by different transportation companies which could be affected by the

conditions along the route, the quality of maintenance of vehicles or by the driver’s efficiency.

Companies can begin to analyze the data and find better means for routing, loading and

unloading techniques as well as data on congestion along the way.

A recurring issue with retailers is out-of-stock items (OOS) which lead to lost sales opportunities.

According to a study conducted by Andersen Consulting, 53% of OOS situations happen as a

result of poor stocking decisions. Another 8% of OOS situations happen as a result of lags in

moving inventory from the warehouse to the shelves. Some pioneers, like U.K. supermarket chain

Tesco, have taken the lead in adopting RFID technology to monitor shipments of high-value

nonfood goods to avoid OSS.