qlikview_in-memory intelligence for the way we live and work_20100604final[1]

7/27/2019 QlikView_In-Memory Intelligence for the Way We Live and Work_20100604FINAL[1]

http://slidepdf.com/reader/full/qlikviewin-memory-intelligence-for-the-way-we-live-and-work20100604final1 1/15

A CITO Research Explainer:

In-MemoryIntelligence forthe Way WeLive and Work

Sponsored by QlikView

CITO ResearchTell Us a Question.

JUNE 2010



ContentsIntroduction 1ePlannedTyrannyofBusinessIntelligence 1

eQlikViewHypothesis 2

eProblemwithQueriesandCubes 3

BigBIandLittleBI 4

Cubes:RigidPrecalculation 4eStackArgument:HowMuchIsEnough? 5

eBIChasm 5

QlikView’sAssociativeIn-MemoryArchitecture 5

Step1:Load 7

Step2:Search 7

Step3:Visualize 7

CommonQuestions 8

BusinessImplicationsofQlikView’s AssociativeIn-MemoryArchitecture 11

Databecomesmorevaluable 11

LivingReports 11

Examples 12

Fraud 12Shipping 12

CustomerProtability 13

BusinessDiscoveryDay 13



1QlikView

In-Memory Intelligence


IntroductionQlikView’s goal is simple: to provide tools or exploring and visualizing data that

enable you to personally fnd answers and drive innovation. To understand the value

o what QlikView oers, it’s important to frst take a close look and what’s right and

wrong with the way we currently use inormation. In business, the practice o using

inormation to run a business more eectively is called business intelligence (BI). This

paper aims to explain how QlikView’s Associative In-Memory Architecture brings BI to

a new level o power that is under the direct control o the person asking the ques-

tions. This paper should appeal to anyone wondering what QlikView can do or them

and to users wanting deeper insights into how the application works in order to make

the most o it. It also explains how QlikView is undamentally dierent rom traditionalBI across a number o dimensions.

ePlannedTyrannyofBusinessIntelligence

Problems are almost by defnition unexpected. We rarely wake up in a cold sweat at

4:00 AM knowing that there’ll be a crisis later that aternoon and that we’d better

start researching now to be ready. When we start asking and answering questions, the

pathway o analysis is rarely predictable. One question always begets others.

Ooading the task o fnding and understanding inormation to someone else isn’t

very productive, either. There’s not much point in waiting while they sit through

search results and return with their fndings, only to send them back again with

another idea once the results come up short. Putting others between us and the inor-

mation we need means missing out on ideas that occur to us during the search itsel.

And how many iterations o this could you stand to go through in a day?

This research process summarizes the problems with most modern BI systems:

n The questions must be planned in advance

n The search is rustrated; instead o knowing exactly what we want or being

open to new ideas, the process is dragged out and hindered by query mechanisms

every step o the way

O course, many questions—what are my revenues? my profts? my margins?—can, in

act, be answered ahead o time. Queries do work. But typically the action is moving

too ast, and BI systems are too monolithic to change and keep up with day-to-day

needs. We want to recalculate proft by region today, by product tomorrow. Most o



2QlikView



the time, BI cannot keep with our changing needs or provide an answer in a split

second. The result most oten is that data is not used to help answer key questions.When the questions must be answered, the typical BI process results in a large bottle-

neck acing anyone trying to answer new questions and meet new challenges.

eQlikViewHypothesisQlikView takes an alternative view o analysis that can be summarized in three steps:

n Consolidate: Identiy related data sets, map the associations between them,

and load it all into a QlikView fle resident in memory

n Search: Explore the data using “list boxes” that display the unique values in

each feld and can be highlighted and aggregated. Both inormation includedand excluded rom the selection criteria is displayed and updated instantly

n Visualize: Maps, charts and assorted graphics can be created and instantly

updated

QlikView aims to increase your chances o making genuine discoveries and eliminate

much o the grind:

n No queries, no middleman: You click to select data and click again to deselect it

n Matching and non-matching data is displayed: It’s not just what you see,

but what you don’t see—which data was excluded and why? What happens when

you mix it in?

n No waiting: The answers are right in ront o you

The idea behind QlikView is that closing the loop on asking questions and encour-

aging individual exploration leads to better answers, insights, and innovations.

Examples can be seen at demo.QlikView.com. Why this is true might best be explained

by Eric Von Hippel’s theory o user-driven innovation, which argues in avor o provid-

ing tools directly to end-users.

What’s more dicult to grasp is how, exactly, QlikView’s sotware manages to do

what’s just been described. That is the ocus o this paper. The ollowing sections

explain QlikView’s In-Memory Associative Architecture and its implications or your

business.



3QlikView



User-DrivenInnovationWhy should companies pretend to know what customers want? Instead, why

not give customers the tools to create or improve their own solutions? In a nut-

shell, this is the argument o MIT Sloan School o Management proessor Eric

von Hippel. Decades o research has ound that innovations requently trickle

up rom users—one o his studies ound that 82% o new capabilities or scien-

tifc instruments like electron microscopes developed this way. Why? Because

users came up with changes the manuacturers never considered. Von Hippel

believes that users can help companies innovate more quickly and inexpen-

sively than companies can internally.

The key is giving users the tools. This has proven relatively easy to do in sot-

ware and on the Internet, where open source breakthroughs like Linux are old

hat. As Moore’s Law and economies o scale conspire to drive down costs and

complexity, it becomes easier than ever to place powerul instruments in the

hands o the people, an untapped orce von Hippel once described as “the dark

matter o innovation.”1

eProblemwithQueriesandCubesDatabase queries—the bedrock o traditional BI—are powerul, but they’ve always

aced limitations. For one thing, they’re oten incredibly complex and dicult to

construct, requiring subqueries o data subsets that can boggle the mind. (Entire

books have been written on how to build better queries.) Figure 1 shows an abstract

view o what an SQL query looks like.

Figure 1: An Abstract View of an SQL Query

1 http://www.nytimes.com/2007/03/25/business/yourmoney/25Proto.html

AggregateFunctions

List of Fields Where Join CriteriaSelectionCriteria

Select

Select Field 1, Field 2 Sum (Field 3) where Key 1= Key 2 and Field 3 > 0



4QlikView



Queries have an ontological problem as well: they can tell you known unknowns—

common, dependable metrics such as sales and customers by region—but they can’ttell you unknown unknowns, the questions you haven’t thought to ask. The ormer

is the province o traditional BI; the latter is where insights are discovered. A query

returning a list o every widget sold to every customer in North America might be

useul, but a list o who didn’t buy and what didn’t sell is equally i not more useul.

QlikView always shows you both at the same time: what matches the selection criteria

and what does not.

Why are queries so rigid? Because BI was developed 25 years ago at a time when

storage, memory, and computing resources were scarce. It was impossible to process

millions, i not billions, o records in a timely manner. As a necessary shortcut, the

relevant data was culled and mapped to precomputed parts o answers called cubes

to be queried later. It was an elegant solution to hardware constraints that no longer

exist, but decisionmakers are still living with the consequences o the earlier and more

limited design.

BigBIandLittleBI

The current way inormation is used in most organizations might best be described

as “Big BI”—monolithic enterprise applications that are infnitely scalable but require

ull-time care and eeding rom IT departments to create an endless array o queries,

cubes, tools, and dashboards requested by users. This inevitably creates a bottleneck

when simple requests or a dashboard are waitlisted or nine months or queued

with a dozen similar dashboards until implementing them all at once justifes the

resources. Big BI is devoted to making as many people as happy as possible with a

solution or the lowest common denominator; i you’re one o the unlucky ew, your

only other option is “Little BI”—breaking that dashboard into ad hoc queries and

spreadsheets and getting by with what you have.

Cubes:RigidPrecalculation

Big BI is in the business o building precalculated cubes: simply tell them ahead o

time every possible dimension you could want to analyze, and they will calculate a

metric across all o those dimensions and store it somewhere—in a cube. When you

ask it or that metric—say, proftability—it instantly returns the answer. The upside is

speed—there’s no waiting. The downside is that you can only ask questions that you

already knew you wanted to ask.



5QlikView



eStackArgument:HowMuchIsEnough?

Big BI is also in the business o selling you eatures, like a car salesman would. The

most important question when choosing a vehicle is: will it get you where you want to

go? But sometimes the decision gets bogged down in choosing eatures rather than

a solution. Big BI is a large commercial utility vehicle—it bristles with eatures and has

the brute strength to do just about anything. It’s also expensive, temperamental, and

not very maneuverable. You can’t just take it out or a spin, or hand the keys to just

anyone. In act you may need to hire drivers with a special license. You don’t need a

tank when a bicycle will do.

eBIChasm

Between Big BI and Little BI is the “BI chasm” —the missing middle ground between

queries and spreadsheets, where enough o Big BI’s capabilities can be brought to

bear powerully, with enough speed and ease-o-use that users can grasp its value

immediately. What do you need to bridge that gap?

QlikView’sAssociativeIn-MemoryArchitectureQlikView aims to bridge the BI chasm by replacing queries and cubes with its associa-

tive in-memory architecture, shown in Figure 2. Rather than precalculating answers,

the sotware loads data sets into memory and maps the associations between them.

Because it’s associative, there are no predetermined paths and no precalculations—

you can layer on as many metrics and ask as many questions as you like. Because it’s

all done in-memory, the answers are returned instantly and updated continuously.

Your BI is no longer as good as your IT department’s last cube—it’s as good as the

questions you ask.



6QlikView



Figure 2: Associative In-Memory Architecture Explained in 3 Steps



7QlikView



The Associative In-Memory Architecture is put to work every time you use QlikView

through the three basic steps explained next.

Step1:Load

QlikView uses load scripts to load data rom dierent repositories into a single large

table where associations are automatically made. The load scripts do the job o the

join criteria o an SQL statement. Experts might think o what is in memory as the

result set o an outer join o all data loaded. Everyday users can think o the data in

memory as being loaded into a massive in-memory table, like a spreadsheet, that has

columns o all loaded data and where each row has all the data or all rows linked with

the join criteria expressed in the load script. Loading the data into a QlikView fle sets

the stage or analysis.

Step2:Search

QlikView doesn’t show you a view o the raw in-memory table ull o data. Instead,

you see list boxes, one per column. Each box shows all the unique values in the feld.

Doing the work o the selection criteria in an SQL statement is as simple as clicking

specifc values in each list box: green items are the ones you’ve selected. In other list

boxes, values included in your selection are white and those excluded are gray. The

relationships are easy to see. QlikView uses the in-memory table to keep track o

which data is associated with the selection criteria.

Using list boxes to search through and understand data is more like a conversation

than a query. Which products were sold in North America? Who are my gold cus-

tomers? And how do they overlap? In this manner, you can construct complex queries

on the y while never losing sight o which data has been included or excluded—

oten it’s the latter where crucial insights are ound. What didn’t sell in North America?

And who didn’t buy those products? Maybe you never thought to ask that beore.

Or maybe you couldn’t.

Step3:Visualize

Data is visualized in two ways: through metrics that summarize the data and graphicsthat display the data that is selected along with graphical orms o the summaries.



8QlikView



Aggregate metrics allow unctions to be calculated or all selected data. Aggregate

metrics are defned by choosing rom more than 200 standard mathematical andstatistical unctions (standard deviation, mean, median, count, and so on). They can

be combined to create custom metrics or your business. These metrics are shown

in charts based on what has been selected by list boxes and are updated instanta-

neously whenever selection criteria are changed.

QlikView’s graphic visualizations render feld and aggregate values in any number

o charts and representations, updated continuously by the underlying data. The

columns and sizes in a bar chart change instantly as the selection criteria are updated.

While the details o the associative in-memory architecture could fll a book, the eel-

ing that you get when using QlikView is that the data set comes alive. You see howaggregate metrics and graphics are transormed as dierent sets o data are selected.

You do this yoursel, with no waiting, so the trip rom a question to an answer is short

and direct.

CommonQuestionsHere are some o the common questions that we’ve heard about QlikView.

What Do Load Scripts Do?

Load scripts and QlikView’s ability to make associations between data sources rep-

licate the job o the join criteria in an SQL query. A QlikView fle may consist o data

rom many sources. Unlike a spreadsheet where you may have one worksheet with

one type o data and another worksheet with unrelated data, all data in a QlikView

fle must be related to the other data in the fle. The job o the load scripts is to bring

the data in rom external sources and explain the way that the data is related. This is

what is meant by associative. Each feld and row is connected to the others in the fle.

What Does Associative Mean?

Associative means that each feld and each row is connected to every other feld androw. The load scripts tell QlikView how they are related. The in-memory data structure

can then be used or rapid recalculation. Figure 3 shows the Table Viewer in QlikView

that keeps track o how each data source is connected to the others.



9QlikView



QlikView automatically discovers associations based on common feld names, or the

associations can be set in a load script ater the data is loaded.

What Is a List Box?

At the heart o QlikView are “list boxes”—roughly the equivalent o a column in a

spreadsheet or a database with one important dierence. List boxes display every

unique value in the feld—duplicate values are not displayed.

The other large dierence is that list boxes control what data is selected. By clicking

on a row or range o rows in a list box, all data elements with those values in that

feld are selected. You see all selected data turn white in the other list boxes. Let’s

say you have list boxes or products sold and the regions they were sold in. Theremight be 200 products and fve regions. Each region and product is a list box. Select

a region and QlikView displays the products sold there; select a product and it will

show the regions it was sold in. These relationships are mapped automatically. In this

way, QlikView acts as an “association engine,” oering end users a simple way to see

and understand the relationships between data.

Figure 3: QlikView’s Table Viewer Shows the Associations Between Data Sources



10QlikView



What Is an Aggregate?

Aggregates are mathematical and statistical unctions combined to create metrics.

An example might be the average price paid or a product in a given region—or any

o more than 200 built-in unctions. Aggregates are custom metrics that are relevant

your business.

What Do You Mean by Visualization?

In QlikView, visualization means more than it does in other sotware. Charts (and any

number o graphic representations) are updated continuously by the data under-

neath them. This in itsel is not unique to QlikView. What sets QlikView apart is being

able to visualize the associations between the data, which is even more useul than

dynamically updating charts.

Why Does In-Memory Matter?

Historically, BI made the best o a bad situation by coaxing insights rom tremendous

volumes o data beyond the ability o most hardware to handle. It discovered a short-

cut by pre-processing queries and cubes ahead o time. While computing has been

transormed since, BI hasn’t. Today, BI itsel is the bottleneck.

QlikView’s Associative In-Memory Architecture is an attempt to reconceive BI or the

modern computing environment—one in which the tools or analysis and discovery

are placed in users’ hands. Big BI was born into a computing universe o scarcity. But

25 years and a dozen cycles o Moore’s Law have made memory cheap and hard-

ware exponentially more powerul. Someone like Wal-Mart might not be able to

store its trillions o transactions in memory yet, but it will. Many businesses already

can. QlikView was the frst to handle all data in memory; other vendors are starting to

ollow suit.

What Database Does QlikView Run On?

QlikView does not run on a database at all, but can load data into memory rom asmany dierent databases as you may have in your landscape.



11QlikView



BusinessImplicationsofQlikView’s AssociativeIn-MemoryArchitecture The business value o QlikView’s architecture is commonly revealed in the ollowing

ways.

Databecomesmorevaluable

The structure o traditional BI reects the structure o the traditional corporation,

where power was centralized, consolidated, and controlled. Vertical organizations

long ago gave way to horizontal ones, but BI has struggled to keep up. The great

advantage o QlikView’s easy-to-use ootprint and modern architecture means

that the data you’ve been studiously collecting throughout your organization can

be pushed to where it’s needed and put to use directly by the people who need it

to make decisions. By placing the tools o analysis in the hands o users, you’re

potentially unlocking a resource that no BI system has ever been able to harness—

the intelligence o your colleagues.

Senior managers—the traditional constituents o BI—fnd QlikView to be a powerul

tool or accelerating their own analysis and decision-making. Those most accustomed

to the limitations o BI will appreciate associative search more than anyone.

LivingReportsBI systems produce reports telling you about the past—what happened last year, last

quarter, last week, yesterday? The analysis is out-o-date the moment it’s published.

QlikView’s graphic visualizations are drawn in real-time rom the underlying data

as it changes. In this way, QlikView provides a living, breathing real-time view o your

business.

End-User Creativity Is Unleashed

Traditionally, BI applications produce one report and copies are orwarded to every-

one who needs them. Putting QlikView in the hands o dozens, hundreds, or even

thousands o users has the potential to transorm how BI is used by taking it out o thehands o IT and placing it in the hands o end users. While this can potentially place

incredible strain on the underlying hardware, QlikView customers report that the

benefts o democratizing “business discovery” are transormative.



12QlikView



All Levels of Users Are Supported

QlikView is one o the rare technologies that is popular with the entire range o theend-user population in a company, rom the most advanced business analysts and

hands-on users who want the answers in just ew clicks. Early adopters tend to have

PhDs, programming experience, and the initiative to create 50 QlikView-powered

applications that they then seed to business analysts. These superusers tend to make

the case internally or QlikView’s worth and broader adoption.

Less technically skilled but eager to accelerate the process without IT, business

analysts look to QlikView to quickly create apps and explore relationships between

databases, oten building them rom top-to-bottom. They might need some guidance

to create load scripts and integrate data sources, but ater that, they’re sel-sucient.

End users need the most help o all, requiring analysts and data experts to create

models and handle the heavy liting, but once the interace is in their hands, they’re

also to dive deeply into the data.

ExamplesHere are a ew examples o how QlikView is being used today.

Fraud

A number o retailers using QlikView have discovered instances o raud that hadeluded their best eorts. One company uncovered previously invisible patterns in

ordering, shipping, and returns that pointed to millions o dollars in raudulent trans-

actions accumulated a ew hundred dollars at a time. It discovered the losses ater

using QlikView to combine and analyze ormerly separate databases—its inability to

do so had created the loophole in the frst place. The raud was the result o an inside

job, perpetrated by employees amiliar with the limitations o its systems.

Shipping

While hunting down raud, the same retailer discovered that it had never taken a close

look at the relationship between order patterns and shipping container logistics.Simply by rearranging how it packed containers—and by oering extra incentives on

especially ecient orders—it saved $400,000 a year.



13QlikView



CustomerProtability

Customers are using QlikView to make many areas more ecient. Sales and market-

ing teams have embraced the tool to identiy high-potential clients and campaigns

in an eort to maximize scarce resources. Sales teams are not usually among early

adopters, but the value o the data is compelling or them and directly impacts the

bottom line.

BusinessDiscoveryDay

Another customer has instituted “Business Discovery Fridays,” days on which devel-

opers are encouraged to put aside regular duties and ollow the data where it leads

them. This has a two-old purpose: one is training developers to think associatively

and to improve their pattern recognition skills; the other is to discover new projects

to implement in QlikView.

ACITOExplainer

CITO Research is a source o news, analysis, research, and knowledge or CIOs, CTOs,

and other IT and business proessionals. CITO Research engages in a dialogue with its

audience to capture technology trends that are harvested, analyzed and communi-

cated in a sophisticated way to help practitioners solve dicult business problems.

This document is a CITO Research Explainer, a orm o content intended to explain a

topic that is o potential importance to CIOs and CTOs. This explainer was sponsored

by QlikView in order to create a clear explanation o its products. To fnd out more

about how explainers are created, go to www.CITOResearch.com.

qlikview_in-memory intelligence for the way we live and work_20100604final[1]

Documents