Using Bitmap Index to Speed up Analyses of High-Energy Physics Data

John Wu, Arie Shoshani, Alex Sim, Junmin Gu, Art Poskanzer

Lawrence Berkeley National Laboratory

Wei-Ming ZhangKent State University

Jerome LauretBrookhaven National Laboratory

Outline

Overview bitmap index Introduction to FastBit Overview of Grid Collector Two use cases

“common” jobs “exotic” jobs

Basic Bitmap Index

Compact: one bit per distinct value per object

Easy to build: faster than common B-trees

Efficient to query: only bitwise logical operations A < 2 b0 OR b1

2<A<5 b3 OR b4

Efficient for multi-dimensional queries Use bitwise operations to

combine the partial results

Datavalues

015312041

100000100

010010001

000001000

000100000

000000010

001000000

=0 =1 =2 =3 =4 =5

b0 b1 b2 b3 b4 b5

An Efficient Compression Scheme-- Word-Aligned Hybrid Code

10000000000000000000011100000000000000000000000000000……………….00000000000000000000000000000001111111111111111111111111

2015 bits

01000…

Literal word

100…111111

Fill word

001…111

Literal word

Run length is 63WAH includes three words

Groups bits into 65 31-bit groups

Encode each group using one word

31 bits 63*31 bits 31 bits

31 bits 31 bits…31 bits

Merge neighboring groups with identical bits

Compressed Bitmap Index Is Compact

Expected index size of a uniform random attribute (in number of words) is smaller than typical B-trees (3N~4N)N is the number of rows, w is the number of bits per word, c is the

number of distinct value, i.e., the attribute cardinality

100 M, synthetic 25 M, combustion

N2)(1/c)1/c)(1(1 c22w22w1w

Nc

Compressed Bitmap Index Is OptimalFor 1-dimensional Query

Compressed bitmap indices are optimal for one-attribute range conditions

Query processing time using is at worst proportional to the number of hits

Only a small number of most efficient indexing schemes, such as B-tree, has this property

Bitmap indices are also efficient for multi-dimensional queries

Compressed Bitmap Index Is Efficient For Multi-dimensional Queries

Log-log plot of query processing time for different size queries

The compressed bitmap index is at least 10X faster than B-tree and 3X faster than the projection index

Data Analysis Process In STAR

Users want to analyze “some” (not all) events Events are stored in millions of files Files distributed on many storage systems To perform an analysis, a user needs to

– Prepare an analysis– Write the analysis code– Specify the events of interest

– Run an analysis1. Locate the files containing the events of interest2. Prepare disk space for the files3. Transfer the files to the disks4. Recover from any errors5. Read the events of interest from files6. Remove the files

Components of the Grid Collector

Legend: red – new components, purple – existing components

1. Locate the files containing the events of interest– Event Catalog, file & replica catalogs

2. Prepare disk space and transfera) Prepare disk space for the files

– Disk Resource Manager (DRM)b) Transfer the files to the disks

– Hierarchical Resource Manager (HRM) to access HPSS– On-demand transfers from HRM to DRM

c) Recover from any errors– HRM recovers from HPSS failures– DRM recovers from network transfer failures

3. Read the events of interest from files– Event Iterator with fast forward capability

4. Remove the files– DRM performs garbage collection using pinning and lifetime

Consistent

with otherSRM based

strategies and tools

Grid Collector: Architecture

Analysiscode

New query

Event iterator

Event CatalogIn: conditions

Out: logical files, event IDs

File LocatorIn: logical name,

Out: physicallocation

Grid Collector

File SchedulerIn: physical file

DRM

Administrator

Fetch tag fileLoad subsetRollbackCommit

Index BuilderIn: STAR tag fileOut: bitmap index

NFS, local disk

Replica Catalog

HRM 1

HRM 2

Clients Servers

Replica Catalog

FastBit Index For Event Catalog

For 13 million events in a 62 GeV production (STAR 2004)

Event Catalog size (include base data and bitmap indices): 27 GB tags: 6.0 GB (part of the base data of Event Catalog) MuDST: 4.1 TB event: 8.6 TB raw: 14.6 TB

Time to produce tags, MuDST and event files from raw data: 3.5 months, 300+ CPUs

Time to build the catalog: 5 days, one CPU

Grid Collector Speeds up Reading

0

1

2

3

4

5

0 0.2 0.4 0.6 0.8 1

selectivity

sp

ee

du

p

Sample 1

Sample 2

Sample 3

Test machine: 2.8 GHz Xeon, 27 MB/s read speed Without Grid Collector, an analysis job reads all events Speedup = time to read all events / time to read

selected events with Grid Collector Observed speedup ≥ 1 When searching for rare events, say, selecting one

event out of 1000, using GC is 20 to 50 times faster

1

10

100

1000

0.00001 0.0001 0.001 0.01 0.1 1

selectivity

sp

ee

du

p

Sample 1

Sample 2

Sample 3

Grid Collector Speeds Up Actual Analysis

Speedup = time used with existing filtering mechanism / time used with GC selecting the same events

Tested on flow analysis jobs Test data set contains 51 MuDST files, 8 GB, 25,000 events (P04ij) Test data uses an efficient organization that enhances the filtering

mechanism – reads part of the event data for filtering

1

2

3

4

5

6

0.01 0.1 1

selectivity

sp

ee

du

p

elapsed CPU

Real analysis jobs typically include its own filtering mechanisms

Real analysis jobs may also spend significant amount of time perform computation

On a set of “real” analysis jobs that typically select about 10% of events, using Grid Collector has a speedup of 2 for CPU time, 1.4 for elapsed time.

Speeding all jobs by 1.4 means the same computer center can accommodate 40% more analysis jobs

Grid Collector Enables Hard Analysis Jobs

Searching for anti-3He Lee Barnby, Birmingham Initial study identified

collision events that possibly contain anti-3He, need further analysis (2000)

Searching for strangelet Aihong Tang, BNL Initial study identified

events that may indicate existence of strangelets, need further investigation (2000)

Without Grid Collector, one has to retrieve every file from HPSS and scan them for the wanted events – may take weeks or months, NO ONE WANTS TO DO IT

With Grid Collector, both completed in a day

Summary

Grid Collector Makes use of two distinct technologies,

FastBit, And SRM (Storage Resource Manager)

To speed up common analysis jobs where files are already on disk,

And, enable difficult analysis jobs where some files may not be on disk.

Contact Information John Wu John.Wu@nersc.gov Jerome Lauret JLauret@bnl.gov