A Multi-Level Parallel Implementation of a Program for Finding Frequent Patterns in a Large Sparse Graph

Steve Reinhardt, Interactive Supercomputing sreinhardt@interactivesupercomputing.com

George Karypis, Dept. of Computer Science, University of Minnesota

Outline Problem definition Prior work Problem and Approach Results Issues and Conclusions

Graph Datasets Flexible and powerful representation

Evidence extraction and link discovery (EELD) Social Networks/Web graphs Chemical compounds Protein structures Biological Pathways Object recognition and retrieval Multi-relational datasets

Finding Patterns in GraphsMany Dimensions

Structure of the graph dataset many small graphs

graph transaction setting one large graph

single-graph setting Type of patterns

connected subgraphs induced subgraphs

Nature of the algorithm Finds all patterns that satisfy the minimum support requirement

Complete Finds some of the patterns

Incomplete Nature of the pattern’s occurrence

The pattern occurs exactly in the input graph Exact algorithms

There is a sufficiently similar embedding of the pattern in the graph Inexact algorithms

MIS calculation for frequency exact approximate upper bound

Algorithm vertical (depth-first) horizontal (breadth-first)

M. Kuramochi and G. Karypis. Finding frequent patterns in a large sparse graph. In SIAM International Conference on Data Mining (SDM-04), 2004. http://citeseer.ist.psu.edu/article/kuramochi04finding.html

Single Graph Setting

Find all frequent subgraphs from a single sparse graph. Choice of frequency definition

Input Graph Size 7Frequency = 6

Size 6Frequency = 1

vSIGRAM: Vertical Solution Candidate generation by extension

Add one more edge to a current embedding. Solve MIS on embeddings in the same equivalence class. No downward-closure-based pruning

Two important components Frequency-based pruning of extensions Treefication based on canonical labeling

vSIGRAM: Connection Table Frequency-based pruning. Trying every possible extension is expensive and inefficient.

A particular extension might have been tested before. Categorize extensions into equivalent classes (in terms of

isomorphism), and record if each class is frequent or not. If a class becomes infrequent, never try it in later exploration.

Parallelization Two clear sources of parallelism in the algorithm

Amount of parallelism from each source not known in advance The code is typical C code

structs, pointers, frequent mallocs/frees of small areas, etc. nothing like the “Fortran”-like (dense linear algebra) examples shown for

many parallel programming methods Parallel structures need to accommodate dynamic parallelism

Dynamic specification of parallel work Dynamic allocation of processors to work

Chose OpenMP taskq/task constructs Proposed extensions to OpenMP standard Support parallel work being defined in multiple places in a program, but

be placed on a single conceptual queue and executed accordingly ~20 lines of code changes in ~15,000 line program

Electric Fence was very useful in finding coding errors

Algorithmic ParallelismvSiGraM (G, MIS_type, f)1. F ← 2. F1 ← all frequent size-1 subgraphs in G3. for each F1 in F1 do4. M(F1) ← all embeddings of F1

5. for each F1 in F1 do // high-level parallelism6. F ← F vSiGraM-Extend(F1, G, f)return F

vSiGraM-Extend(Fk, G , f)1. F ← 2. for each embedding m in M(Fk) do // low-level parallelism3. Ck+1 ← C k+1 {all (k+1)-subgraphs of G containing m}4. for each Ck+1 in Ck+1 do5. if Fk is not the generating parent of Ck+1 then6. continue7. compute Ck+1.freq from M(Ck+1)8. if Ck+1.freq < f then9. continue10. F ← F vSiGraM-Extend(Ck+1, G, f)11.return F

Simple Taskq/Task Examplemain(){ int val;#pragma intel omp taskq val = fib(12345);}

fib(int n){ int partret[2]; if (n>2)#pragma intel omp task for(i=n-2; i<n; i++) { partret[n-2-i] = fib(i); } return (partret[0] + partret[1]); } else { return 1; }}

High-Level Parallelism with taskq/task// At the bottom of expand_subgraph, after all child

// subgraphs have been identified, start them all.

#pragma intel omp taskq

for (ii=0; ii<sg_set_size(child); ii++) {

#pragma intel omp task captureprivate(ii)

{

SubGraph *csg = sg_set_at(child,ii);

expand_subgraph(csg, csg->ct, lg, ls, o);

} // end-task

}

Low-Level Parallelism with taskq/task#pragma omp parallel shared(nt, priv_es) {#pragma omp master { nt = omp_get_num_threads(); //#threads in par priv_es = (ExtensionSet **)kmp_calloc(nt, sizeof(ExtensionSet *)); } #pragma omp barrier#pragma intel omp taskq { for (i = 0; i < sg_vmap_size(sg); i++) {#pragma intel omp task captureprivate(i) { int th = omp_get_thread_num(); if (priv_es[th] == NULL) { priv_es[th] = exset_init(128); } expand_map(sg, ct, ams, i, priv_es[th], lg); } } } } // end parallel section; next loop is serial reduction for (i=0; i < nt; i++) { if (priv_es[i] != NULL) { exset_merge(priv_es[i],es); } } kmp_free(priv_es);}

Implementation due to Grant Haab and colleagues from Intel OpenMP library group

Experimental Results SGI Altix™ 32 Itanium2™ sockets (64 cores), 1.6GHz 64 GBytes (though not memory limited) Linux No special dplace/cpuset configuration

Minimum frequencies chosen to illuminate scaling behavior, not provide maximum performance

Dataset 1 - ChemicalGraph Frequency Type of

ParallelismNumber of processors

1 2 4 8 16 30 60

Time in seconds (speed-up)

dtp

500

High31.94 17.01

(2.03)14.76 (2.40)

13.89(2.58)

14.00(2.56)

13.97(2.57)

Low32.51(0.98)

31.52(1.01)

37.95(0.83)

42.18(0.74)

49.56(0.63)

Both17.52(1.96)

14.88(2.37)

15.80(2.21)

29.85(1.08)

44.37(0.70)

100

High93.96 48.86

(1.97)27.12(3.71)

16.82(6.39)

15.05(7.29)

14.52(7.61)

Low94.36(1.00)

92.18(1.02)

112.17(0.83)

133.40(0.70)

116.31(0.80)

Both48.38(1.99)

27.27(3.69)

61.52(1.55)

315.94(0.29)

281.83(0.33)

50

High282.15 142.02

(2.00)62.73(4.64)

34.44(8.76)

19.40(16.56)

15.06(22.27)

15.80(21.03)

Low283.19(1.00)

293.6(0.96)

400.55(0.70)

262.82(1.07)

197.27(1.44)

Both140.47(2.03)

81.18(3.55)

242.09(1.17)

513.39(0.55)

581.04(0.48)

Dataset 2 – aviationGraph Frequency Type of

ParallelismNumber of processors

1 2 4 8 16 30 60

Time in seconds (speed-up)

air1

1750

High358.27 54.92

(7.19)21.74

(22.30)18.85

(27.29)

Low171.04(2.13)

1500 High771.82 112.30

(7.20)39.40

(22.89)33.99

(27.30)

1250 High1503.49 209.08

(7.37)67.54

(24.31)56.56

(29.58)

1000High

3909.95 490.38(8.06)

155.33(26.13)

158.14(25.65)

Performance of High-level Parallelism When sufficient quantity of work (i.e., frequency

threshold is low enough) Good speed-ups to 16P Reasonable speed-ups to 30P Little or no benefit above 30P No insight into performance plateau

Poor Performance of Low-level Parallelism

Several possible effects ruled out Granularity of data allocation Barrier before master-only reduction

Source: highly variable times for register_extension ~100X slower in parallel than serial, … but different instances from execution to execution Apparently due to highly variable run-times for malloc Not understood

Issues and Conclusions OpenMP taskq/task were straightforward to use in this

program and implemented the desired model Performance was good to a medium range of processor

counts (best 26X on 30P) Difficult to gain insight into lack of performance

High-level parallelism 30P and above Low-level parallelism

Backup

Datasets

DatasetConnected

ComponentsVertices Edges

Vertex

Labels

Edge

Labels

Aviation 2,703 101,185 98,482 6,173 51

Citation 16,999 29,014 42,064 50 12

VLSI 2,633 12,752 11,542 23 1

Generally, vSIGRAM is 2-5 times faster than hSIGRAM (with exact and upper bound MIS)

Largest pattern contained 13 edges.

Aviation Dataset

But, hSIGRAM can be more efficient especially with upper bound MIS (ub). Largest pattern contained 16 edges.

Citation Dataset

VLSI Dataset

Exact MIS never finished. Longest pattern contained 5 edges (constraint).

Comparison with SUBDUE

Dataset

SUBDUE vSIGRAM (approximate MIS)

Freq. SizeRuntime

[sec]Freq.

Largest

SizePatterns

Runtime

[sec]

Credit

341

395

387

6

5

5

517200 9 11,696 4

20 13 613,884 461

DTP

4,957

4,807

1,950

2

2

6

1,525500 7 190 20

10 21 112,535 311

VLSI

773

773

244

1

1

1

16200 5 137 3

25 5 1,452 18

Similar results with SEuS

Summary With approximate and exact MIS, vSIGRAM is 2-5 times faster than

hSIGRAM. With upper bound MIS, however, hSIGRAM can prune a larger

number of infrequent patterns. The downward closure property plays the role.

For some datasets, using exact MIS for frequency counting is just intractable.

Compared to SUBDUE, SIGRAM finds more and longer patterns in shorter amount of runtime.

Thank You! Slightly longer version of this paper is also available as a

technical report.

SIGRAM executables will be available for download soon from http://www.cs.umn.edu/~karypis/pafi/

Complete Frequent Subgraph Mining—Existing Work So Far Input: A set of graphs (transactions) + support threshold Goal: Find all frequently occurring subgraphs in the input dataset.

AGM (Inokuchi et al., 2000), vertex-based, may not be connected. FSG (Kuramochi et. al., 2001), edge-based, only connected subgraphs AcGM (Inokuchi et al., 2002), gSpan (Yan & Han, 2002), FFSM (Huan et al.,

2003), etc. follow FSG’s problem definition. Frequency of each subgraph

The number of supporting transactions. Does not matter how many embeddings are in each transaction.

What is the reasonable frequency definition?

Two reasonable choices: The frequency is determined by the total number of embeddings.

Not downward closed. Too many patterns. Artificially high frequency of certain patterns.

The frequency is determined by the number of edge-disjoint embeddings (Vanetik et al, ICDM 2002).

Downward closed. Since each occurrence utilizes different sets of edges, occurrence

frequencies are bounded. Solved by finding the maximum independent set (MIS) of the

embedding overlap graph.

Embedding Overlap and MIS

Edge-disjoint embeddings { E1, E2, E3 }

{ E1, E2, E4 }

Create an overlap graph and solve MIS Vertex Embedding Edge Overlap

E1E2

E3 E4

OK. Definition is Fine, but … MIS-based frequency seems reasonable.

Next question: How to develop mining algorithms for the single graph setting.

How to Handle Single Graph Setting?

Issue 1: Frequency counting Exact MIS is often intractable.

Issue 2: Choice of search scheme Horizontal (breadth-first) Vertical (depth-first)

Issue 1: MIS-Based Frequency We considered approximate (greedy) and upper bound

MIS too. Approximate MIS may underestimate the frequency. Upper bound MIS may overestimate the frequency.

MIS is NP-complete and not be approximated. Practically simple greedy scheme works pretty well.

Halldórsson and Radhakrishnan. Greed is good, 1997.

Issue 2: Search Scheme Frequent subgraph mining

Exploration in the lattice of subgraphs Horizontal

Level-wise Candidate generation and pruning

Joining Downward closure property

Frequency counting Vertical

Traverse the lattice as if it were a tree.

hSIGRAM: Horizontal Method Natural extension of FSG to the single graph setting. Candidate generation and pruning.

Downward closure property Tighter pruning than vertical method

Two-phase frequency counting All embeddings by subgraph isomorphism

Anchor edge list intersection, instead of TID list intersection. Localize subgraph isomorphism

MIS for the embeddings Approximate and upper bound MIS give subset and superset respectively.

TID List Recap

Lattice of Subgraphs

size k

size k + 1

TID( ) = { T1, T3 }TID( ) = { T1, T2, T3 }

T1

T2

T3

TID( ) TID( ) ∩ TID( ) ∩ TID( ) = { T1, T3 }

TID( ) = { T1, T2, T3 }

Anchor Edges

Each subgraph must appear close enough together. Keep one edge for each.

Complete embeddings require too much memory. Localize subgraph isomorphism.

Lattice of Subgraphs

size k

size k + 1

Treefication

Lattice of Subgraphs Treefied Lattice

size k - 1

size k

size k + 1

: a node in the search space (i.e., a subgraph) Based on subgraph/supergraph relation Avoid visiting the same node in the lattice more than once.