go.indeed.com/IndeedEngTalks

Machine Learning at Indeed

Scaling Decision Trees

Andrew HudsonCTO

I help people get jobs.

Indeed is aSearch Engine for Jobs

Which jobs to show?

18,749 jobs

Which jobs to show?

Maximize job seeker’s chance to get the job

Which jobs to show?

Maximize job seeker’s chance to get the job

● Will job seeker click on the job?● Is the job still available?● Will job seeker apply to the job?● Is job seeker qualified for the job?

Which jobs to show?

Maximize job seeker’s chance to get the job

● Will job seeker click on the job?● Is the job still available?● Will job seeker apply to the job?● Is job seeker qualified for the job?

How?

Log job seeker behavior

Analyze logs, what best explains why they clicked on some jobs and not on others?

May help predict future behavior

How?

Log job seeker behavior

Analyze logs, what best explains why they clicked on some jobs and not on others?

May help predict future behavior

Supervised learning

Supervised Learning Approaches

Neural networks Bayesian methods Decision trees

Genetic programming

Logistic model tree Nearest neighbor

Support Vector Machines

Random forests Boosting

Bagging Regression Ensemble methods

Supervised Learning Approaches

Neural networks Bayesian methods Decision trees

Genetic programming

Logistic model tree Nearest neighbor

Support Vector Machines

Random forests Boosting

Bagging Regression Ensemble methods

Supervised Learning Approaches

Decision trees

Genetic programming

Logistic model tree

Random forests Boosting

Bagging Ensemble methods

Decision Trees

What is a Decision Tree?

A tree like structure that presents a relevant sequence of questions which determine a path and ultimately some outcome or prediction

I’m Thinking About Buying a Laptop

I’m Thinking About Buying a Laptop

Is quality important?

I’m Thinking About Buying a Laptop

ASUSIs quality important?NO

I’m Thinking About Buying a LaptopASUS -or whatever woot hasIs quality important?

NO

I’m Thinking About Buying a Laptop

YES

ASUS -or whatever woot has

NO

Want to run linux?

Is quality important?

I’m Thinking About Buying a Laptop

MACBOOKWant to run linux?

YES

ASUS -or whatever woot hasIs quality important?

NO

NO

YES

I’m Thinking About Buying a Laptop

LENOVO

MACBOOKWant to run linux?

YES

ASUS -or whatever woot hasIs quality important?

NO

NO

I’m Thinking About Buying a Laptop

DELLIDGAF

SYSTEM76HELLYESYES

LENOVO

MACBOOKWant to run linux?

YES

ASUS -or whatever woot hasIs quality important?

NO

NO

Benefits of Decision Trees

Algorithm relatively simple to understand and implement

Model produced also human understandable

Decision Tree Learning

Programmatic creation of decision trees

Decision Tree Learning

Given a set of documents, split it into two or more subsets that optimize some criteria

Repeat this process until a set can no longer be split

Titanic Example

1309 passengers500 survivors38.2% survival rate

What best explains who survived?

classclass of ticket; first, second or third

fsizefamily size; number of family members onboard

gendermale or female

What best explains who survived?

1309 passengers500 survivors

38.2% survival

1309 passengers500 survivors

38.2% survival

class = 1

1309 passengers500 survivors

38.2% survival

class = 1323 passengers

200 survivors61.9% survival

class ≠ 1986 passengers

300 survivors30.4% survival

class = 1323 passengers

200 survivors61.9% survival

1309 passengers500 survivors

38.2% survival

class ≠ 1986 passengers

300 survivors30.4% survival

class = 1323 passengers

200 survivors61.9% survival

Score = ?1309 passengers

500 survivors38.2% survival

Score

conditional entropy

Conditional Entropy as Score

lower conditional entropy↓

less uncertainty about prediction based on term

class ≠ 1986 passengers

300 survivors30.4% survival

class = 1323 passengers

200 survivors61.9% survival

Score = 0.62671309 passengers

500 survivors38.2% survival

class ≠ 1986 passengers

300 survivors30.4% survival

class = 1323 passengers

200 survivors61.9% survival

Score = 0.6267 Best Score:0.6267, class = 1

1309 passengers500 survivors

38.2% survival

class = 1

Best Score:0.6267, class = 1

1309 passengers500 survivors

38.2% survival

Best Score:0.6267, class = 1

1309 passengers500 survivors

38.2% survival

class ≤ 2

Best Score:0.6267, class = 1

1309 passengers500 survivors

38.2% survival

class ≤ 2600 passengers

319 survivors53.2% survival

Best Score:0.6267, class = 1

1309 passengers500 survivors

38.2% survival

class ≤ 2600 passengers

319 survivors53.2% survival

class > 2709 passengers

181 survivors25.5% survival

Best Score:0.6267, class = 1

1309 passengers500 survivors

38.2% survival

class ≤ 2600 passengers

319 survivors53.2% survival

class > 2709 passengers

181 survivors25.5% survival

Score = 0.6244

Best Score:0.6244, class ≤ 2

1309 passengers500 survivors

38.2% survival

class ≤ 2600 passengers

319 survivors53.2% survival

class > 2709 passengers

181 survivors25.5% survival

Score = 0.6244

Best Score:0.6244, class ≤ 2

1309 passengers500 survivors

38.2% survival

class ≠ 3600 passengers

319 survivors53.2% survival

class = 3709 passengers

181 survivors25.5% survival

Score = 0.6244

Best Score:0.6244, class ≤ 2

1309 passengers500 survivors

38.2% survival

gender = female

Best Score:0.6244, class ≤ 2

1309 passengers500 survivors

38.2% survival

gender = female466 passengers

339 survivors72.7% survival

Best Score:0.6244, class ≤ 2

1309 passengers500 survivors

38.2% survival

gender ≠ female843 passengers

161 survivors19.1% survival

gender = female466 passengers

339 survivors72.7% survival

Best Score:0.6244, class ≤ 2

1309 passengers500 survivors

38.2% survival

gender ≠ female843 passengers

161 survivors19.1% survival

gender = female466 passengers

339 survivors72.7% survival

Score = 0.5525

Best Score:0.5525, gender=f

1309 passengers500 survivors

38.2% survival

gender ≠ female843 passengers

161 survivors19.1% survival

gender = female466 passengers

339 survivors72.7% survival

Score = 0.5525

Best Score:0.5525, gender=f

1309 passengers500 survivors

38.2% survival

fsize = 0790 passengers

239 survivors30.3% survival

Score = 0.6448

fsize ≠ 0519 passengers

261 survivors50.3% survival

Best Score:0.5525, gender=f

19.1% survival

72.7% survival

gender=male843 passengers

161 survivors19.1% survival

gender=male843 passengers

161 survivors19.1% survival

class = 1179 passengers

61 survivors34.1% survival

gender=male843 passengers

161 survivors19.1% survival

class = 1179 passengers

61 survivors34.1% survival

class ≠ 1664 passengers

100 survivors15.1% survival

Score = 0.4700

class = 1 class ≠ 1

class = 1 class ≠ 1

15.1% survival

34.1% survival

38.2%

72.7%19.1%MALE

38.2%

FEMALE

72.7%19.1%MALE

34.1%15.1%

38.2%

FEMALE

CLASS≠1 CLASS=1

72.7%19.1%MALE

34.1%15.1%

13.1% 33.9%

38.2%

FEMALE

CLASS≠1 CLASS=1

FSIZE≠2 FSIZE=2

72.7%19.1%MALE

34.1%15.1%

13.1% 33.9%

93.2%49.1%

38.2%

FEMALE

CLASS≠1 CLASS=1 CLASS>2 CLASS<=2

FSIZE≠2 FSIZE=2

72.7%19.1%MALE

34.1%15.1%

13.1% 33.9%

93.2%49.1%

24.4% 54.9%

38.2%

FEMALE

CLASS≠1 CLASS=1 CLASS>2 CLASS<=2

FSIZE≠2 FSIZE=2 FSIZE>2 FSIZE<=2

Predicting Click Probabilities

Passenger → Job ImpressionSurvived → Clicked on Job

For each candidate job, follow path through tree then take click through rate of terminal node

sales account manager

representative manager associate

outside service inside

YES

2.9%

YES

4.4%

YES

2.1%

NO

NO

YES

2.9%

NO

YES

1.8%

NO

1.9%

NO

2.6%

NO NO NO

4.6%

NO

3.8%

YES

YES

5.1%

YES

YES

Simplified Decision Tree for query="sales"

sales

YES

representative

YES

outsideNO

serviceNO

insideNO

4.6%

job title = “sales representative”

YES YES

2.9%

YES

4.4%5.1%

NO

YES

3.8%

YES

2.1%

accountNO

manager

NO

YES

2.9%

managerNO

YES

1.8%

associate

NO

1.9%

NO

2.6%

job title = “account executive”

YES YES

2.9%

YES

4.4%5.1%

NO

YES

3.8%

YES

2.1%

accountNO

manager

NO

YES

2.9%

managerNO

YES

1.8%

associate

NO

1.9%

NO

2.6%

sales

3.8%

account

NO NO NO

YES

representative

YES

outside service inside 4.6%

YES

2.9%

YES

4.4%

YES

2.1%

NO

manager

NO

YES

2.9%

managerNO

YES

1.8%

associate

NO

1.9%

NO

2.6%

NO NO NO

service inside 4.6%

salesNO

representative

outside

5.1%

job title = “outside sales representative”

account

3.8%YES

YES

YES

YES

YES

manager

1.8%

associate

job title = “sales associate”

YES

2.1%

NO

managerNO

1.9%NO

account

3.8%

YES

YES YES

2.9%

YES

4.4%5.1%

NO NO NO

outside service inside 4.6%

sales

representative

YES

NO

2.6%

YES

2.9%

NO NO

YES

sales

representative

outside service

4.4%

inside

job title = “inside sales representative”

YES

2.1%

NO

manager

NO

YES

2.9%

managerNO

YES

1.8%

associate

NO

1.9%

NO

2.6%

NO

account

3.8%

YES

YES YES

2.9%5.1%

4.6%

YES

NO NO NO

YES

YES

NO

2.9%

manager

job title = “sales manager”

YES

2.1%

NO

managerNO

1.9%NO

account

3.8%

YES

YES YES

2.9%

YES

4.4%5.1%

NO NO NO

outside service inside 4.6%

YES

sales

representative

YES

YES

1.8%

associateNO

2.6%NO

YES

job title = “sales consultant”

YES YES

2.9%

YES

4.4%5.1%

NO NO NO

outside service inside 4.6%

YES

YES

2.9%

YES

1.8%

YES

2.1%

NO

managerNO

1.9%NO

account

3.8%

YES

manager associate

sales

representative

YES

NO

2.6%NO NO

sales account manager

job title = “store manager”

YES YES

2.9%

YES

4.4%5.1%

NO

YES

2.9%

managerNO

YES

1.8%

associateNO

2.6%

NO NO NO

YES

representative

YES

outside service inside 4.6%

NO

1.9%

YES

2.1%3.8%

YES

NONO

2.9%

job title = “service sales representative”

YES

2.1%

NO

manager

NO

YES

2.9%

managerNO

YES

1.8%

associate

NO

1.9%

NO

2.6%

NO

account

3.8%

YES

YES

4.4%

NO

inside 4.6%NO

YES

5.1%

sales

representative

outside service

YES

NO

YES

YES

job title = “customer service representative”

YES YES

2.9%

YES

4.4%5.1%

NO

YES

2.9%

managerNO

YES

1.8%

associateNO

2.6%

NO NO NO

representative

YES

outside service inside 4.6%

YES

sales accountNONO

managerNO

1.9%

YES

2.1%3.8%

YES

Final CTR Predictions

5.1% outside sales representative4.6% sales representative4.4% inside sales representative3.8% account executive2.9% sales manager2.9% service sales representative2.6% sales consultant2.1% store manager1.9% customer service representative1.8% sales associate

Single Machine Implementation

Overview

Tree Building Strategies

One node at a time- depth first- breadth first

1

Depth First

1

2 3

Depth First

1

3

Depth First

2

4 5

1

3

Depth First

2

5

6 7

4

1

3

Depth First

2

54

6 7

1

3

Depth First

2

54

6 7

1

3

Depth First

2

4

6 7

5

1

Depth First

2

4

6 7

5

3

8 9

1

Breadth First

1

2 3

Breadth First

1

3

Breadth First

2

4 5

1

Breadth First

2

4 5

3

6 7

1

Breadth First

2

5

3

6 7

8 9

4

1

Breadth First

2 3

6 7

8 9

4 5

1

Breadth First

2 3

8 9

4 5

10 11

6 7

1

Breadth First

2 3

8 9

4 5

10 11

6

12 13

7

Tree Building Strategies

One node at a time- depth first- breadth first

One layer at a time, all nodes simultaneous

1

1iter #1

1

2 3

iter #1

1

2 3

iter #1

iter #2

1

2

4

3

5 6 7

iter #1

iter #2

iter #3

1

2 3

5 6 7

iter #1

iter #2

4

8 9 0 10 11 12 13

iter #3

1

2

4

3

5 6 7

iter #1

iter #2

8 9 0 10 11 12 13

iter #3

1

2

4

3

5 6 7

iter #1

iter #2

iter #48 9 0 10 11 12 13

iter #3

1

2

4

3

5 6 7

iter #1

iter #2

8 9 0 10 11 12 13

iter #3

1

2

4

3

5 6 7

iter #1

iter #2

Data Formatid class fsize gender survived

0 1 0 f 1

1 1 3 m 1

2 1 3 f 0

3 1 3 m 0

4 1 3 f 0

5 1 0 m 1

6 1 1 f 1

7 1 0 m 0

8 1 2 f 1

9 1 0 m 0

id class fsize gender survived

10 1 1 m 0

11 1 1 f 1

12 1 0 f 1

13 1 0 f 1

14 1 0 m 1

15 1 0 m 0

16 1 1 m 0

17 1 1 f 1

18 1 0 f 1

19 1 0 m 0

….

Data Format

Create an inverted index

Key to efficiently building one layer at a time

Inverted Index

Maps terms to the list of documents that contain that term

Terms and docs stored in sorted order

Inverted Index

class=1 → 0,1,2,3,4,5,6,7,8,9,10,11,12,13….class=2 → 323,324,325,326,327,328,329….class=3 → 600,601,602,603,604,605,606….

Inverted Index

class=1 → 0,1,2,3,4,5,6,7,8,9,10,11,12,13….class=2 → 323,324,325,326,327,328,329….class=3 → 600,601,602,603,604,605,606….

Field

Inverted Index

class=1 → 0,1,2,3,4,5,6,7,8,9,10,11,12,13….class=2 → 323,324,325,326,327,328,329….class=3 → 600,601,602,603,604,605,606….

Term

Inverted Index

class=1 → 0,1,2,3,4,5,6,7,8,9,10,11,12,13….class=2 → 323,324,325,326,327,328,329….class=3 → 600,601,602,603,604,605,606….

Docs

Inverted Index

class=1 → 0,1,2,3,4,5,6,7,8,9,10,11,12,13….class=2 → 323,324,325,326,327,328,329….class=3 → 600,601,602,603,604,605,606….

Docs

Inverted Index

class=1 → 0,1,2,3,4,5,6,7,8,9,10,11,12,13….class=2 → 323,324,325,326,327,328,329….class=3 → 600,601,602,603,604,605,606….

Docs

Inverted Index

fsize=0 → 0,5,7,9,12,13,14,15,18,19,22….fsize=1 → 6,10,11,16,17,26,27,36,49,50….fsize=2 → 8,20,21,42,76,77,78,79,81,82….fsize=3 → 1,2,3,4,54,55,56,57,90,339….fsize=4 → 249,250,251,252,253,449,806….….

Inverted Index

gender=f → 0,2,4,6,8,11,12,13,17,18,21….gender=m → 1,3,5,7,9,10,14,15,16,19,20….

Inverted Index

survived=0 → 2,3,4,7,9,10,15,16,19,25….survived=1 → 0,1,5,6,8,11,12,13,14,17….

Inverted Index Implementations

Lucene

Flamdex

Primary Lookup Tables

groups[doc]Where in the tree each doc isInitialized to all ones, all docs start in root

values[doc]Value to be classified, for each docIn this case it’s 1 if survived, 0 otherwise

Primary Lookup Tables

values[doc]

Constructed from an inverted index of the values

Invert the field of interest (e.g. survived)

Main Loop Overview

foreach fieldforeach term

get group statsevaluate splits

apply best splitsrepeat n times or until no more splits found

Main Loop - First Iteration

foreach field (class, fsize, gender)

Main Loop - First Iteration

foreach field (class, fsize, gender)foreach term (class=1,class=2,class=3...)

Main Loop - First Iteration

foreach field (class, fsize, gender)foreach term (class=1,class=2,class=3...)

get group stats

Get Group Stats

count[grp]Count of how many documents within that group contain current term, initialized to zeros

vsum[grp]Summation of the value to be classified from the documents within that group that contain current term, initialized to zeros

Get Group Stats

for current field/term

Get Group Stats

for current field/termforeach doc

Get Group Stats

for current field/termforeach doc

grp = grps[doc]

Get Group Stats

for current field/termforeach doc

grp = grps[doc]if grp == 0 skip

Get Group Stats

for current field/termforeach doc

grp = grps[doc]if grp == 0 skipcount[grp]++vsum[grp] += vals[doc]

Get Group Stats

for current field/term (class=1)

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skip

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 0, vsum[1] = 0

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 1, vsum[1] = 1

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 2, vsum[1] = 2

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 3, vsum[1] = 2

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 4, vsum[1] = 2

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 5, vsum[1] = 2

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 6, vsum[1] = 3

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[1] = 323, vsum[1] = 200

1309 passengers500 survivors

38.2% survival

class = 1

1309 passengers500 survivors

38.2% survival

class = 1

Group 1

1309 passengers500 survivors

38.2% survival

class = 1

Group 1

1309 passengers500 survivors

38.2% survival

class = 1323 passengers count[1]

Group 1

1309 passengers500 survivors

38.2% survival

class = 1323 passengers

200 survivors

count[1]

vsum[1]

Group 1

Get Group Stats

for current field/term (class=2)foreach doc (323,324,325,326,327,328,329...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (0,1,0,0,0,0,1,0,1…)

…count[1] = 277, vsum[1] = 119

Get Group Stats

for current field/term (class=3)foreach doc (600,601,602,603,604,605,606...)

grp = grps[doc] (1,1,1,1,1,1,1,1,1…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (0,0,0,1,1,1,1,1,0…)

…count[1] = 709, vsum[1] = 181

Main Loop - First Iteration

foreach field (class, fsize, gender)foreach term (class=1,class=2,class=3...)

get group statsevaluate splits

Evaluate Splits

Consider current field/term as a potential split for each group

1) check if split is admissiblebalance check, significance check

2) score the splitconditional entropy or some other heuristic

3) keep best scoring split

Evaluate Splits

totalcount[group] / totalvalue[group]Total number of documents and total values for each group, i.e. # passengers / # survivors

bestsplit[group] / bestscore[group]Current best split and score for each group, initially nulls

foreach field/term (class=1)get group stats (count[1]=323,vsum[1]=200)foreach group

if not admissible( … ) skipscore = calcscore(cnt[grp], vsum[grp],

totcnt[grp], totval[grp])if score < bestscore[grp]

bestscore[grp] = scorebestsplit[grp] = field/term

foreach field/term (class=1)get group stats (count[1]=323,vsum[1]=200)foreach group

if not admissible( … ) skipscore = calcscore(cnt[grp], vsum[grp],

totcnt[grp], totval[grp])if score < bestscore[grp]

bestscore[grp] = scorebestsplit[grp] = field/term

Main Loop - First Iteration

foreach field (class, fsize, gender)foreach term (class=1,class=2,class=3...)

get group statsevaluate splits

apply best splits (bestsplit[1]=“gender=f”)

Apply Best Splits

Each split is a combination of a target group, a condition, a positive destination group, and a negative destination group

Apply Best Splits

Each split is a combination of a target group, a condition, a positive destination group, and a negative destination group

target group: 1 1

Apply Best Splits

Each split is a combination of a target group, a condition, a positive destination group, and a negative destination group

target group: 1condition: gender=female

1

Apply Best Splits

Each split is a combination of a target group, a condition, a positive destination group, and a negative destination group

target group: 1condition: gender=femalepositive group: 3

3

1

Apply Best Splits

Each split is a combination of a target group, a condition, a positive destination group, and a negative destination group

target group: 1condition: gender=femalepositive group: 3negative group: 2 2 3

1

Apply Best Splits

Each split is a combination of a target group, a condition, a positive destination group, and a negative destination group

target group: 1condition: gender=femalepositive group: 3negative group: 2 2 3

1

Apply Best Splits

Using inverted index, iterate over docs that match split condition

If current document is in targeted group, move it to the positive group

At the end, move anything left in target group to negative group

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 1 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 1 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 1 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 1 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 1 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 1 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 1 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 1 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 1 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 1 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 1 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 1 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 1 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 1 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 3 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 1 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 3 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 3 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 1 group[15] = 1group[2] = 3 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 1group[4] = 3 group[11] = 1 group[18] = 1group[5] = 1 group[12] = 1 group[19] = 1group[6] = 1 group[13] = 1 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 3 group[15] = 1group[2] = 3 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 3group[4] = 3 group[11] = 3 group[18] = 3group[5] = 1 group[12] = 3 group[19] = 1group[6] = 3 group[13] = 3 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 3 group[15] = 1group[2] = 3 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 3group[4] = 3 group[11] = 3 group[18] = 3group[5] = 1 group[12] = 3 group[19] = 1group[6] = 3 group[13] = 3 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 1 group[14] = 1group[1] = 1 group[8] = 3 group[15] = 1group[2] = 3 group[9] = 1 group[16] = 1group[3] = 1 group[10] = 1 group[17] = 3group[4] = 3 group[11] = 3 group[18] = 3group[5] = 1 group[12] = 3 group[19] = 1group[6] = 3 group[13] = 3 group[20] = 1

Apply Best Splits

gender=f -> 0,2,4,6,8,11,12,13,17,18,21,23….

group[0] = 3 group[7] = 2 group[14] = 2group[1] = 2 group[8] = 3 group[15] = 2group[2] = 3 group[9] = 2 group[16] = 2group[3] = 2 group[10] = 2 group[17] = 3group[4] = 3 group[11] = 3 group[18] = 3group[5] = 2 group[12] = 3 group[19] = 2group[6] = 3 group[13] = 3 group[20] = 2

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splitsrepeat n times or until no more splits found

1

iter #1

1

iter #1

gender = female

1

2 3

iter #1

gender = femalegender ≠ female

1

iter #1

iter #22 3

iter #1

1

Main Loop - Second Iteration

foreach field (class, fsize, gender)foreach term (class=1,class=2,class=3...)

get group stats

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (3,2,3,2,3,2,3,2,3…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

Get Group Stats

for current field/term (class=1)foreach doc (0,1,2,3,4,5,6,7,8...)

grp = grps[doc] (3,2,3,2,3,2,3,2,3…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,1,0,0,0,1,1,0,1…)

…count[2] = 179, vsum[2] = 61count[3] = 144, vsum[3] = 139

Get Group Stats

for current field/term (class=2)foreach doc (323,324,325,326,327,328,329...)

grp = grps[doc] (2,3,2,2,2,2,3,2,2…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (0,1,0,0,0,0,1,0,1…)

…count[2] = 171, vsum[2] = 25count[3] = 106, vsum[3] = 94

Get Group Stats

for current field/term (class=3)foreach doc (600,601,602,603,604,605,606...)

grp = grps[doc] (2,2,2,3,3,2,2,3,2…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (0,0,0,1,1,1,1,1,0…)

…count[2] = 493, vsum[2] = 75count[3] = 216, vsum[3] = 106

Get Group Stats

for current field/term (gender=female)foreach doc (0,2,4,6,8,11,12,13,17,18,21,23….)

grp = grps[doc] (3,3,3,3,3,3,3,3,3,3,3,3…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,0,0,1,1,1,1,1,1…)

…count[2] = 0, vsum[2] = 0count[3] = 467, vsum[3] = 339

Get Group Stats

for current field/term (gender=male)foreach doc (1,3,5,7,9,10,14,15,16,19,20,22...)

grp = grps[doc] (2,2,2,2,2,2,2,2,2,2,2…)if grp == 0 skipcount[grp]++vsum[grp] += vals[doc] (1,0,1,0,0,0,1,0,0…)

…count[2] = 844, vsum[2] = 161count[3] = 0, vsum[3] = 0

What AboutInequality Splits?

e.g. class ≤ 2

Main Loop + Inequality Splits

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Main Loop + Inequality Splits

foreach fieldreset inequality statsforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Main Loop + Inequality Splits

foreach fieldreset inequality statsforeach term

get group statsupdate inequality statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Main Loop + Inequality Splits

foreach fieldreset inequality statsforeach term

get group statsupdate inequality statsevaluate splitsevaluate inequality splits

apply best splits for each grouprepeat n times or until no more splits found

Scalability

Performs quite well on a single machine

Worked well for a while, but started to hit limits

Ultimately needed to distribute to multiple machines

Multiple Machine Implementation

Hadoop?

Hadoop

Experimented with using Hadoop

Each level took five sequential map reduce jobs

Much slower than single machine; repeatedly writes intermediate data and lots of shuffling

Hadoop

Experimented with using Hadoop

Each level took five sequential map reduce jobs

Much slower than single machine; repeatedly writes intermediate data and lots of shuffling

Hadoop not great for iterative algorithms

Partition Data

Inverted Index

Inverted Index

Inverted Index

Inverted Index

Shard 1 Shard 2

Shard 1 Shard 2

Machine 1 Machine 2

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

FTGS

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

FTGS

Main Loop

foreach field

foreach termget group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

FTGS

Main Loop

foreach field

foreach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

FTGS

FTGS Stream - Single Machine

class=1|1|323|200class=2|1|277|119

class=3|1|709|181fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

class=1|1|323|200class=2|1|277|119

class=3|1|709|181fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200class=2|1|277|119

class=3|1|709|181fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200class=2|1|277|119

class=3|1|709|181fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200class=2|1|277|119

class=3|1|709|181fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

Sorted

FTGS Stream - Single Machine

class=1|1|323|200class=2|1|277|119

class=3|1|709|181fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119class=3|1|709|181

fsize=0|1|790|239fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181fsize=0|1|790|239

fsize=1|1|235|126fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239fsize=1|1|235|126

fsize=2|1|159|90fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126fsize=2|1|159|90

fsize=3|1|43|30fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90fsize=3|1|43|30

fsize=4|1|22|6fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30fsize=4|1|22|6

fsize=5|1|25|5fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6fsize=5|1|25|5

fsize=6|1|16|4fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5fsize=6|1|16|4

fsize=7|1|8|0fsize=10|1|11|0gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4fsize=7|1|8|0fsize=10|1|11|0gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0fsize=10|1|11|0gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339gender=m|1|843|161

FTGS Stream - Single Machine

class=1|1|323|200

class=2|1|277|119

class=3|1|709|181

fsize=0|1|790|239

fsize=1|1|235|126

fsize=2|1|159|90

fsize=3|1|43|30

fsize=4|1|22|6

fsize=5|1|25|5

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|466|339

gender=m|1|843|161

FTGS Stream - Single Machine

FTGS Stream

How to distribute?

Shard 1 Shard 2

Machine 1 Machine 2

Shard 1 Shard 2

FTGS 1

Machine 2

Shard 1 Shard 2

FTGS 1 FTGS 2

Shard 1 Shard 2

FTGS 1 FTGS 2

Machine 3

Shard 1 Shard 2

FTGS 1 FTGS 2Merge

Machine 3

FTGS Stream Merge

class=1|1|198|111class=2|1|277|119class=3|1|511|129fsize=0|1|790|239fsize=1|1|94|53

fsize=2|1|75|48

fsize=3|1|21|17

fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

Machine 1

FTGS Stream Merge

class=1|1|125|89class=3|1|198|52

fsize=1|1|141|73fsize=2|1|84|42

fsize=3|1|22|13

fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

Machine 2

FTGS Stream Merge

class=1|1|198|111class=2|1|277|119class=3|1|511|129fsize=0|1|790|239fsize=1|1|94|53

fsize=2|1|75|48

fsize=3|1|21|17

fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89class=3|1|198|52

fsize=1|1|141|73fsize=2|1|84|42

fsize=3|1|22|13

fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

Machine 1 Machine 2

FTGS Stream Merge

class=1|1|198|111class=2|1|277|119class=3|1|511|129fsize=0|1|790|239fsize=1|1|94|53

fsize=2|1|75|48

fsize=3|1|21|17

fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89class=3|1|198|52

fsize=1|1|141|73fsize=2|1|84|42

fsize=3|1|22|13

fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

class=1|1|323|200

+

Machine 1 Machine 2

FTGS Stream Merge

class=1|1|198|111

class=2|1|277|119class=3|1|511|129fsize=0|1|790|239fsize=1|1|94|53fsize=2|1|75|48

fsize=3|1|21|17

fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52fsize=1|1|141|73

fsize=2|1|84|42fsize=3|1|22|13

fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

class=1|1|323|200

Machine 1 Machine 2

FTGS Stream Merge

class=1|1|198|111

class=2|1|277|119class=3|1|511|129fsize=0|1|790|239fsize=1|1|94|53fsize=2|1|75|48

fsize=3|1|21|17

fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52fsize=1|1|141|73

fsize=2|1|84|42fsize=3|1|22|13

fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

class=2|1|277|119class=1|1|323|200

Machine 1 Machine 2

FTGS Stream Mergeclass=1|1|198|111

class=2|1|277|119

class=3|1|511|129fsize=0|1|790|239fsize=1|1|94|53fsize=2|1|75|48fsize=3|1|21|17

fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52fsize=1|1|141|73

fsize=2|1|84|42fsize=3|1|22|13

fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

class=2|1|277|119class=1|1|323|200

Machine 1 Machine 2

FTGS Stream Mergeclass=1|1|198|111

class=2|1|277|119

class=3|1|511|129fsize=0|1|790|239fsize=1|1|94|53fsize=2|1|75|48fsize=3|1|21|17

fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52fsize=1|1|141|73

fsize=2|1|84|42fsize=3|1|22|13

fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

class=3|1|709|181class=2|1|277|119

class=1|1|323|200

+

Machine 1 Machine 2

FTGS Stream Mergeclass=1|1|198|111

class=2|1|277|119

class=3|1|511|129

fsize=0|1|790|239fsize=1|1|94|53fsize=2|1|75|48fsize=3|1|21|17fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52

fsize=1|1|141|73fsize=2|1|84|42

fsize=3|1|22|13fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39

class=3|1|709|181class=2|1|277|119

class=1|1|323|200

Machine 1 Machine 2

FTGS Stream Mergeclass=1|1|198|111

class=2|1|277|119

class=3|1|511|129

fsize=0|1|790|239fsize=1|1|94|53fsize=2|1|75|48fsize=3|1|21|17fsize=4|1|3|1

fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52

fsize=1|1|141|73fsize=2|1|84|42

fsize=3|1|22|13fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39fsize=0|1|790|239

class=3|1|709|181class=2|1|277|119

class=1|1|323|200Machine 1 Machine 2

FTGS Stream Mergeclass=1|1|198|111

class=2|1|277|119

class=3|1|511|129

fsize=0|1|790|239

fsize=1|1|94|53fsize=2|1|75|48fsize=3|1|21|17fsize=4|1|3|1fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52

fsize=1|1|141|73fsize=2|1|84|42

fsize=3|1|22|13fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39fsize=0|1|790|239class=3|1|709|181

class=2|1|277|119class=1|1|323|200

Machine 1 Machine 2

FTGS Stream Mergeclass=1|1|198|111

class=2|1|277|119

class=3|1|511|129

fsize=0|1|790|239

fsize=1|1|94|53fsize=2|1|75|48fsize=3|1|21|17fsize=4|1|3|1fsize=5|1|3|1

gender=f|1|308|237

gender=m|1|678|122

class=1|1|125|89

class=3|1|198|52

fsize=1|1|141|73fsize=2|1|84|42

fsize=3|1|22|13fsize=4|1|19|5

fsize=5|1|22|4

fsize=6|1|16|4

fsize=7|1|8|0

fsize=10|1|11|0

gender=f|1|158|102

gender=m|1|165|39fsize=1|1|235|126fsize=0|1|790|239

class=3|1|709|181class=2|1|277|119

class=1|1|323|200

+

Machine 1 Machine 2

Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6

FTGS 1 FTGS 2 FTGS 3 FTGS 4 FTGS 5 FTGS 6

FTGS 1 FTGS 2 FTGS 3 FTGS 4 FTGS 5 FTGS 6

k-way merge

FTGS 1-6

FTGS 1 FTGS 2 FTGS 3 FTGS 4 FTGS 5 FTGS 6

FTGS 1-6 FTGS 7-12 FTGS 13-18

FTGS 1-6 FTGS 7-12 FTGS 13-18

FTGS 1-18

FTGS 1-18 FTGS 19-36

FTGS 1-36

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

FTGS

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Main Loop

foreach fieldforeach term

get group statsevaluate splits

apply best splits for each grouprepeat n times or until no more splits found

Regroup

FTGS 1-6 FTGS 7-12 FTGS 13-18

FTGS

Regroup 1-6 Regroup 7-12 Regroup 13-18

Regroup

FTGS 1-6 FTGS 7-12 FTGS 13-18

FTGS

Regroup 1-6 Regroup 7-12 Regroup 13-18

Regroup

Imhotep

Imhotep

Distributed System that does efficient FTGS and Regroup operations on inverted indexes

Imhotep

32 machines

2 cpu x 6 core xeon westmere E5649128GB RAM10x1TB 7200 RPM SATA

Total:384 cores, 4TB RAM, 320TB disk

Decision tree on 13 billion documents

Imhotep

Decision tree on 13 billion documents330GB → ~25 bytes per doc

Imhotep

Decision tree on 13 billion documents330GB → ~25 bytes per doc

First FTGS: 314 secondsFirst Regroup: 9.6 seconds

Imhotep

Decision tree on 13 billion documents330GB → ~25 bytes per doc

First FTGS: 314 seconds (36.3 million terms)First Regroup: 9.6 seconds

Imhotep

Decision tree on 13 billion documents330GB → ~25 bytes per doc

First FTGS: 314 seconds (36.3 million terms)First Regroup: 9.6 seconds (7 groups)

Imhotep

Decision tree on 13 billion documents330GB → ~25 bytes per doc

First FTGS: 314 seconds (36.3 million terms)First Regroup: 9.6 seconds (7 groups)

Second FTGS: 57 secondsSecond Regroup: 23 seconds (217 groups)

Imhotep

Imhotep

Distributed System that does efficient FTGS and Regroup operations

Powers our internal analytical tools

Imhotep

Distributed System that does efficient FTGS and Regroup operations

Powers our internal analytical tools

… and more

Imhotep - Next @IndeedEng Talk

Sharding and shard managementSession / FTGS network protocolMemory managementInverted IndexesFTGS MergeRegroup operationsFault Tolerance

Conclusion

Now scales to larger and larger data sets by adding more machines

Increased freshness and frequency of builds

Decision trees have lots of tunable components, regularly get 1% wins via A/B test

Continuous Improvement

Sponsored Job Click-through Rate (CTR)

Thanks.

Q & A

More Questions?Jason David James Jeff

Next @IndeedEng TalkImhotep: Large Scale Analytics

and Machine Learning at Indeed

Jeff Plaisance, Engineering ManagerMarch 26, 2014

http://engineering.indeed.com/talks