Data Structures for Big Data:
Bloom Filter
Vinicius Vielmo Cogo
Smalltalks, DI, FC/UL. October 16, 2014.
2 / 30
is relative
is not defined by a specific number of TB, PB, EB
is when it becomes big for you
is when your solutions become inefficient/impractical
Data Structures for Big Data
Traditional DSs are subject to the same problems
e.g., lists, trees
(e.g., YARN, NoSQL)
or
(e.g., index, metadata)
reached the point of thinking in new DSs for BD
3 / 30
Outline
Bloom Filter
Use Cases
Implementations
Other Filters
Other Data Structures for Big Data
4 / 30
Bloom Filter
Membership testing
Does my collection contain this element?
5 / 30
Bloom Filter
City
Coimbra
Leiria
6 / 30
Bloom Filter
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Index i
bf[i]
http://billmill.org/bloomfilter-tutorial/ 7 / 30
Bloom Filter
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Index i
bf[i]
City
Coimbra
Leiria
Hash Function
Fnv
Murmur
8 / 30
Bloom Filter
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Index i
bf[i]
City
Coimbra
Leiria
Hash Function
Fnv
Murmur
i=4
i=7
9 / 30
Bloom Filter
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 0 0 1 0 0 1 0 0 0 0 0 0 0
Index i
bf[i]
City
Coimbra
Leiria
Hash Function
Fnv
Murmur
i=4
i=7
10 / 30
Bloom Filter
Index i
bf[i]
City
Coimbra
Leiria
Hash Function
Fnv
Murmur
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 0 0 1 0 0 1 0 0 0 0 0 0 0
11 / 30
Bloom Filter
Index i
bf[i]
City
Coimbra
Leiria
Hash Function
Fnv
Murmur
i=2
i=9
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 0 0 1 0 0 1 0 0 0 0 0 0 0
12 / 30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 1 0 1 0 0 1 0 1 0 0 0 0 0
Bloom Filter
Index i
bf[i]
City
Coimbra
Leiria
Hash Function
Fnv
Murmur
i=2
i=9
13 / 30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 1 0 1 0 0 1 0 1 0 0 0 0 0
Bloom Filter
Index i
bf[i]
City
Coimbra
Leiria
Hash Function
Fnv
Murmur
14 / 30
Bloom Filter
City
Braga
Guarda
Coimbra
Lisboa
15 / 30
Result: false
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 1 0 1 0 0 1 0 1 0 0 0 0 0
Bloom Filter
Index i
bf[i]
City
Braga
Guarda
Coimbra
Lisboa
Hash Function
Fnv
Murmur
i=10
i=14
16 / 30
Result: false
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 1 0 1 0 0 1 0 1 0 0 0 0 0
Bloom Filter
Index i
bf[i]
City
Braga
Guarda
Coimbra
Lisboa
Hash Function
Fnv
Murmur
i=2
i=12
17 / 30
Result: true
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 1 0 1 0 0 1 0 1 0 0 0 0 0
Bloom Filter
Index i
bf[i]
City
Braga
Guarda
Coimbra
Lisboa
Hash Function
Fnv
Murmur
i=4
i=7
18 / 30
Result: true (but it is a false positive)
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 1 0 1 0 0 1 0 1 0 0 0 0 0
Bloom Filter
Index i
bf[i]
City
Braga
Guarda
Coimbra
Lisboa
Hash Function
Fnv
Murmur
i=7
i=9
19 / 30
Bloom Filter
DS proposed by Burton Howard Bloom in 1970
Design principles
Space-efficient
Smaller than the original dataset
Time-efficient
Low latency R/W
O(k), which is much smaller than O(n)
High throughput
Probabilistic
E.g., myCollection.mightContain(myObject)
False positives happen (but in a configurable way)
20 / 30
= Optimal number of hash functions
Hash Function
Fnv
Murmur
Important variables
Bloom Filter
= Expected collection size
= Bitmap size
= False positive rate (e.g., 0.0001% or 1 in 1M)
City
Coimbra
Leiria
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21 / 30
Important variables
Bloom Filter
22 / 30
Users define two of them (normally n and any other)
The other two are calculated with those equations
Interesting relations:
Bigger collection ( ) Larger bitmap ( )
Bigger collection ( ) More false positives ( )
Larger bitmap ( Less false positives ( )
Larger bitmap ( ) Less hash functions ( )
Less hash functions ( )
Bloom Filter
23 / 30
Bloom filter size vs. False positive rate
Bloom Filter
24 / 30
Use Cases
Reducing unnecessary disk reads
Client BloomFilter Dataset
RAM Hard Disk
1
2
3
F
T
T F
T
1?
2?
3?
necessary
read(2)
unnecessary
read(3)
No
2
No
F
25 / 30
Use Cases
Google BigTable, Apache Cassandra and HBase
Reducing disk lookups
Google Chrome
Lookup a list of known malicious URLs
Bitcoin
Get only the transactions relevant to your wallet
Others
In my Ph.D. work
Lookup a list of known privacy-sensitive DNA
sequences 26 / 30
Implementations
-libraries https://code.google.com/p/guava-libraries/
Orestes-Bloomfilter https://github.com/Baqend/Orestes-Bloomfilter
java-bloomfilter https://github.com/magnuss/java-bloomfilter
java-longfastbloomfilter https://code.google.com/p/java-longfastbloomfilter/
27 / 30
Other Filters
Counting Bloom filters Allow deletions (use a 4-bit counter instead of 1 bit)
Buffered Bloom filters Sub-filters in SSD with buffered R/W exploring bit locality
Quotient and Cascade filters Uses an SSD, instead of the main memory, for scalability
28 / 30
Other DSs (and techniques) for Big Data
Locality-sensitive hashing (LSH) Hashing similar elements into the same bucket with high probability
HyperLogLog for computing cardinality Counting the number of distinct elements in a collection
Log Structured Merge (LSM) trees Indexed access to files with high insert volume and background batch synchronization
29 / 30
Thank you!
Vinicius Vielmo Cogo
Smalltalks, DI, FC/UL. October 16, 2014.