Computer Science

iBigTable: Practical Data Integrity for BigTable in Public Cloud

CODASPY 2013

Wei Wei, Ting Yu, Rui Xue

1/40

Computer Science

iBigTable – Overview BigTable – Scalable Storage System

o Store large data sets with petabytes or even more Business transactions, software logs, social network messages

o Benefits from processing large data sets Identify business opportunities, find software bugs, mine social relationship

o Widely used in Google, Facebook, Twitter However, small companies and researchers usually lack of capabilities to

deploy BigTableo Large cluster requiredo Technical difficultieso High maintenance cost

Deploying BigTable in a public cloud is an economic solution.

However, one may not always trust the public cloud provider.

2/40

Computer Science

iBigTable – Overview Our Focus

o Provide integrity assurance for BigTable in public cloud

Basic Ideao Build Merkle Hash Tree based Authenticated Data Structureo Decentralize integrity verification across multiple nodes

3/40

Computer Science

Agenda

IntroductionSystem ModelSystem DesignExperimental EvaluationRelated WorkConclusion

4/40

Computer Science

Merkle Hash Tree (MHT)

Verification Object (VO)o Data returned along with result and used to authenticate the result

Exampleo Authenticate data d1, and the VO for d1 is {h2 and h34}

• h1=H(d1)

• h12=H(h1|h2) • h34=H(h3|h4)

• hroot=H(h12|h34)• sroot=S(hroot)

• h2=H(d2) • h1=H(d3) • h1=H(d4)

5/40

Computer Science

BigTable – Data Model A table is a sparse, distributed, persistent multidimensional

sorted map (OSDI 2006). Data Model

o Table schema only defines its column families Each family consists of any number of columns Each column consists of any number of versions Columns only exist when inserted, NULLs are free Columns within a family are sorted and stored together

o Table contains a set of rows sorted based on row key Row: a set of column families Column Family: a set of columns Cell: arbitrary string (uninterpreted string)

6/40

Computer Science

BigTable – Data Organization Tablet

o Root tableto Metadata tableto User tablet

Tablet Servero Each tablet is only stored

in a tablet servero Multiple tablets can be

stored in a tablet server

Master Responsible for load balancing and assigning tablets

7/40

Computer Science

BigTable – Data Operations Queries

o Single row query by specify the row keyo Range query by specifying start and end row keyso Projection query to retrieve specific column, column family

Changeso Data insert, update, and deleteo Tablet split & merge

8/40

Computer Science

System Model Similar to Database Outsourcing

o Host data in untrusted party and support data retrievalo Principle ideas of integrity verification

Different from Database Outsourcingo Distributed data among large number of nodes

How to handle authenticated data structures during tablet merging or splitting Impractical to store authenticated structures in a single node Not scalable to adopt a centralized integrity verification scheme at a single point

o Simple data model and query interfaces Design much simpler and efficient authenticated structures and protocols to verify

data integrity

The actual design and deployment of authentication schemes are significantly different

9/40

Computer Science

System Model Assumptions

o The public cloud is not trusted, and BigTable is deployed in the public cloud, including the master and tablet servers

o The data owner has a public/private key pair, and public key is known to all

o The data owner is the only party who can update datao Public communications are through a secure channel

Attacks from The Public Cloudo Return incorrect data by tampering some datao Return incomplete data result by discarding some datao Report that data doesn’t exist or return old data

10/40

Computer Science

System Model cont’d Goal

o Deploy BigTable over Public Cloud with Practical Integrity Assurance

Design Goalso Security (Integrity)

Correctness, completeness, freshnesso Practicability

Simplicity, flexibility, efficiency

11/40

Computer Science

System Design Basic Idea

o Embed a MHT-based Authenticated Data Structure in each tablet

12/40

Computer Science

Distributed Merkle Hash Tree

User Tablet User Tablet

• …

Meta Tablet

Root Tablet

• …

Data Owner

Root hash

Pros Authenticated data distributed across nodes Only maintain one hash for all data

Cons Require update propagation Concurrent update could cause issues Hard to synchronize hash tree update Complicate protocols between tablet servers

13/40

Computer Science

Our Design

User Tablet User Tablet

• …

Meta Tablet

Root Tablet

• …

Data Owner

Root hash

14/40

Computer Science

Our Design

User Tablet User Tablet

• …

Meta Tablet

Root Tablet

• …

Data Owner

Root hash

Root hash

Root hash

Root hash

……

15/40

Computer Science

System Design Basic Idea

o Embed a MHT-based Authenticated Data Structure in each tableto Store the root hash of each MHT in a trusted party (e.g., data owner)

o Decentralize the integrity verification across multiple tablet servers

Data integrity is guaranteed by the correctness of the root hash of the MHT in each tablet.

16/40

Computer Science

Decentralized Integrity Verification

• 1.1 meta key (root, meta, table name, start row key)

• Tablet Server • serving ROOT tablet

• Client• 1.3 meta row (meta tablet location, start and end keys)

•1.

4 ve

rify

• 2.1 meta key (meta, table name, start row key)

• Tablet Server • serving META tablet

• Client• 2.3 meta row (user tablet location, start and end keys)

•2.

4 ve

rify

• 3.1 start and end row keys

• Tablet Server • serving USER tablet

• Client• 3.3 rows within the start and end row

keys

•3.

4 ve

rify

• 1.2 generate VO

• 2.2 generate VO

• 2.2 generate VO

• , VO

• , VO

• , VO

17/40

Computer Science

iBigTable – Authenticated Data Structure Signature Aggregation Compared with Merkle Hash Tree

o Both of them can guarantee correctness and completenesso Incur significant computation cost in client side and large storage cost

in server sideo Not clear how to address freshness

MHT-based Authenticated Data Structureo SL-MBT: A single-level Merkle B+ tree

Build a Merkle B+ tree based on all key value pairs in a tablet Each leaf is a hash of a key value pair

o ML-MBT: A multi-level Merkle B+ tree Builds multiple Merkle B+ trees in three different levels

o TL-MBT: A two-level Merkle B+ tree (adopted)

18/40

Computer Science

iBigTable – TL-MBT

Index Levelo Only one tree – index treeo Each leaf points to a data tree Data Levelo Row Tree: generate hashes for all rows and

each leaf is a hash of a row

o Column Family Tree: generate hashes for a column family of all rows and each leaf is a hash of a column family of a row

o Column Tree: generate hashes for a column of all rows and each leaf is a hash of a column of a row

19/40

Computer Science

iBigTable – TL-MBT Verification Object Generation

o Find the data tree(s) based on the specific queryo Use the data tree(s) to generate VO based on the query range

Proso Performance is comparable to ML-MBT for row-based queryo Much more efficient than SL-MBT and ML-MBT for projection queryo Flexible authenticated data structure

Conso Update cost may increase by 3 timeso Large storage cost if column trees are created

20/40

Computer Science

iBigTable – Data Access Range query within tablet

o Find metadata tablet, user tablet, data through specific tablet server

Range query across tabletso Break a large range into small sub-ranges

Based on the end key of each tablet Sub-range falls in a tablet

o Execute the sub-range queries

21/40

Computer Science

iBigTable – Single Row Update Partial Tree Verification Object (VO)

o Data included Only keys and hashes of data for two boundaries Hashes of nodes for computing the root hash Keys in related inner nodes

o Used for direct update within the range of partial tree

• 3.1 new row

• Tablet Server • serving USER tablet

• Data Owner• 3.3 partial tree VO

• 3.4 verify and update tablet root hash • 3.2 generate VO

22/40

Computer Science

6030

10 50 80

0 10 20 5030 40 80 9060 70

70

iBigTable – Single Row Update cont’d

Initial MB+ row tree of a tablet in a tablet server.

23/40

Computer Science

6030

50

5030 40

45

• New Key 45

Insert a row with key 45 into partial tree VO

iBigTable – Single Row Update cont’d

40 45

6030

50

5030

• New Key 45

40

• Partial tree VO after 45 is inserted

24/40

Computer Science

iBigTable – Efficient Batch Update Single row update is inefficient

o one verification for single row Range query is efficient

o One verification for multiple rows How can we do batch update like range query?

• 3.1 request partial tree VO for a range

• Tablet Server • serving USER tablet

• Data Owner• 3.3 partial tree VO

• 3.4 verify and update tablet root hash

• 3.4 new rows

• 3.n new rows

• … … …

• 3.2 generate VO

25/40

Computer Science

iBigTable – Tablet Changes Tablet split

o Grow too largeo Load balancingo Better management

Tablet mergeo Only a few data in a tableto Improve query efficiency

How to guarantee data integrity?o Make sure the root hash of each tablet is correctly updated

26/40

Computer Science

6030

10 50 80

0 10 20 5030 40 60 70 80 90

70

iBigTable – Tablet Split

• (a) A MBT of a tablet in a tablet server, and split tablet at key 45.

27/40

Computer Science

6030

10 50 80

10 20 5030 40 60

70

• Two boundary keys• Left boundary node • Right boundary node

iBigTable – Tablet Split cont’d

• (b) Partial tree returned to the data owner.

28/40

Computer Science

6030

10 50

10 20 30 40

6030

50 80

50 60

70

• Left Partial Tree • Right Partial Tree

• Split

iBigTable – Tablet Split cont’d

• (c) Split it into two partial trees by data owner.

29/40

Computer Science

6030

10 50

10 20 30 40

3010

10 20 30 40

iBigTable – Tablet Split cont’d

• (d) Data owner adjusts left partial tree and computes the new root hash for the new tablet.

30/40

Computer Science

6030

50 80

50 60

70

70

60 80

50 60

iBigTable – Tablet Split cont’d

• (e) Data owner adjusts right partial tree and computes the new root hash for the new tablet.

31/40

Computer Science

• Left Partial Tree • Right Partial Tree

• Merge

3010

30 40

70

60

50

• Merged Tree

3010

30 40

70

60

50

50

iBigTable – Tablet Merge

• Data owner merges two partial trees sent from tablet servers into one for the new merged tablet

32/40

Computer Science

iBigTable – Experimental Evaluation System Implementation

o Implementation based on HBaseo Extend some interfaces to specify integrity optionso Add new interfaces to support efficient batch updates

Experiment Setupo 5 hosts in Virtual Computing Lab (VCL)o Intel(R) Xeon(TM) CPU 3.00GHzo Red Hat Enterprise 5.1, Hadoop-0.20.2, and HBase-0.90.4o Client network with 30Mbps download and 4Mbps upload

33/40

Computer Science

iBigTable – Baseline

• Ex 1. Time to receive data from server • Ex 2. VO size vs # of rows

Observationso It almost takes the same time to transmit data less than 4ko Time is doubled from 4k to 8k till around 64k.o After 64k, the time dramatically increases.o The VO size increases as the range increases, but the VO size per row

actually decreases.

34/40

Computer Science

iBigTable – Write

• Ex 3. Write performance. • Ex 4. The breakdown of write cost

Observationso The performance overhead ranges from 10% to 50%.o iBigTable with Efficient Batch Update only causes a performance

overhead about 1.5%.o Communication cost is high, but computation cost is small about

2~5%.

35/40

Computer Science

iBigTable – Read Observations

o The read performance overhead is small, which ranges from 1% to 8%.o The total computation cost of both client and servers is about 1%.o The major part of performance downgrade is caused by

communication.

• Ex 5. Read performance • Ex 6. The breakdown of read cost

36/40

Computer Science

iBigTable – TL-MBT Observations

o As the number of trees that need to be updated increases, the performance decreases dramatically.

o For different data size, we see the large performance variation for different cases.

• Ex 7. TL-MBT update performance. • Ex 8. Projection query with TL-MBT37/40

Computer Science

iBigTable – Related Work Research related to BigTable

o Performance evaluation [Carstoiu et al., NISS 2010]o High performance OLAP analysis [You et al., IMSCCS 2008]o BigTable in a hybrid cloud [Ko et al., HotCloud 2011]o Integrity layer for cloud storage [Kevin et al., CCS 2009]

Outsourcing Databaseo Different authenticated data structures [DASFAA 2006]o Probabilistic approaches [Xie et al.VLDB 2007]o Approaches to address complex queries [Yang et al., SIGMOD 2009]o Partitioned MHT (P-MHT) [Zhou et al., MS-CIS 2010]

38/40

Computer Science

iBigTable – Conclusion Contributions

o Explore the practicability of different authenticated data structures Focus on Merkle Hash Tree based authenticated data structures

o Design a set of efficient mechanisms to handle authenticated data structure changes

Efficient data batch update Handle tablet split and merge

o Implement a prototype of iBigTable based on Hbase, an open source implementation of BigTable

o Conduct experimental evaluation of performance overhead

39/40

Computer Science

•Thank you•Questions?

40/40