servicing range queries on multidimensional datasets with partial replicas

24
Ohio State University Department of Computer Science and Engineering Servicing Range Queries Servicing Range Queries on Multidimensional Datasets on Multidimensional Datasets with Partial Replicas with Partial Replicas Li Weng, Umit Catalyurek, Li Weng, Umit Catalyurek, Tahsin Kurc, Gagan Agrawal, Joel Saltz Tahsin Kurc, Gagan Agrawal, Joel Saltz

Upload: kathie

Post on 10-Feb-2016

29 views

Category:

Documents


0 download

DESCRIPTION

Servicing Range Queries on Multidimensional Datasets with Partial Replicas. Li Weng, Umit Catalyurek, Tahsin Kurc, Gagan Agrawal, Joel Saltz. Outline. Introduction Motivation Partial Replicas Considered system overview Query execution and algorithm design Computing goodness value - PowerPoint PPT Presentation

TRANSCRIPT

Slide 1Servicing Range Queries
Li Weng, Umit Catalyurek,
Ohio State University
Outline
Introduction
Motivation
Computing goodness value
Replica selection algorithm
Motivating Applications
Cancer Studies using MRI Telepathology with Digitized Slides
Satellite Data Processing Virtual Microscope
Oil Reservoir Management is used by us as a case study.
Ohio State University
Motivation
The combination of large dataset sizes, geographic distribution of users and resources and complex analysis results in the requirements of efficient access and high-performance processing.
To achieve good performance for various query types and increasing need of clients, we need to harness an optimization technique Partial Replication.
Under a distributed environment, assembling required data efficiently from replicas and the original dataset for a query is an interesting challenge.
Ohio State University
Partial Replicas Considered
Hot range
Use a group of representative queries to identify the portions of the dataset to be replicated.
Chunking
Affect data read cost.
Affect data seek cost.
Partial Replicas Considered
To maximize I/O parallelism, users need to partition each chunk of one replica across all available data source nodes.
After re-organizing, re-distributing and re-ordering hot ranges of the dataset, there will not be one-to-one mapping between data chunks in the original dataset and those in replicas.
Ohio State University
System Overview
The Replica Selection Module is coupled tightly with our prior work
on supporting SQL Select queries on scientific datasets in a cluster
environment.
STORM Runtime System
A middleware to support data selection, data partitioning, and data transfer operations on flat-file datasets hosted on a parallel system.
Services
Outline
Introduction
Motivation
Computing goodness value
Replica selection algorithm
Computing Goodness Value
Chunk retrieval cost
k1 : average read time for a page
Cread-operation : number of pages fetched
k2 : average seek time
Cseek-operation : number of seeks
intermediate unit between a replica and its chunks
a group of full or partial chunks having same goodness value in a replica
goodness = useful dataper-fragment / costper-fragment
An Example –
3 fragments
Replica 2
Replica Selection Algorithm
Fmax : the fragment with the maximum goodness value in F
S : the ordered list of the candidate fragments in decreasing order of their goodness value
The runtime complexity is O(m2), where m is the number of fragments intersecting the query boundary.
Calculate the fragment set F
Append Fmax Into S
Re-compute the goodness value
An Example –
Assume Fragment 4 has the maximum goodness value.
Candidate fragments set is { 1, 2 (with overlap), 3, 4, D }.
Ohio State University
Replica Selection Algorithm
Extension to the greedy algorithm
S : the ordered list of the candidate fragments in decreasing order of their goodness value
Fi : a fragment in S
C : a chunk in Fi
r : the union range contained by the filtered areas of other fragments
The runtime complexity is O(n2), where n is the number of chunks intersecting the query boundary.
S
Foreach chunk C in Fi
Drop it from Fi
Input
Output
No
Yes
No
Yes
An Example –
Recommended Chunks
Final recommendation
Overlap region has been deleted from Fragment 4 and retrieved in Fragment 2 instead.
We get fewer I/O operations and less filtering computation.
Ohio State University
Outline
Introduction
Motivation
Computing goodness value
Replica selection algorithm
Experimental Setup & Design
A Linux cluster connected via a Switched Fast Ethernet. Each node has a PIII 933MHz CPU, 512 MB main Memory, and three 100GB IDE disks.
Scalability test when increasing data size;
Performance test when the number of nodes hosting dataset is varied;
Showing the robustness of the proposed algorithm.
Ohio State University
Ohio State University
Query #1
SELECT * from IPARS where TIME>=1000 and TIME<=TIMEVAL and X>=0 and X<=11
and Y>=0 and Y<=28 and Z>=0 and Z<=28;
Set #1 in the previous table 1.
Our algorithm has chosen {0,1,2,4} out of 6 replicas in Set #1.
The query filters 83% of the retrieved data when using the original dataset only; however,
it need to filter about 25% of the retrieved data in the presence of replicas as in set #1.
Chart2
[1000,1199]
[1000,1199]
[1000,1399]
[1000,1399]
[1000,1599]
[1000,1599]
[1000,1799]
[1000,1799]
=
Six
replicas
with
different
partial
ranges
and
chunk
size/shape
0
Chunk->
(1,10,40,400,400)
Range->
[(0,1000,0,0,0),(0,1799,79,3199,3199)]
1
Chunk->
(1,10,40,400,1600)
Range->
[(0,1000,0,0,0),(0,1799,159,1599,6399)]
2
Chunk->
(1,10,40,1600,400)
Range->
[(0,1000,0,0,0),(0,1799,159,6399,1599)]
3
Chunk->
(1,10,160,400,400)
Range->
[(0,1000,0,0,0),(0,1799,159,799,799)]
4
Chunk->
(1,10,80,800,800)
Range->
[(0,1000,40,1600,1600),(0,1799,119,4799,4799)]
5
Chunk->
(1,10,60,600,600)
Range->
[(0,1000,40,3400,3400),(0,1799,159,6399,6399)]
#1.
Query
1
select
6
0
100
1,2,3
SOIL
1
0
100
1,2,3
SOIL
3
0
100
1,2,3
SOIL
7
0
100
1,2,3
SOIL
8
0
100
1,2,3
SOIL
*Replicas
2*
11
0
100
1,2,3
SOIL
12
0
100
1,2,3
SOIL
0
0
100
1,2,3
SOIL
5
0
100
1,2,3
SOIL
2
0
100
1,2,3
SOIL
select
=
Six
replicas
with
different
partial
ranges
and
chunk
size/shape
0
Chunk->
(1,10,40,400,400)
Range->
[(0,1000,0,0,0),(0,1799,79,3199,3199)]
1
Chunk->
(1,10,40,400,1600)
Range->
[(0,1000,0,0,0),(0,1799,159,1599,6399)]
2
Chunk->
(1,10,40,1600,400)
Range->
[(0,1000,0,0,0),(0,1799,159,6399,1599)]
3
Chunk->
(1,10,160,400,400)
Range->
[(0,1000,0,0,0),(0,1799,159,799,799)]
4
Chunk->
(1,10,80,800,800)
Range->
[(0,1000,40,1600,1600),(0,1799,119,4799,4799)]
5
Chunk->
(1,10,60,600,600)
Range->
[(0,1000,40,3400,3400),(0,1799,159,6399,6399)]
#1.
Query
1
select
6
0
100
1,2,3
SOIL
1
0
100
1,2,3
SOIL
3
0
100
1,2,3
SOIL
7
0
100
1,2,3
SOIL
8
0
100
1,2,3
SOIL
*Replicas
2*
11
0
100
1,2,3
SOIL
12
0
100
1,2,3
SOIL
0
0
100
1,2,3
SOIL
5
0
100
1,2,3
SOIL
2
0
100
1,2,3
SOIL
select
Query #2
SELECT * from IPARS where TIME>=1000 and TIME<=1599 and X>=0 and X<=11
and Y>=0 and Y<=31 and Z>=0 and Z<=31;
Set #1 in the previous table 1.
Our algorithm has chosen {0,1,2,4} out of 6 replicas in Set #1.
Upto 4 nodes, query execution time scales linearly.
Due to the dominating seek cost in the total I/O overhead, execution time is not reduced by half while using 8 nodes.
Chart5
1
1
2
2
4
4
8
8
=
Six
replicas
with
different
partial
ranges
and
chunk
size/shape
0
Chunk->
(1,10,40,400,400)
Range->
[(0,1000,0,0,0),(0,1799,79,3199,3199)]
1
Chunk->
(1,10,40,400,1600)
Range->
[(0,1000,0,0,0),(0,1799,159,1599,6399)]
2
Chunk->
(1,10,40,1600,400)
Range->
[(0,1000,0,0,0),(0,1799,159,6399,1599)]
3
Chunk->
(1,10,160,400,400)
Range->
[(0,1000,0,0,0),(0,1799,159,799,799)]
4
Chunk->
(1,10,80,800,800)
Range->
[(0,1000,40,1600,1600),(0,1799,119,4799,4799)]
5
Chunk->
(1,10,60,600,600)
Range->
[(0,1000,40,3400,3400),(0,1799,159,6399,6399)]
#1.
Query
1
select
6
0
100
1,2,3
SOIL
1
0
100
1,2,3
SOIL
3
0
100
1,2,3
SOIL
7
0
100
1,2,3
SOIL
8
0
100
1,2,3
SOIL
*Replicas
2*
11
0
100
1,2,3
SOIL
12
0
100
1,2,3
SOIL
0
0
100
1,2,3
SOIL
5
0
100
1,2,3
SOIL
2
0
100
1,2,3
SOIL
select
Query #3
SELECT * from IPARS where TIME>=1000 and TIME<=TIMEVAL and X>=0 and X<=15
and Y>=0 and Y<=63 and Z>=0 and Z<=63;
Set #1 in the previous table 1.
Our algorithm extension could detect the redundant I/O in the candidate replicas for this query. The final recommendation is to avoid using replicas.
Chart1
[1000,1199]
[1000,1199]
[1000,1399]
[1000,1399]
[1000,1599]
[1000,1599]
[1000,1799]
[1000,1799]
=
Six
replicas
with
different
partial
ranges
and
chunk
size/shape
0
Chunk->
(1,10,40,400,400)
Range->
[(0,1000,0,0,0),(0,1799,79,3199,3199)]
1
Chunk->
(1,10,40,400,1600)
Range->
[(0,1000,0,0,0),(0,1799,159,1599,6399)]
2
Chunk->
(1,10,40,1600,400)
Range->
[(0,1000,0,0,0),(0,1799,159,6399,1599)]
3
Chunk->
(1,10,160,400,400)
Range->
[(0,1000,0,0,0),(0,1799,159,799,799)]
4
Chunk->
(1,10,80,800,800)
Range->
[(0,1000,40,1600,1600),(0,1799,119,4799,4799)]
5
Chunk->
(1,10,60,600,600)
Range->
[(0,1000,40,3400,3400),(0,1799,159,6399,6399)]
#1.
Query
1
select
6
0
100
1,2,3
SOIL
1
0
100
1,2,3
SOIL
3
0
100
1,2,3
SOIL
7
0
100
1,2,3
SOIL
8
0
100
1,2,3
SOIL
*Replicas
2*
11
0
100
1,2,3
SOIL
12
0
100
1,2,3
SOIL
0
0
100
1,2,3
SOIL
5
0
100
1,2,3
SOIL
2
0
100
1,2,3
SOIL
select
=
Six
replicas
with
different
partial
ranges
and
chunk
size/shape
0
Chunk->
(1,10,40,400,400)
Range->
[(0,1000,0,0,0),(0,1799,79,3199,3199)]
1
Chunk->
(1,10,40,400,1600)
Range->
[(0,1000,0,0,0),(0,1799,159,1599,6399)]
2
Chunk->
(1,10,40,1600,400)
Range->
[(0,1000,0,0,0),(0,1799,159,6399,1599)]
3
Chunk->
(1,10,160,400,400)
Range->
[(0,1000,0,0,0),(0,1799,159,799,799)]
4
Chunk->
(1,10,80,800,800)
Range->
[(0,1000,40,1600,1600),(0,1799,119,4799,4799)]
5
Chunk->
(1,10,60,600,600)
Range->
[(0,1000,40,3400,3400),(0,1799,159,6399,6399)]
#1.
Query
1
select
6
0
100
1,2,3
SOIL
1
0
100
1,2,3
SOIL
3
0
100
1,2,3
SOIL
7
0
100
1,2,3
SOIL
8
0
100
1,2,3
SOIL
*Replicas
2*
11
0
100
1,2,3
SOIL
12
0
100
1,2,3
SOIL
0
0
100
1,2,3
SOIL
5
0
100
1,2,3
SOIL
2
0
100
1,2,3
SOIL
select
Query #4
SELECT * from IPARS where TIME>=1000 and TIME<=1199;
An accurate cost modeling should take into account both the seek cost and the read cost.
Original
Related Work
Supporting regular strided access to uniform distributed datasets
File level and dataset level replication and replica management
Exact replica copies
Availability and reliability
Conclusions
We have investigated a compiler-runtime approach for execution of range queries on distributed environment when employing partial replication.
We have proposed a cost metric and algorithm to select the set of replicas and possibly the original dataset to answer a given query efficiently.
Experimental results demonstrate the efficacy, scalability and robustness of our algorithm.
Processing Remotely
(AVHRR)

are gathered to form an
instantaneous field of view
A single file of
IFOV’s