ieee icpads 2008 - kalman graffi - skyeye.kom: an information management over-overlay for getting...

IEEE_ICPADS08_KG_SkyEye.ppt

KOM - Multimedia Communications LabProf. Dr.-Ing. Ralf Steinmetz (director)

Dept. of Electrical Engineering and Information TechnologyDept. of Computer Science (adjunct professor)

TUD – Technische Universität Darmstadt Merckstr. 25, D-64283 Darmstadt, Germany

Tel.+49 6151 166150, Fax. +49 6151 166152 www.KOM.tu-darmstadt.de

© author(s) of these slides 2008 including research results of the research network KOM and TU Darmstadt otherwise as specified at the respective slide

QuaP2P Improving the quality of P2P systems

DFG research group

Dipl.-Math., Dipl.-Inform. Kalman [email protected]

04/12/23

SkyEye.KOM: An Information Management Over-Overlay for Getting the Oracle View on

Structured P2P Systems

Kalman Graffi, Aleksandra Kovacevic,Song Xiao and Ralf Steinmetz

H(„my data“)= 3107

2207

7.31.10.25

peer-to-peer.info

12.5.7.31

95.7.6.10

86.8.10.18

planet-lab.orgberkeley.edu

29063485

201116221008709

611

89.11.20.15

?

? Does my p2p system work?

Underlay:The Internet

StructuredOverlay: DHT

Information Management Over-Overlay

Pick good peers

KOM – Multimedia Communications Lab 2

Outline

Motivation Information Management in P2P Systems Example

SkyEye.KOM Requirements Architecture

Evaluation Performance Costs

Conclusion


The Peer-to-Peer Paradigm

Peer-to-Peer Systems: Users of a system provide the infrastructure Service is provided from users/peers to users/peers Peer-to-Peer overlays:

virtual networks, providing new functionality E.g. Distributed Hash Tables, Keyword-based Search

Evolution of applications File sharing:

No QoS requirements Voice over IP

Real-time requirements Video-on-demand

Real-time and bandwidth requirements Online community platforms

Potential for high user interaction

Costs Security

Quality of P2P Systems

Retrievability

Coherence

Consistency

Correctness

PerformanceScalability

Flexibility

Stability

Dependability

Service Provisioning

Overlay Operations

Individual Node

Complete System

IP Infrastructure

Availability

Reliability

Robustness/ Fault tolerance

Integrity

Confidentiality

Authentication

Non-repudation

TrustValidityEfficiencyAdaptability

Costs Security

Quality of P2P Systems

Retrievability

Coherence

Consistency

Correctness

PerformanceScalability

Flexibility

Stability

Dependability

Service Provisioning

Overlay Operations

Individual Node

Complete System

IP Infrastructure

Availability

Reliability

Robustness/ Fault tolerance

Integrity

Confidentiality

Authentication

Non-repudation

TrustValidityEfficiencyAdaptability


Open Issues in P2P Research

Current P2P-applications do not consider sufficiently the heterogeneity of peers the state of the whole P2P-system

P2P systems and applications would benefit from monitoring the system generating system statistics determining the state of the P2P system accounting the capacities of peers

Possible results of the aforementioned actions Detection of limitations or failures in running P2P system Adaptation of the peers to the current state of the P2P system


Example: Let’s Build a new P2P Application!

Get ideas, make design, implement it…Cool hot product, lot of features!

A lot of modules involved All of them have some requirements, need specific peers See K.Graffi, “A Distributed Platform for Multimedia Communities”

at Proc. of IEEE Int. Symposium on Multimedia ’08, 15.-17.Dec.’08

Each functional layer requires specific peers Distribute Computing: Give me 4 peers with Java > 6.0 and

processor power > 10M flops Streaming: Give me 2 peers with sufficient processing power

for media file recoding Replication: Give me 1 peer with a lot of bandwidth

and 10 peers with specific life time and storage space Overlay: Give me 3 peers, with a node-degree > 50 Network wrapper: Give me 10 peers from my ISP Security: Find 5 trustworthy peers

Challenge No.1: Capacity-based Peer Search For identifying peer capacities in the network we can use


Hooray it Runs! Publish and Advertize!

P2P

P2P CompanyGive us money©


P2P App.

Deployment and Running a P2P System

Challenge No.1: Capacity-based Peer Search

More challenges?

Is everything running fine?

How to debug and to gain insight?

Challenge No. 2:

Live statistics on P2P systems

H(„my data“)= 3107

2207

7.31.10.25

peer-to-peer.info

12.5.7.31

95.7.6.10

86.8.10.18

planet-lab.orgberkeley.edu

29063485

201116221008709

611

89.11.20.15

?

? Does my p2p system work?

Underlay:The Internet

StructuredOverlay: DHT


Optimizing the RUNNING P2P System

Assume we have statistics on our P2P application We see that something is wrong / can be improved We know which parameters to change We change the parameter somehow (by hand?)How to change parameters in a running system?

Challenge No.3: Parameter optimization based on statistics

Our P2P application


Outline




Conclusion


Requirements on an Information Mgmt. System

For all structured P2P overlays Covered by DHT-function: route(msg, key), lookup(key) Usable by all functional layers/modules in the P2P system

Generating system statistics System wide average and absolute values, standard deviation, confidence intervals Num. of peers, online times, hops per lookup, hit rate, relative delay penalty, node degree Load (CPU, memory, storage space, bandwidth), additionally weighted with individual

peer capabilities

Capacity-based peer search Search for a specified number of peers with specific capacities E.g. give me 3 peers with:

Storage space > 20Mb Bandwidth > 100kb/s

Remote consistent parameter setting Reconfigure all peers during runtime


SkyEye: Information Management Over-overlay

For all structured P2P overlays Covered by DHT-function:

route(msg, key), lookup(key) Usable by all functional layers in the

P2P system

Features: Overlay-independency Robustness (double-churn) Load-balancing Supporting peer heterogeneity Scalability (# of info and peers) Lightweight Low overhead

Function: Generating system statistics Capacity-based peer search Enable parameter setting reconfiguration

Internet

DHT overlayoffering route(key,msg,nextHop), resp(key) .

API for DHT’s ID space and functions

Unified ID space and abstr. functions

Coordinator Support Peer Peer


SkyEye.KOM Architecture

Concept of Over-overlay Built on underlying structured overlay Communicates via common API (KBR)

route(msg, key), lookup(key) Unified ID space [0,1] decouples

from specific DHT implementation:e.g. divide ID by maxID

Information Domains: ID space separated in intervals (domains) Peer responsible for a specific ID (e.g. middle) is responsible for ID domain Peers in the domain send updates to this Coordinator Coordinators may choose Support Peers to share the load/responsibility

Coordinators define maximum load to carry Updates propagated upwards the tree

API for DHT’s ID space and functions

Unified ID space and abstr. functions



Example Figures

0 1

11050

2030

40

45

15

0,09 0,2 0,3 0,4 0,51 0,6 0,75 0,9

CoordinatorID 0,25

CoordinatorID 0,125

0D

CoordinatorID 0,375

CoordinatorID 0,751D

0 10,18 0,24 0,4 0,55 0,8

CoordinatorID 0,50C

1C 1C

2C 2C


SkyEye.KOM Architecture Details

Update strategy Each node

sends information up knows where to send updates to

Update consists of: Non-aggregatable peer capacity information Aggregatable systems statistics part

Updates are processed before forwarding Update information has a limited lifetime

Supporting Peers for Load Balancing Coordinator may chose Supporting Peers Good peers chosen by e.g. 50/50 ratio

Pick e.g. 20 best peers in the domain Best 10 peers in domain advertised one level up Second best 10 peers can be used as support

Workload can be delegated to supporting peers Tree depth / peer load adjustable


For SP: best 11-20

best 1-10 best 1-10

best 1-10best 1-10

For SP: best 11-20 from below

For SP: best 11-20

For SP: best 11-20 from below


Setting the Parameters

Capturing the state of every peer Periodically measuring and aggregating

statistics Forwarding of aggregated statistics

Evaluation of the collected information

Calculation of system statistics Enables the integration of data mining

tools for implementing self-organization Transmission of the new information

Adaption to the propagated information

Enabling the adaptive behavior of a peer

Evaluation of dataidentification ofcounteraction

0C

1C 1C

tC tC

Measuring and aggregatingmetrics






Queries in the Information Mgmt. System

Query Type: Capacity-based Peer Search Give me M peers Fulfilling specific requirements on

Bandwidth, storage space, computational capabilities, Online time, peer load, reputation … (wide set of requirements definable)

Query processing First sent to coordinator Query traverses bottom-up, until M matching peers found Result is sent then to requesting peer Tradeoff:

Upper peers in tree know more Load should be kept on lower levels of the tree


Very Simple Tree Maintenance

Join, Leave, Repair, Maintenance: Not needed Join: Just send an update to Coordinator After failure of “simple peer”: peer specific information times out After failure of Coordinator / Support Peer: route(msg, key) identifies new Coordinator

Coordinator picks a new Support Peer New responsible peer is updated in the next update interval

No additional maintenance needed (done by structured overlay)


Outline




Conclusion


Evaluation of SkyEye: Scalability and Information Freshness

Scalability Tree-structure of coordinators form

information architecture Supp. peers: Strong peers take the load Query performance: O(log N) hops

Freshness Freshness tightly related to tree depth Information freshness:

O(log N * updateInterval)

Conclusion: Information is fresh even under churn

Failing peers are quickly replaced It takes one update interval to recover


Evaluation of SkyEye: Quality of Information

Completeness of Knowledge Peers may define maximum load

Some information may be dropped Completeness of knowledge: Ratio of

considered peers in the corresp. domain All >90%, some decline at level 9

Quality of Information: Peers in the results: >98% are online Less useful peer information is dropped

Load on the Peers Constant update load (1 or 3 msg./intvl.) Either just leaf or Coordinator as well Most of the peers are at level 8-9 Information density is highest at level 8-9

Some information is dropped theredue to load limit of peers


Evaluation of SkyEye: Query Performance

Position of query resolving: Remember: most peers in level 8-9 Query takes 2-4 hops to be resolved Most of the queries resolved in level 4-5 Leaves room for optimization:

Injecting queries lower in the tree Adapting tree height to query load

Impact of query complexity Query complexity ranging from 5 to 15 Small impact on position of resolving peer Leaves room for optimization:

Resolving easier queries at less loaded peers


Outline




Conclusion


Conclusion and Future Work on SkyEye.KOM

Some functional requirements and features addressed: Gathering system statistics and capacity-based peer search solved Overlay-independency

Unified ID space Usage of DHT-function route(msg, key), lookup(key)

Supporting peer heterogeneity Load dispatching to Support Peers

Low overhead Relying on robust underlying DHT, no maintenance

Room for optimization Load-balancing

Several Support Peers per Coordinator: synchronization issues Scalability (# of info and peers)

Limitations on the information size Adapting to usage patterns

Injects queries and updates on various levels in the tree


Questions?

ieee icpads 2008 - kalman graffi - skyeye.kom: an information management over-overlay for getting...

Documents

p2psystem p2p systems

p2p p2p company

running p2p system

p2p system challenge

dht p2p

p2p system work

p2p system adaptation

specific peers