ieee p2p 2013 - bootstrapping skynet: calibration and autonomic self-control of structured...
DESCRIPTION
Timo Klerx and Kalman Graffi. Bootstrapping Skynet: Calibration and Autonomic Self-Control of Structured Peer-to-Peer Networks. In IEEE P2P ’13: Proceedings of the International Conference on Peer-to-Peer Computing, 2013. Abstract—Peer-to-peer systems scale to millions of nodes and provide routing and storage functions with best effort quality. In order to provide a guaranteed quality of the overlay functions, even under strong dynamics in the network with regard to peer capacities, online participation and usage patterns, we propose to calibrate the peer-to-peer overlay and to autonomously learn which qualities can be reached. For that, we simulate the peer- to-peer overlay systematically under a wide range of parameter configurations and use neural networks to learn the effects of the configurations on the quality metrics. Thus, by choosing a specific quality setting by the overlay operator, the network can tune itself to the learned parameter configurations that lead to the desired quality. Evaluation shows that the presented self-calibration succeeds in learning the configuration-quality interdependencies and that peer-to-peer systems can learn and adapt their behavior according to desired quality goals.TRANSCRIPT
Bootstrapping Skynet:Calibration and Autonomic Self-Control of
Structured Peer-to-Peer Networks
Timo Klerx and Kalman Graffi
Department of Computer ScienceUniversity of Paderborn
Research Group Knowledge-Based SystemsHans Kleine Büning
September 11, 2013
Knowledge-Based SystemsUNIVERSITY OF PADERBORN
Motivation Approach Evaluation Conclusion
Outline
1 Motivation
2 Approach
3 Evaluation
4 Conclusion & Future Work
Bootstrapping Skynet Klerx and Graffi 1/17
Motivation Approach Evaluation Conclusion
Outline
1 Motivation
2 Approach
3 Evaluation
4 Conclusion & Future Work
Bootstrapping Skynet Klerx and Graffi 2/17
Motivation Approach Evaluation Conclusion
Bootstrapping SkyNetTowards self-optimization
SkyNet: Management layer in PeerfactSim.KOM(P2P-)Systems become more and more complex
ApplicationsParametersLayers. . .
Ideally, systems manage themselvesChoose parametersDefend attacksRestore network structure. . .
Bootstrapping Skynet Klerx and Graffi 3/17
Motivation Approach Evaluation Conclusion
MAPEHow to achieve self-management?
Monitor
Analyze
Plan
Execute
Systems implementing the MAPE circuit are autonomous.Everything except Planning is already implemented.
Bootstrapping Skynet Klerx and Graffi 4/17
Motivation Approach Evaluation Conclusion
Outline
1 Motivation
2 Approach
3 Evaluation
4 Conclusion & Future Work
Bootstrapping Skynet Klerx and Graffi 5/17
Motivation Approach Evaluation Conclusion
Plan PhaseIdea
OfflineGather data by simulationLearn the interdependencies in the dataConstruct a regressor with goal as input to compute parametervalues
OnlineDefine a desired goalAsk the regressor for optimal parameter valuesChange parameter values on every node
Bootstrapping Skynet Klerx and Graffi 6/17
Motivation Approach Evaluation Conclusion
Plan PhaseIdea
OfflineGather data by simulationLearn the interdependencies in the dataConstruct a regressor with goal as input to compute parametervalues
OnlineDefine a desired goalAsk the regressor for optimal parameter valuesChange parameter values on every node
Bootstrapping Skynet Klerx and Graffi 6/17
Motivation Approach Evaluation Conclusion
Neural NetworksBasics
Classification and regression(Often) supervised learning – need labeled training dataLearn effects of parametersInput must be specified preciselyCan approximate arbitrary functions with arbitrary precision
Bootstrapping Skynet Klerx and Graffi 7/17
Motivation Approach Evaluation Conclusion
Data GenerationData characteristics
Three types of figuresEnvironment Parameters (E , |E | = 5) – Changed by all usersnode count, churn, . . .Overlay Parameters (O, |O| = 8) – Changeable by single nodesmessage timeout, max hop count, . . .Metrics (M, |M| = 18) – Performance valuesavg. hop count, avg. network messages in, . . .
View as function f : E × O → M
Bootstrapping Skynet Klerx and Graffi 8/17
Motivation Approach Evaluation Conclusion
Data GenerationCombination approaches
Full factorial designAll possible combinations of parameters∏n
i=1 |pi |Takes too much time
One factorial designOnly one parameter varied at a timeRest set to default values∑n
i=1 |pi |Few data points
Mixed factorial designTradeoff between one and full factorial designSome parameters (E ) in full factorial design, others (O) set todefault values∏s
j=1 |ej | ·∑t
k=1 |ok |
Bootstrapping Skynet Klerx and Graffi 9/17
Motivation Approach Evaluation Conclusion
Data GenerationCombination approaches
Full factorial designAll possible combinations of parameters∏n
i=1 |pi |Takes too much time
One factorial designOnly one parameter varied at a timeRest set to default values∑n
i=1 |pi |Few data points
Mixed factorial designTradeoff between one and full factorial designSome parameters (E ) in full factorial design, others (O) set todefault values∏s
j=1 |ej | ·∑t
k=1 |ok |
Bootstrapping Skynet Klerx and Graffi 9/17
Motivation Approach Evaluation Conclusion
Data GenerationCombination approaches
Full factorial designAll possible combinations of parameters∏n
i=1 |pi |Takes too much time
One factorial designOnly one parameter varied at a timeRest set to default values∑n
i=1 |pi |Few data points
Mixed factorial designTradeoff between one and full factorial designSome parameters (E ) in full factorial design, others (O) set todefault values∏s
j=1 |ej | ·∑t
k=1 |ok |
Bootstrapping Skynet Klerx and Graffi 9/17
Motivation Approach Evaluation Conclusion
Neural NetworksLearn the data characteristics
Remember function f : E × O → MReorder f to f̂ : M × E → O
M: The preferred stateE : The current environment statev ∈ f̂ : (m1, ..., mr , e1, ..., es , o1, ..., ot)
Approximate f̂ : Predict the overlay parameter values when givenenvironment state and a goalOnly realistic goals as inputTrain with resilient backpropagationOne neural network for each overlay parameterSplit data in three disjoint sets: training, validation, prediction
EnvironmentParameters
Overlay Parameter
Metrics
Hidden Layer(s)
...
...
Bootstrapping Skynet Klerx and Graffi 10/17
Motivation Approach Evaluation Conclusion
Neural NetworksLearn the data characteristics
Remember function f : E × O → MReorder f to f̂ : M × E → O
M: The preferred stateE : The current environment statev ∈ f̂ : (m1, ..., mr , e1, ..., es , o1, ..., ot)
Approximate f̂ : Predict the overlay parameter values when givenenvironment state and a goalOnly realistic goals as inputTrain with resilient backpropagationOne neural network for each overlay parameterSplit data in three disjoint sets: training, validation, prediction
EnvironmentParameters
Overlay Parameter
Metrics
Hidden Layer(s)
...
...
Bootstrapping Skynet Klerx and Graffi 10/17
Motivation Approach Evaluation Conclusion
Neural NetworksLearn the data characteristics
Remember function f : E × O → MReorder f to f̂ : M × E → O
M: The preferred stateE : The current environment statev ∈ f̂ : (m1, ..., mr , e1, ..., es , o1, ..., ot)
Approximate f̂ : Predict the overlay parameter values when givenenvironment state and a goalOnly realistic goals as inputTrain with resilient backpropagationOne neural network for each overlay parameterSplit data in three disjoint sets: training, validation, prediction
EnvironmentParameters
Overlay Parameter
Metrics
Hidden Layer(s)
...
...
Bootstrapping Skynet Klerx and Graffi 10/17
Motivation Approach Evaluation Conclusion
Outline
1 Motivation
2 Approach
3 Evaluation
4 Conclusion & Future Work
Bootstrapping Skynet Klerx and Graffi 11/17
Motivation Approach Evaluation Conclusion
OverviewWhich generation approach leads to good results?
Mixed factorial design (65,100 combinations)Most of the overlay parameter values are the default valueAlways predict "default" results in small errors
One factorial design (80 combinations)Use feature selection (CFS and PCA)Not all parameters predicted successfully
Bootstrapping Skynet Klerx and Graffi 12/17
Motivation Approach Evaluation Conclusion
Prediction QualityError while/after training
o1 =message timeouto6 =update fingertable intervalo7 =update neighbors intervalo8 =update successor interval
o2 =message resendo3 =operation timeouto4 =operation max. redoso5 =max hop count
0
20
40
60
80
100
120
o1 o2 o3 o4 o5 o6 o7 o8
Err
or in
Per
cent
Parameter
cfspca
no fs
(a) validation
0
20
40
60
80
100
120
o1 o2 o3 o4 o5 o6 o7 o8
Err
or in
Per
cent
Parameter
cfspca
no fs
(b) prediction
Figure: Error on prediction and validation setBootstrapping Skynet Klerx and Graffi 13/17
Motivation Approach Evaluation Conclusion
Prediction QualityError in the MAPE circuit without feature selection
0.12 0.14 0.16 0.18
0.2 0.22 0.24 0.26 0.28
0.3
w1 w1 w1 w2 w2 w2 w3 w3 w3
Avg
. Dur
atio
n (m
16)
Message Timeout (o1)
optimalpredicted
0 20 40 60 80
100 120 140 160 180 200
w1 w1 w1 w2 w2 w2 w3 w3 w3
Avg
. Net
Mes
sage
s O
ut (m
2)
Update Fingertable Interval (o6)
optimalpredicted
20
30
40
50
60
70
80
90
w1 w1 w1 w2 w2 w2 w3 w3 w3
Avg
. Net
Mes
sage
s O
ut (m
2)
Update Neighbors Interval (o7)
optimalpredicted
30
35
40
45
50
55
60
65
w1 w1 w1 w2 w2 w2 w3 w3 w3
Avg
. Net
Mes
sage
s O
ut (m
2)
Update Successor Interval (o8)
optimalpredicted
Bootstrapping Skynet Klerx and Graffi 14/17
Motivation Approach Evaluation Conclusion
Outline
1 Motivation
2 Approach
3 Evaluation
4 Conclusion & Future Work
Bootstrapping Skynet Klerx and Graffi 15/17
Motivation Approach Evaluation Conclusion
Conclusion
Mixed factorial design not suitableMAPE circuit closed in a proof-of-conceptFeature selection not beneficialEvaluation results are ambiguous
Good results for some parameters, bad for others
Bootstrapping Skynet Klerx and Graffi 16/17
Motivation Approach Evaluation Conclusion
Future Work
Investigate the other parametersTry full factorial design with less parameters, but more granularDesign more metricsEmbed the implemented MAPE circuit in a real systemDecentralize the neural network(s) – use local view
Thank you for your attention!
Bootstrapping Skynet Klerx and Graffi 17/17
Motivation Approach Evaluation Conclusion
Future Work
Investigate the other parametersTry full factorial design with less parameters, but more granularDesign more metricsEmbed the implemented MAPE circuit in a real systemDecentralize the neural network(s) – use local view
Thank you for your attention!
Bootstrapping Skynet Klerx and Graffi 17/17
Parameter values Evaluation
Outline
5 Parameter values
6 Evaluation
Bootstrapping Skynet Klerx and Graffi A-1
Parameter values Evaluation
Overlay Parameters
Code Overlay Parameter Unit default
o1 Message Timeout s 10o2 Message Resend # 3o3 Operation Timeout s 120o4 Operation Max. Redos # 3o5 Max Hop Count # 50o6 Upd. Finger Table Intv. ms 30o7 Upd. Neighbors Intv. ms 30o8 Upd. Successor Intv. ms 30
Bootstrapping Skynet Klerx and Graffi A-2
Parameter values Evaluation
Environment Parameters
Code Env. Parameter Unit default
e1 Node Count # 1000e2 Churn Factor # 0e3 Mean Session Length s ∞e4 Bandwidth MB/s OECDe5 Random Lookup Rate 1/h 30
Bootstrapping Skynet Klerx and Graffi A-3
Parameter values Evaluation
MetricsCode Metrics Unit
Messagesm1 Avg. Network Message In #m2 Avg. Network Message Out #m3 Avg. Transport Message In #m4 Avg. Transport Message Out #m5 Avg. Forwarded Queries #m6 Avg. Service Message Throughput #/sm7 St. Dev. Service Message Throughput #/sm8 Avg. Service Message Count #m9 St. Dev. Service Message Count #Trafficm10 Avg. Network Bytes Sent kBm11 Avg. Transport Bytes Sent kBm12 Avg. Free Upload Bandwidth kB/sPerformancem13 Avg. Hop Count #m14 Avg. Lookup Hops #m15 St. Dev. Lookup Hops #m16 Avg. Lookup Duration sm17 St. Dev. Lookup Duration sm18 Avg. Operation Duration s
Bootstrapping Skynet Klerx and Graffi A-4
Parameter values Evaluation
Parameter Variations
Code Mixed Factorial One Factorial
o1 5, 10, 20 2,3,4,5,8,10,12,15,18,20o2 0, 1, 3, 10 0,1,2,3,4,5,7,8,9o3 60, 120, 300 60,90,120,150,180,
210,240,270,285,300o4 0, 1, 3, 10 0,1,2,3,4,5,7,8,9,10o5 5, 10, 25, 50, 100 3,5,7,10,17,35,50,75,100o6 3, 10, 30, 60 3,5,7,10,15,20,30,40,50,60o7 3, 10, 30, 60 3,5,7,10,15,20,30,40,50,60o8 3, 10, 30, 60 3,5,7,10,15,20,30,40,50,60
e1 10, 33, 100, 330, 10001000, 3300, 10000
e2 0, 110 , 3
10 0e3 30, 60; 180,∞ ∞e4 OECD, random:
1-2, 5-10, 10-30, 1-30
OECD
e5 0, 1, 3, 6, 30 30
Bootstrapping Skynet Klerx and Graffi A-5
Parameter values Evaluation
Outline
5 Parameter values
6 Evaluation
Bootstrapping Skynet Klerx and Graffi A-6
Parameter values Evaluation
Process of Evaluation
ParameterValue
X
ParameterValueX‘
MetricVector
M
MetricVectorM‘
Simulation SimulationNeural
Network
compare compare
Bootstrapping Skynet Klerx and Graffi A-7
Parameter values Evaluation
Results of Evaluation
Table: Comparison of X , X ′, M and M ′
Timestamp X X′ M M′ |1− MM′ | |1− X
X′ |
For o1 and m16w1 6s 6s 0.14 0.15 0.07 0.00w2 11s 11s 0.18 0.20 0.10 0.00w3 19s 18s 0.24 0.27 0.11 0.06
For o6 and m2w1 4s 4s 175.00 173.66 0.01 0.00w2 25s 26s 39.60 38.34 0.03 0.04w3 55s 54s 24.50 25.51 0.04 0.02
For o7 and m2w1 6s 4.6s 65.80 77.27 0.15 0.30w2 25s 27.5s 36.60 35.98 0.02 0.09w3 51s 55s 31.70 31.63 0.00 0.07
For o8 and m2w1 4s 3.9s 60.21 60.90 0.01 0.03w2 37s 36s 34.27 34.10 0.00 0.03w3 55s 52s 33.21 33.00 0.01 0.06
Bootstrapping Skynet Klerx and Graffi A-8