parallelization of stochastic metaheuristics to achieve linear speed-ups while maintaining quality...
Post on 20-Dec-2015
218 views
TRANSCRIPT
Parallelization of Stochastic Metaheuristics to Achieve Linear Speed-ups while Maintaining Quality
Course Project Presentation:Mustafa Imran AliAli Mustafa Zaidi
Outline of Presentation Brief Introduction
Motivation for this problem Simulated Annealing (SA) Simulated Evolution (SimE)
Related Work Classification of Parallel Strategies Previous efforts
Our Efforts and Results Low-Level Parallelization of Both Heuristics Domain-Decomposition Parallelizations of SimE Multithreaded-Search implementations of SA and SimE
Conclusions Drawn, Future Work
Motivation Stochastic Heuristics are utilized to solve a wide variety of
combinatorial optimization problems
VLSI CAD, Operations Research, Network Design etc.
These heuristics attempt to find near-optimal solutions by performing an ‘intelligent search’ of the ‘solution space’
Each algorithm has a built in ‘intelligence’ that allows it to move towards successively better solutions.
But they usually require large execution times to achieve near optimal solutions.
Motivation Parallelization of Heuristics aims to achieve one of two basic goals:
Achieve Speedup Achieve Better Quality Solutions
Parallelization of Heuristics different from Data and Functional parallelism:
Different strategies alter the properties of basic algorithm in different ways Thus, Quality of solution achievable is sensitive to parallelization strategy
Key is to select/develop parallelization strategy that has maximum positive impact on heuristic
With respect to the goals we are trying to achieve: For better quality solutions, strategy should enhance algorithmic “intelligence”. For better runtimes, strategy should focus on reducing work-done per processor.
Our Objectives Our goal is to explore, implement and
develop parallelization strategies that allow us to:
Achieve near-linear speedup (or best effort)
Without sacrificing quality (fixed constraint)
Speedup trends should be as scalable as possible
Approaches taken and Considerations In order to effectively consider this problem, we
must look at it in terms of several aspects: The nature of the Heuristics themselves The nature of the Parallel Environment The nature of the Problem Instance and Cost Functions
All three factors influence both the Runtime, and Achievable Solution Quality
of any parallelization strategy.
For our task, we must optimize all three factors.
Introduction to Simulated Annealing
Basic Simulated Annealing Algorithm With Metropolis Loop
Introduction to Simulated Evolution
Basic Simulated Evolution Algorithm
Related Work Classification of Parallelization Strategies for
Metaheuristics [ref]1. Low-Level Parallelization2. Domain Decomposition3. Parallel or Multithreaded Search
Previous work done for Parallel SA Extensive – general as well as problem specific All three approaches tried,
Type 3 most promising – still active area of research
Previous work done for Parallel SimE Minimal – Type 2 parallelization strategy proposed, both by
designers of SimE
Our Efforts Starting Point for our work this semester:
Basic version of Type 3 Parallel SA Basic version of Type 2 Parallel SimE
Parallelization of SA: Several Enhanced versions of Type 3 Implementation of Type 1
Type 2 not implemented because…
Parallelization of SimE Several Enhancements to Basic Type 2 Implementation of Type 1 Implementation of Type 3
Basic Type 3 Parallel SA Basic Type 3 Parallel SA:
Based on Asynchronous Multiple Markov Scheme developed in [ref]
Best Type 3 scheme developed for SA to date.
Primarily intended for improving solution qualities achievable over serial version
Basic Type 3 Parallel SA Algorithm
Type 3 Parallel SA – Second Version Strategy 2 - Speed-up Oriented Type 3 Parallel SA
From the above starting point, we saw that high-quality characteristics of basic Type 3 may be exploited to produce a speed-up oriented version
Expected to be Capable of achieving quality equivalent to serial version, but not better
While providing near-linear runtimes
Near-linear runtimes forced by dividing workload on each processor by number of processors
M/p metropolis iterations instead of M.
Speed-up Oriented Type 3 Parallel SA Speedup oriented Type 3 Parallel SA
Results consistently show a 10% drop in achievable solution quality from the Serial version
Runtimes show near-linear speedups, as expected
Lessons Learned from Strategy 2 We reasoned that
The 10% quality drop occurs due to negative impact on the ‘intelligence’ of the parallel SA.
To restore achievable quality, we must counteract this effect.
Intelligence of SA: Lies in the “Cooling Schedule” [ref Dr. Sait’s book] Division of workload directly tampers with the Cooling Schedule.
Proposed Solution: Attempt to optimize the Cooling Schedule to take into account the
parallel environment – a “Multi-Dimensional” Cooling Schedule. To maintain Speedup, M/p remained unchanged, while other
parameters varied across processors
Third Version of Type 3 Parallel SA Strategy 3 – Varying Parameter settings across different
processors Expected that this would result in more effective search of the
solution space.
Several sets of parameter settings were tried Primarily by Varying and T across processors Processors with higher T and lower would perform more random
search Processors with lower T and higher would be more greedy Intermittent sharing of information should diversify search from same
position.
Results obtained from these versions: NO IMPROVEMENT OF QUALITY OVER STRATEGY 2 Show results (hadeed_1)
Lessons Learned from Strategy 3 Based on the last two attempts, we reasoned that:
M/p drastically reduces time spent searching for better solutions at a given temperature
Lower temperatures achieved quicker, Adverse effect on hill-climbing property of Heuristic.
Simple division of M/p inadequate for sustaining quality.
What to do next? Develop techniques that minimize adverse effect on algorithmic
intelligence.
Two different tracks possible1. Type 1 parallel SA2. Further Study Runtime vs Quality trends of Type 3 Parallel SA
What to do next? Type 1 Parallel SA
Pros: Leaves algorithmic intelligence intact Solution quality guaranteed.
Cons: High communication frequency adversely affects runtime Environmental factors come into play.
Further explore dynamics of Type 3 Pros:
Low communication frequency Suitable for cluster environment (show chart)
Cons: Uncharted territory - progress not assured
Type 1 Parallel SA
Low-level parallelization: Divide Cost Computation Function
Our cost function has 3 parts: Wirelength, Power, Delay First two are easy to divide
Division of Delay computation posed a significant challenge Too many replicated computations across
processors negated benefits of division. Eventually had to be excluded
Results of Type 1 Parallel SA Performance of Type 1 Parallel SA:
Abysmal!!
Reasons: 2 collective communications per iteration Amount of work divided is small compared to communication
time Communication delay increases with increasing processors
(Show Chart) Found to be completely unsuitable for
MIMD-DM environment, and Our problem instance.
p=2 p=3 p=4 p=5 p=6 p=70.566007 42.886 1816.4 2216.6
Time for Parallel SA Type 1Circuit Name
Number of Cells m(s) SA
Time for Serial SA
Further Exploring Type 3 Parallel SA To improve our achievable quality of our Type 3
parallel SA:
In depth study of the impact of parameter M on achievable solution quality.
All experiments first attempted for the Serial Version, then replicated to the parallel version
Based on what we know of Type 3 Parallelization schemes, see how any new lessons can be incorporated into it.
Impact of ‘M’ on Solution Quality (Serial)Serial Run Characteristics vs division factor
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0 50 100 150 200 250 300
time
qu
alit
y
1
9
17
25
57
Serial Run Characteristics vs division factor
0
0.1
0.2
0.3
0.4
0.5
0.6
0 5 10 15 20 25 30
time
qu
alit
y
9
17
25
57
Impact of ‘M’ on Solution Quality (Serial)
Parellel 7, vs Div Fact
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
-20 0 20 40 60 80 100 120 140
Time
Qu
alit
y
1
9
17
25
57
Impact of ‘M’ on Solution Quality (Parallel 7)
Parellel 7, vs Div Fact
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
-5 0 5 10 15 20 25
Time
Qu
alit
y
9
17
25
57
Impact of ‘M’ on Solution Quality (Parallel 7)
Observations We see that initially, fastest improvement in quality
appears with smallest M
However, Quality saturates earlier with smaller M.
Thus it might be beneficial to increase M as time progresses.
But by mow much? How to minimize runtime.
Learning From Experiments Another observation helps:
Until saturation, rate of improvement nearly constant (per metropolis calls)
This applicable for all runs for a given circuit.
Thus best way to minimize time while sustaining quality improvement:
Set value of M adaptively such that the average rate of improvement remains constant.
New Enhancement to Serial SA.
Since Type 3 parallel SA improves faster than serial version, it is expected that some speedup will be observed
Experiments still under way – parameter tuning being done
Preliminary results
For 7 processors in parallel
Adaptive Type 3 Scheme
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
-10 0 10 20 30 40 50 60 70 80 90 100
time
qu
alit
y
serial
Parallel 7
Preliminary Results
Results are similar to the original implementation
Enhancement to serial SA mitigates the observed benefits of the parallel version Although further parameter tuning/code refinements may
improve parallel results even more.
Conclusions Drawn from Experiments Type 3 parallelization of SA is most robust
Minimum susceptibility to environment and problem instance Type 1 fails due to unsuitability to environment and to problem
instance
For Parallel SA, direct trade off exists between achievable solution quality and speedup
Depending on quality desired, linear speedup possible. For highest quality, speed-up diminishes to 1 in most cases.
Further experimentation needed to verify these points.
Parallel Simulated Evolution Strategies
Base Work for SimE
Type II (Domain Decomposition) implementation
Problem: Poor Runtimes with Quality Degradation
Improvement Proposed: Use of Random Row Allocation over the Fixed Row Allocation used in previous work
Results: Improvement of Quality over Fixed Row Allocation but still short of Serial Quality
Type II Quality Issues
Observation: Parallel Quality will always lag behind serial quality
Reason: Division of Solution into Domains restricts optimum cell movement (worsens with more processors/partitions)
Focus on improving runtime!
Type II Runtime Improvements
How? Reduce Communication time
Reduce Frequency of Communications Reduce Amount of Data Communicated Overlap Communication with Computation
Reduce Computations Can Workload be still better divided? How will workload division affect communication
requirements?
Type II SimE Algorithm Structure
Communication
Cost Computations
Type II Communication Optimization No potential for Computation Communication
Overlap Implicit Barrier synchronization at communication points
Possibility of Reducing Communication Frequency Over Multiple Iterations Do multiple operations (E,S,A) on assigned partition before
communicating Impact on Solution Quality due to accumulated errors Actual Impact on Solution Quality vs. runtime improvement
not presently quantified (future work)
Type II Communication Optimization(2) Reducing Communication Frequency
by combining gather & broadcast operation into 1 MPI call
Efficiency of collective call not too superior in MPI implementation used
Reduce Data Communicated per call Significant impact on runtime due to barrier
synchronization Essentially compressing placement data! Tradeoff: Added computations in data
preprocessing and post processing step
Type II Computation Optimization Can workload be better divided?
Evaluation, Selection and Allocation already localized (divided)
What about cost computation? Goodness Evaluation needs costs computed
Dependencies across partitions Delay computations over long paths
Spans over partitions Wire length & Power Computation
Potential for cost computation division
Wirelength & Power Cost Division More independent of other partitions than
delay computations Effect of computation division can be readily
evaluated for wire length and power (within the limited time constraints)
Tradeoff: Added Computation Partial Costs Need to be communicated to Master Additional communication phase added per
iteration! Actual benefit over non-division: Results will tell!
Results & Observations
Effect of communication worsens speed-ups with increasing processors communication time over shadows gain of computation
division Optimizing communication resulted in greater gains
than cost computation division Dependencies do not allow much gains
Original Type II
Implementation
Communication Optimized Type II
Communication + Computation Type II
Circuit Cells Quality
P=1 P=3 P=5 P=3 P=5 P=3 P=5
s1196 561 0.644 54 sec
~44 sec
~50 sec ~34 sec
~44 sec ~30 sec ~41 sec
S1238 540 0.680 60 sec
~45 sec
~55 sec ~39 sec
~46 sec ~34 sec ~42 sec
Targeting Better Solution Qualities Type II parallelization has poor quality issues How to maintain quality? Type I Parallelization
Same Quality as Serial Implementation guaranteed
Speed-ups governed by gains resulting from division of cost-computation
Type III Parallelization Can we benefit from parallel co-operating
searches?
Type I Parallelization
Target as much computation division without dividing SimE algorithm’s core “intelligence”
Computations that don’t affect intelligence Cost computations Goodness evaluations
Computations that affect intelligence Selection function Allocation function
Type I Parallelization
Again, wire length & power easier to divide than delay (same reasoning as for Type II)
Workload division achieved by dividing the cells among PE Each PE computes costs and goodness for a
fraction of cells Division of cells done at beginning & fixed Slave PEs communicate goodness values to
master PE which does Selection & Allocation
Type I Parallelization Results
Runtimes not drastically reduced Allocation takes significant portion of overall
computation time Speed-ups again limited by communication
Not by data communicated but due to more overheads with increased participants
SimE Type I Parallelization
Circuit
Cells
Quality
P=1 P=3 P=5 P=7
s1196 561 0.762 ~67 sec
~53 sec
~44 sec
~45 sec
s1238 540 0.799 ~72 sec
~49 sec
~35 sec
~40 sec
Type III Parallelization
Parallel Cooperating searches communicating best solutions through a master
Modeled after Parallel SA A Metropolis loop can be equated with a SimE
compound move Intelligence of SimE can benefit from best
solution produced among all May lead to more rapid convergence to better
quality solutions
Type III Parallelization Results
Results quite contrary to expectations Highest quality inferior than serial algorithm No speed-ups observed Parallel Searches used as is doesn’t benefit
possible explanation: greedy behavior of parallel searches is counter productive to SimE intelligence (inability to escape local minima)
SimE Type III Parallelization
Circuit
Cells
Quality
P=1 P=3 P=5 P=7
s1196 561 0.694 ~58 sec
~57 sec
~60 sec
~61 sec
s1238 540 0.709 ~65 sec
~66 sec
~64 sec
~67 sec
Improving Type III Parallelization Proposed schemes for improved SimE parallel
searches
1. varying frequency of communication with time Initially exchanges can be frequent and with frequency
decreasing to allow more diversification
2. Intelligently combining good elements in each solution to get a new starting solution
Best location for each cell (or a cluster) can be identified by examining/comparing goodness values among solutions received from peers and constructing a good solution to improve further