resource selector

Resource Selector

Chuang Liu

What do we want to do?

• A smart Resource Selector

App

R S

Resource requirement

What do we want to do?

App

R SThese Resources seem fit your requirement best

Goal

• A Resource Selector for general purpose• Matching between application’s requirement and a

set of resources• Adaptability to dynamic status of distributed

environment• Support ownership of resource • Support performance model of application• The interface between application and RS should

be simple

RS--Structure

Resource

Selector GRIS

GIISMDSAp

p

RS

MatchMaker

RequirementResource

A subset of resource

App

Challenges

• How to specify resource and request and how to match request with resource.

• Consistency of data in RS and system status.

• How to choose N resources from M available resources. (N <=M)

Challenges

• How to specify the request and resource and how to match the request with resource

• Consistency of data in RS and system status.

• How to choose N resources from M available resources. (N <=M)

ClassAd—Mechanism to specify resources and request

• The classad mechanism is a language for expressing and evaluating attribute

• A classad is a set of named expressions

• Each named expression is called an attribute

• Expression similar to those found in C/C++

ClassAd : example(resource)

• [• OpSys=“LINUX”;• Name= “trapezius.cs.uchicago.edu”;• LoadAvg= 0.03;• Friends = (“foster”, “dave”);• Untrusted = (“evil”, “rival”);• Constraints= !member(other.Owner, Untrusted) &&

(LoadAvg < 0.3);• ]

ClassAd : example(request)

• [

• Owner=“chliu”;

• Requirements= other.LoadAvg < 0.3 && other.opSys=“LINUX”;

• Rank = 1/other.LoadAvg;

• ]

Challenges

• How to specify the resource and request and how to match the request with resource.

• Consistency of data in RS and system status.

• How to choose N resources from M available resources. (N <=M)

Consistency

• Several threads in RS update information about system status based TTL value.– Update information about available resource by

access GIIS– Update information about status of every

resource by access GRIS or GIIS• Tradeoff between Performance and Consistency

Challenges

• How to specify the resource and request and how to match the request with resource.

• Consistency of data in RS and system status.

• How to choose N resources from M available resources which fit application’s requirement best. (N <=M)

Resource Selection

• How to select N resource from M available resources efficiently.

• How to judge which one is best among several matched results

Resource Selection

• How to select N resource from M available resources efficiently.

• How to judge which one is best among several matched results

Criteria to judge the desirability of resource

• Performance model– F(resource Info, application info)

– [ minCPUSpeed > 10 MIPS

– minMemSize > 100 MB

– Rank= minCPUSpeed * NumOfResource

– ]

– Embed a program or function call in Classad• ? Classad don’t support function call in expression.

Resource Selection

• How to Select N resources from M available resources

• How to judge which one is best among several matched results

Bilateral match- Clique

• Organize all available resources into several cliques.

Classads for resource

Classads for clique

Classads for requirement

Match maker

Clique- How to organize clique

Methods to organize clique– Manually– Automatically– Pros:

• Easy and useful

– Cons:• Not flexibility

Clique- Naive

• Naive method– For example:

• Resource: { a, b, c}

• Cliques {a}, {b}, {c}, {a, b}, {b,c}, {a,c}, {a, b, c}

– Cons• The number of clique is 2 to N

Gang Match

• Gang Matching 1 N

Classads for resource

Classads for requirement

Match maker

Gang match- Greedy Algorithm

• Greedy Algorithm– Clique, candidate = null– Match the requirement with every resource– Choose resource with highest ranking as the first number of clique– For(;;) {– If (clique match requirement) and (performance of application increase)– Candidate=clique– Match the requirement with resource which is not in clique– Add node with highest ranking in the left nodes to clique– Else– Return Candidate – }

Gang match- Greedy Algorithm

• Pros:– High performance– Give pretty good optimal result to loosely

coupled application

• Cons:– Locally optimal choice does not always lead to

globally optimal solution

Gang match- Port and docking

• Ports and docking• [ Ports = {• [ Label = Host1;• Requirements= MemorySize > 17.2M;• Constraint = Host1.Arch == "INTEL" && Host1.OpSys == "LINUX";• Rank = Host1.MIPS• ],• [ Label = Host2;• Requirements= MemoryReqs > 18 M;• Constraint = Host2.Arch == "INTEL" && Host2.OpSys == "LINUX"

&&• Host1.Subnet == Host2.Subnet• Rank = Host2.KFlops• ]• Rank = 1 * Host1.MIPS + 8 * Host2.Kflops;• } ]

Gang match- Port and docking

• Pros:– Internal mechanism provided by Classad– ? Available in Classad package

• Cons:– Match performance– Application need to specify how many nodes it

wanted

Gang match- Dynamic programming

• Limited resource use– System administrator control how many

resources user can use– User’s requirement

• [ performance > threadhold value && number of resource is as little as possible]

– Application’s requirement • App specify how many CPU it wanted in request• A 4 X 4 X 4 Grid calculation, numOfCPU < 64

Gang match Dynamic programming

• Multi-dimensions knapsack problem– Knapsack problem : “There are M items, every

item has a Weight and a Value. Try to choose items from these items such that their value is maximum and their total weight is less than W.”

Gang match- Dynamic programming

• Pros:– More flexible– Always provide good resource to application– polynomial time algorithm o(N)

• Cons:– Performance is bad than greedy algorithm

Project Status

• A RS is running, but not smart enough.– Two match strateges have been implemented

• Clique

• Greedy algorithm

Open problem

• How well does RS work?– Performance of application on selected resource– Cost of RS

• Evaluate different strategy in match making

Welcome comment and suggestion

Thank you!