Quality-based Adaptive Resource Management Architecture (QARMA):

A CORBA Resource Management Service

Presenter: David Fleeman { fleeman@ohio.edu }

D. Fleeman, M. Gillen, A. Lenharth, M. Delaney,L. Welch, D. Juedes and C. Liu

Center for Intelligent, Distributed & Dependable SystemsOhio University

Athens, OH

WPDRTS 2004April 26, 2004

2

Presentation Overview

• QARMA– Overview of Architecture– GME/VEST-based Modeling Tool– System Repository Service– Resource Management Service– Enactor Service– Experimental Results

• Integration Efforts– RT CORBA Dynamic Scheduling Service (URI)– Utah CPU Broker (Utah)

• Technology Transition– DARPA PCES BBN OEP (UAV)– DARPA ARMS MLRM component– Place services into TAO

• Conclusions and Future Work

3

Generic Resource Management Architecture

ConfigurationFiles

Monitors

Detectors

InformationRepository

Decision-Maker

Enactors

Computing Environment &User Applications

EnvironmentResource InstrumentationSoftware Instrumentation

Resource and SoftwareManagement Commands

4

CORBA Resource Management Architecture

Resource Management

Service

EnactorService

System Repository

Service

ConfigurationFiles

SpecificationTool

Resource Management Architecture

Application Layer (independent of QoS mechanism)

Replaceable Components

Enactor(Quality Connector)

Host and NetworkMonitor / Detector

Software Performance

Monitor / DetectorHosts/Networks Enactor

(Quality Connector)Enactor

(Quality Connector)

Software Performance

Monitor / Detector

Software Performance

Monitor / Detector

Host and NetworkMonitor / Detector

Host and NetworkMonitor / Detector

TrackingATRUAV VideoResource Management Service Goal

• Each chain of applications achieves a “benefit” based on it’s service attribute settings (e.g., frame rate) and extrinsic attribute settings (e.g., importance).

• The Resource Management Service changes service attributes in order to optimize total benefit, subject to the constraint that all tasks meet their real-time requirements.

QARMA Components

5

QARMA System Repository Service• Purpose

– Central repository for communicating all specification and state data to management components

• Benefits– Reduced complexity of other components– Other components become stateless– Need for replication and fault tolerance of other components is

decreased– Flexibility in replacing storage mechanism behind System Repository

(e.g., memory, XML files, database)– Integration with components developed by other organizations hinges

only on agreement of the data stored in the System Repository

• Drawbacks– Becomes a single point of failure– Not scalable for arbitrarily large systems– These drawbacks can be handled with standard fault-tolerance and

replication mechanisms.

6

QARMA Resource Management Service

• Purpose– Translate specification into algorithmic model.– Make intelligent allocation decisions.

• Benefits– Ability to perform global feasibility analysis for tasks

that require multiple resources.– Ability to perform global optimization for service

quality settings.

• Drawbacks– Not scalable for arbitrarily large systems.

• Hierarchical decision-makers, such as the one presented this morning may be used in place of this component.

7

QARMA Resource Management ServiceCode Snippets

IDL Interface

module QARMA { … interface RM_Service { typedef sequence<string> Hint; typedef sequence<Hint>

Hint_seq; void reactive_rm(in Hint_seq

hints); }; …};

Invocation by Host Detector

// Create a hint for the RM ServiceQARMA::RM_Service::Hint_seq hints;hints.length(1);hints[0].length(2);hints[0][0] =

CORBA::string_dup(“Overload”);hints[0][1] =

CORBA::string_dup(“host1”);

// Invoke the RM Servicerm_service->reactive_rm(hints);

8

QARMA Resource Management ServiceThe Key Problems

• Information Acquisition

• Information Modeling

• Resource Allocation Algorithms– Feasibility Testing– Optimization Objective– Control Mechanisms– Pluggable

9

Information Acquisition

• Obtained from System Repository Service

– Resource Specification:• Describes hosts, network devices, and network connectivity.

– Software Specification:• Describes the extrinsic attributes that affect performance of

applications and the service attributes that can be used to control application resource usage and quality.

• Describes the application connectivity and dependencies, performance requirements, and performance characteristics.

– Monitoring Data:• Host and network resource usage state and statistics• Software performance state and statistics

10

Information Modeling

• Specification data is transformed into mathematical models.– Based on the models presented in chapters 3 and 9

of Jane Liu’s “Real-Time Systems” book.– Service and extrinsic attributes have been added to

extend the model presented in the book.

• Four models are created:– Resource Model– Task Model– Profile Model– Benefit Model

11

Example: Specification => Data Structure => Model

Host ou4.cidds {Memory 256 MB;…Network {

Interface eth0 { Name “10.0.0.11”; …

} } }Host ou3.cidds {

…Network {

Interface eth0 {…

} } }Switch lan1 {

Network {Interface eth0;Interface eth1;

} }Network Topology {

Link { Interface lan1 eth0;Interface ou4.cidds

eth0;Speed 100 MB/sec;Latency 0.0 sec;

} … }…

struct NetworkInterface { string name; … }struct Host {

string name; long memory;sequence<NetworkInterface> interfaces;

}typedef Host NetworkSwitch;struct NetworkLink {

sequence<StringPair> source_interfaces;sequence<StringPair> sink_interfaces;double link_speed;doulbe link_latency;

}

ou4.ciddswith

loopbackinterface

ou3.ciddswith

loopbackinterface

switchlan1

full-duplex

link

full-duplex

link

H

I

L

L

SL

L

H

I

12

QARMA Control Mechanisms

• Long term goal is to have all the following:– Service Level Adaptation– Process Allocation (Startup capabilities)– Process Migration (Ability to stop and restart processes)– Process Replication– Dynamic Network Routing– Network Reservation and Prioritization (IntServ / DiffServ)– CPU Reservation and Prioritization (Utah CPU Broker)

• Currently implemented in QARMA:– Service Level Adaptation– Process Allocation (simplest)– Process Migration (simplest)

13

Example Resource Allocation Algorithm:Greedy Service Level Optimization

• INPUT:– Resource Model, Task Model, Profile Model, Benefit Model

• OUTPUT:– Setting of Service Levels such that the resource utilization on any

resource is less than 69% and utility is maximized.

• ALGORITHM– Choose lowest quality settings for all tasks

– Rank/order systems by importance

– For each system in order of importance:• Set service level to the highest• While not feasible, decrease service level

– Return service level settings

14

QARMA Enactor Service

• Purpose– Enact changes to the allocation of resources as directed by the

Resource Management Service.

• Benefits– Delegates change directives to lower-level enactors.– Specification for enactors allows enactment mechanisms to be

swapped out.

• Drawbacks– ???

15

Integration with the BBN OEP

Resource Management

Service

EnactorService

System Repository

Service

ConfigurationFiles

SpecificationTool

syscondQuO

Contracts

QuO Kernel

read

Resource Management Architecture

syscond

syscond

syscond

Application LayerQuO Middleware

QuOContracts

VIDEO DISTRIBUTIONHOST

VideoDistributor

Video Display

VIDEO DISPLAY HOST

VideoDisplayProxy

QuO

Video Stream

QuO

VideoSourceProcess

MOBILE VIDEOSOURCE HOST

Video File

QuO

Application LayerBBN OEP UAV System

Replaceable Components

set

Host and NetworkMonitor / Detector

Software Performance

Monitor / Detector

EventChannel

Hosts/Networks

** System Condition objects were added.** Contracts were modified to allow RM Service to force a chosen region.

Host and NetworkMonitor / Detector

Host and NetworkMonitor / Detector

Quality Connector Instance

(Quo Enactor)

16

Global Resource Mgmt. Demo Configuration

OU1

CORBAServices

CORBA RMServices

UAV1

Host Mon

OU3

Host Mon

OU2Host Mon

UAV2

Distrib1

Distrib2

Receiver1

Viewer1

Receiver2

Viewer2

LOAD OU4

• There are two Sender-Distributor-Receiver streams.

• Receiver1 is set to the highest priority (because it is viewing a target).

• Use Hourglass to apply CPU load on the Receiver node.

• Load increases every 90 seconds as follows:

– 40% at time 40– 60% at time 130– 90% at time 220– 98% at time 310– 100% at time 400

QuO Managed CPU Load - Frame Rate - Viewer1

0

5

10

15

20

25

30

35

40

0 25 50 75 100 125 150 175 200 225 250 276 301 327 352 378 405 431 458 483

Seconds

Fra

me

Rat

e

QARMA Managed CPU Load - Frame Rate - Viewer1

0

5

10

15

20

25

30

35

40

0 25 50 75 100 125 150 175 200 232 257 282 310 335 360 385 410 435 460 485

Seconds

Fra

me

Ra

te

Local RM Global RM

QARMA Managed CPU Load - Frame Rate - Viewer2

0

5

10

15

20

25

30

35

40

0 25 50 75 100 125 150 175 200 231 256 281 306 331 356 381 406 431 456 481

Seconds

Fra

me

Ra

te

QuO Managed CPU Load - Frame Rate - Viewer2

0

5

10

15

20

25

30

35

40

0 25 50 75 100 125 150 175 200 225 250 276 301 327 351 378 404 431 458 483

Seconds

Fra

me

Ra

te

Global RM vs. Local RM - CPU Load Experiment

Frame Rate

0

5

10

15

20

25

30

35

1 2 3 4

Quo Viewer 1

Quo Viewer 2

QARMA Viewer 1

QARMA Viewer 2

Frame Rate Average (frames per second)

Baseline Viewer1 5.48

Baseline Viewer2 6.01

QARMA Viewer1 8.73

QARMA Viewer2 2.67

Global RM vs. Local RM - CPU Load Experiment

Frame Rate Results

Global RM vs. Local RM - CPU Load Experiment

LatencyQuO Managed CPU Load - Latency - Viewer1

0

1000

2000

3000

4000

5000

6000

0 25 50 75 100 125 150 175 200 225 250 276 301 327 352 378 405 431 458 483

Seconds

Mic

ros

ec

on

ds

CuO Managed CPU Load - Latency - Viewer2

0

1000

2000

3000

4000

5000

6000

0 25 50 75 100 125 150 175 200 225 250 276 301 327 351 378 404 431 458 483

Seconds

Mic

ros

ec

on

ds

QARMA Managed CPU Load - Latency - Viewer1

0

1000

2000

3000

4000

5000

6000

0 25 50 75 100 125 150 175 200 232 257 282 310 335 360 385 410 435 460 485

Seconds

Mic

ros

ec

on

ds

QARMA Managed CPU Load - Latency - Viewer2

0

1000

2000

3000

4000

5000

6000

0 25 50 75 100 125 150 175 200 231 256 281 306 331 356 381 406 431 456 481

Seconds

Mic

ros

ec

on

ds

Local RM Global RM

Global RM vs. Local RM - CPU Load Experiment

Latency Results

Latency Average

Latency Max Standard Deviation

Baseline Viewer1 236 ms 5770 ms 624 ms

Baseline Viewer2 229 ms 5751 ms 621 ms

QARMA Viewer1 106 ms 3270 ms 321 ms

QARMA Viewer2 89 ms 3479 ms 292 ms

0

50

100

150

200

250

Mic

ros

ec

on

ds

Baseline QARMA

Average Latency

0

100

200

300

400

500

600

700

Quo Viewer 1

Quo Viewer 2

QARMA Viewer 1

QARMA Viewer 2

Standard Deviation

Baseline

V1

Baseline

V2

QARMA

V1

QARMA

V2

Global RM vs. Local RM - CPU Load Experiment

Conclusions

• Applications controlled by the global RM

• demonstrate greater stability

• exhibit greater determinism

• have lower average latency

• delivered

• higher frame rate for more important streams

• higher average frame rate

22

Future Work

• Integrate other resource management mechanisms

• Investigate and develop more advanced allocation algorithms

• Provide fault tolerance and stabilization of the Resource Management Service as well as reflexive management.

• Continue integration, transition and validation efforts– Validation with RMBench– Validation with PCES OEP (Boeing and BBN)– Integration with PCES technology

• Scheduling Service, Kokyu, CPU Broker– Transition into DARPA ARMS MLRM infrastructure– Transition into TAO as CORBA Services

23

Envisioned PCES Component Integration

. . . . . .RT CORBA scheduling

pointAdaptation Parameter

Adjustments(i.e., frame rate)

Resource Management Service (OU)

EnactorService (OU)

System Repository

Service (OU)

ConfigurationFiles

SpecificationTool (OU/URI)

Resource Management Architecture

Other PCES Components

Enactor(Quality Connector)

Host and NetworkMonitor / DetectorScheduling

Service (URI)

Hosts/Networks

Host and NetworkMonitor / Detector

Resource Monitoring (KU)

Enactor(Quality Connector) Quality Enactor

(OU)

Application Layer

TrackingATRUAV Video

RT CORBAPluggableScheduler

Kokyu (WUSTL)

Resource Enactors (KU)

Resource Enactors (KU)

Resource Enactor (KU)

CPU Broker (Utah)

CPU Broker (Utah)

CPU Broker (Utah)