cpu utilization control in distributed real-time systems

SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.

CPU Utilization Control in Distributed Real-Time Systems

Chenyang LuDepartment of Computer Science and Engineering

2SIGMETRICS 2008: Introduction to Control Theory. Abdelzaher, Diao, Hellerstein, Lu, and Zhu.

Why CPU Utilization Control?Overload protection

CPU over-utilization system crashNightmare for mission-critical applications and

“always-on” E-businesses

Meet deadlinesCPU utilization < schedulable utilization bound


End-to-End Task Modelin Distributed Real-Time Systems

Periodic task Ti = a chain of subtasks {Tij} located on different processors Subtasks run at a same rate

Task rate can be adjusted within a range Higher rate higher utility

Remote Invocation

Subtask

T1

T2

T3

T11 T12 T13

P1P2 P3


Bi: Utilization set point of processor Pi (1 ≤ i ≤ n)

ui(k): Utilization of Pi in kth sampling period

rj(k): Rate of task Tj (1 ≤ j ≤ m) in kth sampling period

subject to rate constraints:

Rmin,j rj(k) Rmax,j (1 ≤ j ≤ m)

Problem Formulation

n

iii

njkrkuB

j 1

2

}1)|({))((min


Challenge: Uncertainties Execution times?

Unknown sensor data or user input

Request arrival rate? Aperiodic events Bursty service requests

Disturbance? Denial of Service Attacks

Control-theoretic approaches to adaptive softwareRobust performance in face of workload uncertainty


Single-Processor Solution:

Feedback Control Real-Time Scheduling

Adaptation based on single-input-single-output control

Monitor

Processor

OS

Application

Sensor Inputs

Set pointUs = 69%

Task RatesR1: [1, 5] HzR2: [10, 20] Hz

Middleware

ActuatorController

u(k)

{r(k+1)}

FCS

C. Lu, X. Wang, and C. Gill, Feedback Control Real-Time Scheduling in ORB Middleware, IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS'03), May 2003.


What’s New in Distributed Systems? Need to control utilization of multiple processors Utilization of different processors are coupled with each

other due to end-to-end tasksReplicating FCS on all processors does not work!

Constraints on task rates

T1

T2

T3

T11 T12 T13

P1P2 P3


EUCON: Multi-Input-Multi-Output Control

)(

)(1

kr

kr

m

ModelPredictiveController

1 min,1 max,1

min, max,

,

n m m

B R R

B R R

)(

)(1

ku

ku

n

Distributed System

(m tasks, n processors)

UtilizationMonitor

RateModulator RM

UM UM

RM

Feedback Loop

Precedence Constraints

Subtask

ControlInput

MeasuredOutput

C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in Distributed Real-Time Systems with End-to-End Tasks, IEEE Transactions on Parallel and Distributed Systems, 16(6): 550-561, June 2005.


Control Theoretic Methodology

1. Derive a dynamic model of the controlled system

2. Design a controller

3. Analyze stability


Dynamic Model: One Processor

Si: set of subtasks on Pi

cjl: estimated execution time of Til running on Pi

may not be correct gi: utilization gain of Pi

unknown ratio between actual and estimated change in utilization

models uncertainty in execution times

ijl ST

jjliii krcgkuku )1()1()(


Dynamic Model: Multiple Processors

G: diagonal matrix of utilization gains F: subtask allocation matrix

models the coupling among processors fij = cjl task Tj has a subtask Tjl on processor Pi

fij = 0 if Tj has no subtask on Pi

u(k) = u(k-1) + GFr(k-1)

T1

T2

T11

P1P2

T21

T22

T3

T31

3122

2111

0

0

cc

ccF


Model Predictive Control Advanced control technique for coupled MIMO control

problems with actuator constraints. Minimize a cost function over an interval in the future.

Compute an input trajectory that minimizes cost subject to actuator constraints.

Predict cost based on a system model and feedback.

Combines optimization, model-based prediction, and feedback.


Model Predictive Controller

At a sampling instant Compute inputs in future sampling periods

r(k), r(k+1), ... r(k+M-1)

to minimize a cost function Cost is predicted using

(1) feedback u(k-1)

(2) approximate dynamic model Apply r(k) to the system

At the next sampling instant Shift time window and re-compute r(k+1), r(k+2), ...

r(k+M) based on feedback


Model Predictive Controller in EUCON

Least Squares Solver

)(

)(1

ku

ku

n

)(

)(1

kr

kr

m


SystemModel

CostFunction

ReferenceTrajectory

nB

B

1 Rate

Constraints

Least Squares Solver

)(

)(1

ku

ku

n

)(

)(1

kr

kr

m


SystemModel

CostFunction

ReferenceTrajectory

nB

B

1 Rate

Constraints

Difference with reference trajectory

Desired trajectory for u(k) to converge to B

Constrained optimization solver

)1(

)1(1

kr

kr

m


Stability Analysis Stability: system converges to equilibrium point from any

initial condition

Equilibrium point = utilization set points B

If stable, utilization of all processors converge to their set points whenever feasible

Derive stability condition tolerable range of G

tolerable variation of execution times

Stability analysis establishes analytical guarantees on utilization despite uncertainty


Simulation: Stable System

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Time (sampling period)

CP

U u

tiliz

atio

n

P1 P2 Set Point

execution time factor = 0.5(actual execution times = ½ of estimates)


Simulation: Unstable System

execution time factor = 7(actual execution times = 7 times estimates)

0

0.2

0.4

0.6

0.8

1

0 100 200 300

Tim e (sam pling period)

CP

U u

tiliz

atio

n

P1 P2 Set Point


Stability Stability: system converges to desired utilizations from any initial

condition

Derive stability condition tolerable range of execution times

Analytical assurance on utilizations despite uncertainty

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

0 2 4 6 8 10

CP

U u

iliz

ati

on

execution-time factor

Courier, 24Courier, 24Courier, 24Courier, 24Courier, 24Courier, 24Courier, 26Courier, 30Courier, 30Courier, 24Courier, 24Courier, 30Courier, 30

DeviationAverageSet point

Overestimation of execution times prevents oscillation

Predicted bound for stability

actual execution time / estimation


FC-ORB Middleware

Feedback lane

Remote request lanes

Priority Manager

RateModulator

ModelPredictiveController

Remote request lanes

Utilization Monitor

)(

)(

)(

3

2

1

ku

ku

kuMeasuredOutput

)(

)(

2

1

kr

krControlInput

Priority Manager

RateModulator

Utilization Monitor

Priority Manager

RateModulator

Utilization Monitor

X. Wang, C. Lu and X. Koutsoukos, Enhancing the Robustness of Distributed Real-Time Middleware via End-to-End Utilization Control, IEEE Real-Time Systems Symposium (RTSS'05), December 2005.


Workload Uncertainty

time-varying execution times

disturbance from periodic tasks


Processor Failure

1. Norbert fails.2. move its tasks to other processors.3. reconfigure controller4. control utilization by adjusting

task rates


Summary: Model Predictive Control

Robust utilization control for distributed systems Handles coupling among processors Enforce constraints on task rates Analyze tolerable range of execution times


References Centralized control: EUCON

C. Lu, X. Wang and X. Koutsoukos, Feedback Utilization Control in Distributed Real-Time Systems with End-to-End Tasks, IEEE Transactions on Parallel and Distributed Systems, 16(6): 550-561, June 2005.

Decentralized control: DEUCON X. Wang, D. Jia, C. Lu and X. Koutsoukos,

DEUCON: Decentralized End-to-End Utilization Control for Distributed Real-Time Systems, IEEE Transactions on Parallel and Distributed Systems, 18(7): 996-1009, July 2007.

Middleware: FC-ORB X. Wang, C. Lu and X. Koutsoukos,

Enhancing the Robustness of Distributed Real-Time Middleware via End-to-End Utilization Control, IEEE Real-Time Systems Symposium (RTSS'05), December 2005.

Controllability and feasibility X. Wang, Y. Chen, C. Lu and X. Koutsoukos,

On Controllability and Feasibility of Utilization Control in Distributed Real-Time Systems, Euromicro Conference on Real-Time Systems (ECRTS'07), July 2007.

Project page: http://www.cse.wustl.edu/~lu/aqc.htm

cpu utilization control in distributed real-time systems

Documents