Cadence Design Systems, Inc.

Reliable Estimation of Execution Time of Embedded SW: A Statistical Approach Grant Martin, Paolo Giusto and Ed HarcourtFront End Products: System Level Design and Verification26 October 2000

2

AgendaMotivation for SW estimation in HW-SW codesignContext

– Various estimation techniquesThe Statistical approachPreliminary results Ideas for the future

3

Motivation

Improve Quality of EstimatorsProvide Quality-Speed Options Communication

Refinement

Flow To Implementation

Mapping

SystemFunction

SystemArchitecture

PerformanceSimulation

FunctionalSimulation

21

3

4

Function-Architecture Co-Design, as in Cadence VCC

SoftwareProcessor

Estimate of Execution Time

Needed to makeHW-SW tradeoffs

and to estimate ‘fit’of SW to Processor(s)

4

Context: SW Estimation TechniquesTechnique Static vs.

DynamicAccuracy(Error)

InstructionAbstraction

ModellingEffort

Speed

VirtualProcessorModel

Dynamic +/- 20% ? ‘Virtual’ or‘Fuzzy’Instructions

Data Sheet Very Fast

CompiledCode ISS

Dynamic +/- 5% Actualinstructionstream

Extensive-many months.May optimisefor speed byCutting somecorners

Fast

InterpretiveISS

Dynamic +/- 2% Actualinstructionstream

Less thanCompiledCode ISS

Slow

MathematicalKernel function

Static Can be as lowas a few %

None. Must beno datadependencies

A few hoursto weeks

Fastest

Back-annotation

Static If datadependencylow, few %

None. Measurecode on ISS orHW

Few hours toweeks; late

Very Fast

5

Virtual Processor Model Estimation

PerformanceEstimation

Compile

generated C andrun natively

ld ld op ld li op ts -- br

Analyse

basic blockscompute delays

Virtual MachineInstructions

ArchitectureCharacterization

ŽGenerate new C

with delay annotations

v__st_tmp = v__st;__DELAY(LI+LI+LI+LI+LI+LI+OPc);startup(proc);if (events[proc][0] & 1) {__DELAY(OPi+LD+LI+OPc+LD+OPi+OPi+IF); goto L16;}

ŒSpecify behavior

and I/O

ANSI CInput

v__st_tmp = v__st;startup(proc);if (events[proc][0] & 1) goto L16;

Evolution of approaches used in POLIS

6

Virtual Processor Model ExampleXXX Virtual Instruction Set Model (Basis file)

LD,3 LoadLI,1 Load Immdiate.ST,3 StoreOP.c,3 Simple ALU OperationOP.s,3OP.i,4OP.l,4OP.f,4OP.d,6MUL.c,9 Complex ALU OperationMUL.s,10MUL.i,18MUL.l,22MUL.f,45MUL.d,55DIV.c,19DIV.s,110DIV.i,118DIV.l,122DIV.f,145DIV.d,155IF,5 Test and BranchGOTO,2 Unconditional BranchSUB,19 Branch to SubroutineRET,21 Return from Subroutine

Basis filederived from DataBook Register Allocation Technique

– None, LocalScalarsOnly, LocalScalarsAndParameter, ParameterOnly

7

Problems with VPM approachCannot incorporate Target-dependent compiler optimisations

– Special HW, register set (accurately)Does not model processor state

– Pipelines, register occupancyVery Unclear how to handle Superscalar, VLIW, etc. Improvements to model target HW and compilers slow it down

– Can add ‘statistical’ fudge factors - hard to characterise ‘Field of Use’ vs. ‘Field of Abuse’ hard to characterise

– Most suitable for Control-dominated code, not computation-dominated (compiler optimisations and special HW effects)

– Databook approach VERY conservative….worst-case assumptions– What is the confidence level for the estimation?

8

Compiled Code ISS

Target.c

Target

CC

Target.s

Other.o

CC-ISSCC-ISS

Host.c

Host

CC

Host.o

Libraries

Host.exe

Other.s

Host

Linker

Options

One approach to a Compiled Code ISS translates Target assembler code into portable C code that when executed on the Host will reasonably accurately emulate the Timing, Functionality and Memory transaction behavior of the Target Architecture for the specific program --Target.c

9

Problems with Compiled Code-ISSModeling!

– Generation of Compiled code ISS models and tooling often very time-consuming– many person-months to person-years

Slower than VPM approach– Modeling of internal processor state, HW resources etc. comes at a cost– May be 2-5X slower (various approaches available)

Some modelling compromises made to achieve speed– Interpretive ISS approach can be more accurate– BUT different levels of accuracy can be achieved at slower speed– Areas not proven well: modeling parallel resources and HW scheduling

10

Interpretive ISS

Execute Code stream in interpretive fashion on model of processor

– usually C/C++ based model– models processor state and all internal resources– can interact with memory and bus models to achieve high

accuracyModels very often produced by IP providers or 3rd parties

– usually available either before IP or with IP introduction

11

Problems with Interpretive ISS

Very slow - at least one order of magnitude slower than Compiled Code ISS and 2 orders (100X) slower than VPM approach

– Simulates instruction Fetch, Decode and Dispatch dynamically at runtime.

– Models internal processor resources and stateCan be used to validate other approaches for accuracy and

understand ‘field of use’– But unlikely to be useful for significant dynamic execution of

embedded SW

12

Mathematical Kernel Functions (e.g. Dataflow)

SPW/Cossap/Ptolemy Modeling

AFIR

CTONE

B*

DFFT

1 16

1

1

1 1 1

AT&T DSP16xx

Intel Pentium MMX

Motorola 56kFIR(t) = 14 + 4(t - 1)TONE() = 52MULT() = 4FFT(p) = 122 + 32pVECTSUM(n) = 5 + 2 * n

Schedule (16 A C B) D

Cost16 • FIR(128) + 16 • TONE + 16 • MULT + FFT(256)

13

Problems with Mathematical KernelsNot at all suitable for data-dependent Control codeBest for fixed processing of fixed sample sizes

– Has been used for characterising DSP’s – e.g. BDTI - Berkeley Design Technology– FFT on a Motorola 56k of p points has a latency of 122 + 32p cycles– FIR filter on a Motorola 56k has a latency of 14 + 4(t - 1) cycles.

Effective modelling depends on large families of pre-characterised kernels

– finding right balance of accuracy and granularity a problemNeeds in-depth application-specific knowledge to applyCannot be easily automated

14

Manual Back-annotation approachMeasure generated code on target processor with target compiler

– Compilation optimisations handled– Target dependent HW handled– Can use accurate Cycle-counting Instruction Set Simulator or real

HW– could also use emulation systems

Back-annotate source code for use in co-design estimation– Delay models applicable to whole task (black box approach)– Annotate basic blocks of task with measured numbers

Possible to combine measured and kernel function approaches

15

Problems with Manual Back-annotation

Extensive characterisation effort requiredWorks best on computation-intensive code

– Data-dependent control code hard to characterise or tedious to annotate every block

Requires either real HW (very late in process) or accurate ISS– accurate ISS may depend on detailed RTL-level design– not suitable for early speculation on processor features

Manual!– Not easy to automate

16

Can we improve VPM?The Data Book Approach to VPM estimators is very conservative

– e.g. for processor XXX, LD = 3 cycles but compilers can optimise down to something a little over 1 on average

– SUB and RET with 15 user registers:– conservative estimate of 19 and 21 cycles (4+Active Reg, 6+AR)– actual usage is more like 5-10 active registers (statistically)

The Data book approach, or any kind of artificial ‘Calibration Suite’, is applied to all kinds of embedded SW

– But embedded SW is not all the same– control vs. computation– application specific features e.g. wireless, wired comms, automotive, multimedia

image processing

17

Abandon Causality, Embrace CorrelationsVPM approach based on Databook cycle counts assumes a causal relationship between

the presence of a Virtual instruction in the ‘fuzzy’ compililation, and some (fixed) number of cycles in the task cycle count, if the Virtual instruction lies in an executed basic block.

– Conservative assumptions on cycle counts– No application domain specific characteristics– Cannot account for target compiler optimisations

A statistical approach abandons the idea of causal relationships– Rather, it seeks to find correlations between characteristics of the source code and the

overall cycle count, and use them to build better predictor equations

18

Source Code characteristicsNeed dynamic characteristics, not static onesObvious source: Virtual instructions from the VPM model!But rather than assume that each instruction ‘attracts’ a fixed number

of cycles in the task cycle count, seek correlations between virtual instruction frequencies and overall cycle count.

Statistical technique used:– Stepwise multiple linear regression– Virtual instruction counts are not really all independent variables; use

correlations between them to reduce the number of variables used in the predictor equation

Note that this can subsume compiler effects to some extent

19

Statistical Benchmarking approach Identify a benchmark set of tasks: sample set

– Application domain specific Run the tasks through the VPM instruction frequency counting Run the tasks on a cycle-accurate ISS or HW Reduce the VPM instructions to a set of independent variables

– using correlation analysis Run a stepwise and normal multiple linear regression on the sample Produce a predictor equation with quality metrics Apply it to new code

– drawn from the same application domain (should do reasonably)– Other domains (may do poorly)

Determine a discriminating function that can be used to test code to see if it is from the same domain as the sample set

20

Stepwise Multiple Linear RegressionRegression: Y = B + i Ci * Xi

– where Y = Dependent variable (to be predicted)– B = intercept– i I where I = the set of orthogonal (independent) Virtual Instructions

(independent variables)– Ci = the coefficient for

– Xi = the cycle count for Virtual Instruction I

Stepwise - Independent variables are added one at a time– Must pass a test of statistical significance– When no more variables can be added, regression stops

Sample size should be larger than the number of independent variables, else regression is over-determined.

21

Regression

Simple linear regression: Y = B + mXMinimise Sum of Squared Errors = (Ysamplei-Ypredictedi)2 Measure ‘quality’ of Predictor Equation via R2 = the square of the correlation coefficientR2 is defined as “the proportion of the variation of Y that is explained (accounted for) by the variation in the X’s”.

= one sampleY

X

B

22

Note on R2

R2 does not imply causalityRather, it can be used to gauge the ‘quality’ of the resulting predictor

equationR2 of 1.00 means a ‘perfect’ predictor equation is possible

– Want R2 as high as possible, but respectable results can be obtained with low R2

Correlation between a set of independent variables and a dependent variable is a study in variability, not causality

Our hypothesis is that the resulting predictor can be used on new samples (code) drawn from the same population (application domain)

23

First Study: Set of 35 ‘Automotive’ benchmarksControl dominated; not much computationDivided into Sample Set of 18, Control set of 17 randomlyPartial excerpt of virtual instruction counts:

LD LI ST OPi MULi DIVi IF GOTO SUB20 2 2 64 2 1 25 0 120 2 2 64 2 1 25 0 120 5 2 56 2 1 21 0 320 4 4 61 2 1 25 1 120 2 2 64 2 1 26 2 120 5 2 57 2 1 22 1 320 4 4 62 2 1 25 1 120 2 2 59 2 1 23 1 120 2 2 59 2 1 23 1 120 2 2 59 2 1 23 1 120 2 2 59 2 1 23 1 120 2 2 59 2 1 23 1 122 4 3 64 2 1 26 2 220 4 4 62 2 1 25 2 120 2 2 63 2 1 25 1 1

24

First step: correlation of independent variables

Pearson Correlations Section (Row-Wise Deletion)

LD LI ST OPi MULi DIViLD 1.000000 0.013095 -0.679006 0.937686 0.000000 0.000000LI 0.013095 1.000000 0.417704 -0.016410 0.000000 0.000000ST -0.679006 0.417704 1.000000 -0.534283 0.000000 0.000000OPi 0.937686 -0.016410 -0.534283 1.000000 0.000000 0.000000MULi 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000DIVi 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000IF 0.900243 0.004637 -0.439935 0.991981 0.000000 0.000000GOTO 0.409408 0.074820 -0.021693 0.392022 0.000000 0.000000SUB -0.212899 0.688895 0.048507 -0.362069 0.000000 0.000000RET -0.212899 0.688895 0.048507 -0.362069 0.000000 0.000000Cycles 0.707427 -0.116149 -0.502512 0.688486 0.000000 0.000000Cronbachs Alpha = 0.599374 Standardized Cronbachs Alpha = 0.383204

Pearson Correlations Section (Row-Wise Deletion)

IF GOTO SUB RET CyclesLD 0.900243 0.409408 -0.212899 -0.212899 0.707427LI 0.004637 0.074820 0.688895 0.688895 -0.116149ST -0.439935 -0.021693 0.048507 0.048507 -0.502512OPi 0.991981 0.392022 -0.362069 -0.362069 0.688486MULi 0.000000 0.000000 0.000000 0.000000 0.000000DIVi 0.000000 0.000000 0.000000 0.000000 0.000000IF 1.000000 0.444580 -0.400471 -0.400471 0.662023GOTO 0.444580 1.000000 -0.223607 -0.223607 0.155163SUB -0.400471 -0.223607 1.000000 1.000000 -0.312829RET -0.400471 -0.223607 1.000000 1.000000 -0.312829Cycles 0.662023 0.155163 -0.312829 -0.312829 1.000000Cronbachs Alpha = 0.599374 Standardized Cronbachs Alpha = 0.383204

25

Independent variable selection

In this sample set, Muli and Divi were constant over the samplesNo variability implies nothing to contribute to Cycle variability, so

dropSUB and RET were perfectly correlated (no surprise!)LD, OPi and IF highly correlated (over 90%)

– Only need one of them.Therefore, regression run with:

– LD, LI, ST, GOTO and SUB– 5 independent variables, with sample size of 18

26

Regression Results and InterpretationR2 = 0.57Cycles = 219 + 1.3*LD + 10.9*LI - 10.2*ST - 5.2*GOTO - 21.3*SUBNote: This is a pure “Predictor Equation”

– Intercept does not have to make ‘sense’– Coefficients have NO operative meaning– Positive and negative correlations can’t be ‘interpreted’– Task cycle counts ranged from 151 to 246. Many of them less than the intercept of

219!When predictor applied to Control group of 17, error -16% to +3%When predictor back-applied to sample set, error -8% to +13%Data sheet model applied to all 35, error -10% to +45% (conserv.)

27

Stepwise resultsRunning it only adds LD to equation with an R2 of .500 (in other words, LD

variability explains far more of the cycle count variability than any of the other variables.

Predictor equation:– Cycles = 146 + 3.6*LD

Note - when applied to the control set of 17 we get error of -16% to +8% (24% range)

– when back applied to sample set of 18, we get error of -8% to +14% (22% range)Equation is very robust. Since LD’s are present in all programs, unusual

instructions (e.g. Muld’s, etc.) do not skew the results. However, would do very poorly on mathematical code

28

Residual Errors

0.0

1.3

2.5

3.8

5.0

-30.0 -15.0 0.0 15.0 30.0

Histogram of Residuals of Cycles

Residuals of Cycles

Count

29

Residual Errors

-30.0

-15.0

0.0

15.0

30.0

-2.0 -1.0 0.0 1.0 2.0

Normal Probability Plot of Residuals of Cycles

Expected Normals

Resi

duals

of

Cyc

les

30

Larger Sample Set

We took the 35 ‘automotive’ control samples, added 6 ‘Esterel’ examples (believed control dominated), and 4 other ‘control’ oriented pieces of code, for a sample of 45 benchmarks.

Multiple regression was applied, and then we manipulated the set of independent variables (through ‘Art’) to get the predictor:

Cycles = 75 + (Opi + Opc) + 3.4*IF + 20*SUBOn back-substitution, Error range was -30% to +20%Some doubt that sample set was really drawn from similar

enough domains to be meaningful, BUT…..

31

An operative interpretation of the equation

If Cycles = 75 + (Opi + Opc) + 3.4*IF + 20*SUBThen one might interpret (note- this is statistically invalid) that

each task takes – 75 cycles to start and finish (prologue and epilogue)– each basic integer (Opi) and character (Opc) operator is 1 cycle– each branch (IF) ‘attracts’ 3.4 cycles on average– each Subroutine call and return takes 20 cycles

– this last implies an average of 5 user registers in use

Although invalid, this predictor ‘makes some kind of sense’ (æsthetically and psychologically)

32

Discriminating Code Application Domains

This study only just startedBasic idea:

– Find a discriminating test to determine if a predictor equation derived from one sample set can be applied to another, new set of code samples

– I.e. measure the legitimacy of using a predictor on a new setOne suggestion is to look at ‘Control’ vs. ‘Computation’

dominated code Idea is to classify virtual instructions as ‘control’ or ‘computation’

and compare their ratios

33

Ratio of Control to Computation Virtual Instructions

nDIVnMULnOPGOTOIF

nOPGOTOIFRationComputatiotoControl

Equation

...

.___

1_

where

OP.n represents OP.c, OP.s, OP.i, OP.l, Op.d, Op.f

MUL.n represents MUL.c, MUL.s, MUL.i, MUL.l,MUL.d, MUL.f

DIV.n represents DIV.c, DIV.s, DIV.i, DIV.l, DIV.d, DIV.f

If the Control to Computation Ratio is greater than 2/3,

then the VPM should provide reasonable accuracy estimates.

34

A simpler approachThis approach seemed too variable when applied to our benchmark

setsWe looked at a simpler ratio

– number of Virtual IF’s to total Cycle count– on the 35 ‘automotive control’ examples, this ratio ranged from 9 to 13%– on the 6 ‘Esterel’ examples, this ratio ranged from 1 to 5%– suggests that perhaps these 2 sample sets are NOT drawn from the

same population, and therefore a predictor from one set will NOT work on the second

– True in practice: using the a predictor from the first set, on the second gave error ranges of 23 to 60% !

35

Is there a statistical test that can help?

2-sample t-test (and Aspin-Welch unequal-variance test)Used to test the hypothesis that 2 sample sets are drawn from

the same underlying population– assumes that they are normally distributed– t-test assumes that they have equal variances and tests if the

means are equal– Aspin-Welch assumes that they have unequal variances and tests

if the means are equal

36

We applied these tests to our 18 and 17 sample ‘automotive’ SW setsDiscriminator was the ratio of IF’s to total cycle countTests passed - I.e. it is possible that the 2 samples were drawn

from the same underlying populationNormality of sample 1 under question

37

Results of analysis

0.0

2.0

4.0

6.0

8.0

0.0 0.1 0.1 0.1 0.1

Histogram of C1

C1

Count

0.0

2.0

4.0

6.0

8.0

0.1 0.1 0.1 0.1 0.1

Histogram of C2

C2

Count

38

Results of analysis

0.0

0.1

0.1

0.1

0.1

-2.0 -1.0 0.0 1.0 2.0

Normal Probability Plot of C1

Expected Normals

C1

0.1

0.1

0.1

0.1

0.1

-2.0 -1.0 0.0 1.0 2.0

Normal Probability Plot of C2

Expected Normals

C2

39

Results of analysis

0.02

0.05

0.08

0.11

0.14

C1 C2

Box Plot

Variables

Am

ount

40

T-test applied to ‘automotive’ vs. ‘Esterel’

0.00

0.04

0.07

0.11

0.14

C1 C2

Box Plot

Variables

Am

ount

Hypotheses soundly rejectedNormality of 6-sample Esterel set very doubtful!

(Sample size too small)

0.0

5.0

10.0

15.0

20.0

0.0 0.1 0.1 0.1 0.1

Histogram of C1

C1

Count

0.0

1.0

2.0

3.0

4.0

0.0 0.0 0.0 0.0 0.1

Histogram of C2

C2C

ount

41

Conclusions

The statistical analysis method seems to provide:– improved and more robust estimators than data sheets

– better centred and less likely to produce wild outliers– some ability to discriminate among domain sets of SW tasks

Hampered by lack of benchmark examples– very hard to obtain especially for multiple domains– since statistical, need lots of samples drawn from a wide variety of implementations

Needs to be rerun on other processors (just one used so far)– Variant HW features may mean applicable only to very simple embedded RISCs

Users find it very hard to interpret the predictor equations– they expect the coefficients to be all positive and make ‘sense’

42

Future improvements under study Improved Compiler front-end

– Target-Independent Optimisations incorporated– May yield different Virtual instruction counts and thus better predictors of cycle

countOther processors

– Need cycle-accurate ISS’s from vendors– looking at DSP’s to study limits of technique

Better benchmark sets– need non-proprietary benchmarks which work on a variety of embedded

processors– EEMBC Benchmark Suite (www.eembc.org) - EDN Embedded Microprocessor

Benchmark Consortium

43

EEMBC benchmarks

Cadence joined EEMBC as a 3rd party tool provider to get access to benchmark code

Automotive/Industrial– e.g. table lookup, angle to time conversion, pulse width modulation,

IIR filters, CAN remote data request, ….Consumer - e.g. JPEG compression and decompressionNetworking - e.g. path routingOffice Automation - e.g. printer controlTelecommunications

– e.g.autocorrelation, Viterbi, convolutional encoding, ...