performance testing based on time complexity analysis for embedded software

Performance Testing Based on Time Complexity Analysis for Embedded Software

Hu Jin1, Liang-Yin Chen2, Ling-Ming Zeng1, Bao-Lin Li2 1Computer Science Department, Chengdu University of Information Technology,

Chengdu610225, China 2College of Computer Science, Sichuan University, Chengdu 610065, Sichuan, China

[email protected] [email protected]

Abstract

Architecture design and software implementation both contribute to the correctness of ultimate software products. So performance testing is very helpful to test the inconsistence between design and implementation since it is the mix reflection of both sides. Especially in embedded system, with limited resources, execution time is more likely to expose hidden defects. This research presents a time performance analysis method for software testing. Firstly, software modules were divided according to functionality in architecture view, and their time complexity were computed in static; Secondly, testing activities were designed to track running time cost of those modules; Next, expected time complexity was compared with actual running time to figure out abnormal function modules defects resident. Lastly, experiments were conducted in an embedded software project. The results showed time performance testing is an efficient way to find out some kinds of defects concerned with the inconsistence of design and implementation. Keywords: Software Testing, Embedded System, Time Complexity, Architecture Design 1. Introduction

Software testing is one of the most important ways to assure the software products quality. Conventional testing methods take more care about the usability and functional correctness but often ignore those benign defects which may not lead to devastation disaster. These trivial defects may not lead to much trouble when running environment is favorable. But with much more tough circumstance, such as in embedded system environment with much more limited resources, those benign defects will contribute much to the performance

de-efficiency of software and even make system function failure.

Performance testing [1] is an efficient method for detecting benign faults. It is the incarnation of both design and implementation. Software architecture design describes software systems at a high level of abstraction and provides a basis for course-grain static analysis [2]. However, at implementation level, there exist risks such as misunderstanding or error coding. Then, performance is not only affected by the internal design attributes but also by the implementation detail. Performance testing can help to find out inefficient software modules cost much fulfill time which could be improved with more design consideration. Also, it can help to detect the inconsistence of design and implementation by figuring out abnormal time consuming modules. This research shows performance testing is especially helpful to test those problems not fatal to the software.

This issue is organized as follows. Section 2 describes how to divide software modules for static time complexity analysis at architecture view. Section 3 then uses dynamic testing to analyze the reasonable time cost of software execution time on the basis of the previous time complexity model. Section 4 does experiments in WinCE to verify the effectiveness of performance testing method. Last section summarizes the work of this paper. 2. Static time complexity analysis

Software architecture describes system at high levels of abstraction. As the size and complexity of software systems increase, the design and specification of overall system structure become a significant issue [3]. People have long advocated a component-based approach to software construction as a way of simplifying the design and maintenance of large software system [4]. All components are rather

The 2008 International Conference on Embedded Software and Systems (ICESS2008)

978-0-7695-3287-5/08 $25.00 © 2008 IEEE

DOI 10.1109/ICESS.2008.90

243

Authorized licensed use limited to: King Mongkuts Institute of Technology Ladkrabang. Downloaded on July 17,2010 at 08:23:31 UTC from IEEE Xplore. Restrictions apply.

independent modules to fulfill specific functions, and integrated to make the software work as a whole. Components are composed of limited number of instructions, and the components can be expressed as C = <is, I, ie>. I is the set of instructions, and is, ie is the corresponding entry and exit instruction of component [5]. Between is and ie are the instructions sequenced by the control flow. There is only one pair of is and ie for each component, which guarantee the component to be closed. No intersection instructions can exist in two or more different components, Ci∩Cj = Φ, i≠j. The software is P =∪Ci with all components are referenced. Connections amongst different components are deeply influenced by the architecture being applied. For a most often used pipes and filters architecture style, the main interaction factors include input variables to the component and global variables of the program. Next figure 1 shows this type of component model.

With the software components deployment in

design, time complexity of the component could be computed out respectively. The expected total time cost of the component is the sum of desired execution time of all instructions in it, its time complexity can be represented as: TM = ∑t(im)×e(Input,

min ) [6]. im is

the abstract instruction type for all programming languages, and t(im) is the execution time of that instruction for specific implementation language. For a

component, mi

n is for how many times instruction im

should be executed. It is mainly an internal attribute of the program design. The total execution time of component will be influenced by the input variables with ),(

minInpute , if the global variables are not

considered. TM is thus the time complexity of the corresponding component. The relationship between TM and input variables is as a function: TM: InputVar time.

The program now can be represented using such component model. All the components are associated with actual control flow path with structural analysis [7]. A control flow path could be thought of as the composition of components arranged in sequence and be represented as vectors: CFPi = ),...,,( 10

in

ii MMM , abbreviated as Pi. So, for all control flow paths in program, they are vectors set form: ),...,,( 21

Tk

TT PPP . These control flow paths can be figured out automatically by many typical technologies such as: traversing abstract syntax tree [8] or adjacent matrix. Now, the control flow paths can be expressed as the product of coefficient matrix and component matrix:

],...,,[ 21T

mTT PPP = Am×n × Diag(M0,M1,…,Mn)

Element aij of matrix A is the execution times of the jth component in the ith control flow path.

The component deployment is by testers’ decision

according to the granularity of design view. Just like figure 2, C2 is subdivided into small components.

Ie

C1

Input Var. Global Var.

Ie

Is

Pipes

Filter

Fig. 1 Pipes and filters architecture styles

C3


Ie

Is

C2


Ie

Is

C4


Ie

Is

C4

Input Var.Global Var.

Ie

Is

Fig. 2 Different granularity for component

G

I JH

K

L

F C2A

B

C D

E

C1

H

L

F C3 C4

G

I J

K

L

F

+

244


On the basis of previous analysis, the expected theoretical execution time of designed components along some specified control flow path could be computed out. By reasons of implementation or coding, the actual program execution time may be far away from the theoretical values, which may indicate some benign errors. Next, research was done to find out those defects. 3. Dynamic Performance Testing

Program executes with specific input data and attains corresponding results. The Input-Output pair with associate execution conditions is called as test cases [9]. Test cases are used to verify program compliance with specific test requirements, and they reflect the dynamic characters of program [10]. According to executed control flow paths, test data space are subdivided into different subsets. For each of subset, the elements in itself are all equivalent in the sense of executing the same control flow path [11]. All the test cases in one subset means performing a testing task, driven by which, sequential software components are executed in order. The testing task is expressed as: TAi = [pi1, pi2, …, pin | vi1, vi2,…, vik]. In this presentation, two parts are included. One is the executed control flow path. The other is the associate test cases or input variable set. The execution control flow path can be substituted with the multiplication of coefficient matrix and component matrix. That is very convenient for performance testing analysis: t(TAi) =

],...,,|)(),...,(),([ 212

21

1 ikiiniiniiii vvvMtaMtaMta ××× .

Now, the actual execution time should be in accordance with the components’ time complexity computed out in static. Therefore, the testing tasks driven by equivalent test cases set have the same coefficients. Time costs on the same testing task could only depend on the components’ time complexity with different input variables in equivalent test case set. It is then a very useful conclusion for time performance testing: k

sj

sskis

ji TMTMMtMt :)(:)( = . i specifies the

ith testing task. j and k are for two different test cases in an equivalent test case set. s is for the components along with the testing task. If program is tested with an equivalent test case set, time consumed on the same component will only depend on the input variables assignment: e(InputVar,

min ). To some extent, the

equivalent test cases set will also guarantee the same external execution environment, thus making global variables in the component constant when execution. That is also the reason why it can not simply use component alone to decide its execution time.

From the previous description, the components’ execution time is only influenced by the input variables. That time can be looked upon as the dependant variable of input invariables: f:(V1,V2,…,Vn) T. It is possible to use this execution time function to accomplish testing work. If actual execution time can comply with their static time complexity, then their execution time could be thought as normal. Else, there probably defects existed. Usually it is impractical to analyze such time function with more than tree input variables. Also, not all but some of variables are attractive for testers. This paper just discusses one independent input variable case. Regression [12] method obviously is the most proper solution for this work.

Linear regression is the simplest form of regression. Bivariate linear regression models a random variable Y as linear function of another independent variable X, that is, Y = α+βX. α, β are regression coefficients, α specifies Y-intercept and β is the slope of the line. Take the execution time as Y and input variable as X. Then, the dynamic testing data in equivalent set can be collected for regression analysis. Coefficient α and β can be computed out with the next formula.

∑∑

=

=

−

−−= s

i i

s

i ii

xx

yyxx

12

1

)(

))((β

xy βα −=

where x , y is the average of testing data x1, x2, …, xs, and y1, y2, …, ys respectively. Once the regression line is determined, it would like to evaluate the effectiveness of the fitted line. A measure that helps to answer this question is provided by formula: SSE =

2

1)(∑

=

∧−

n

iii YY , where ii XY βα +=

∧. If SSE = 0, the

regression line fits perfectly. As the fit gets worse, SSE gets larger. This can be used to detect the abnormal points which have large deviations.

This linear regression relationship with only one independent variable is easy to be tested. Polynomial regression can also be applied for analyzing none linear dependence. By applying transformations to the variables, the none-linear regression model still could be converted into a linear one. Since not all of the input variables need to be cared about according to specific testing purpose, we can just isolate those input variables interested to delivery component time testing.

Here summarizes the testing procedure briefly. Program should firstly be executed with equivalent test cases as input. Then, those interested components running time are to be recorded for performance testing

245


analysis. The ration of the execution time and input variable is arranged for linear regression. Thus, those abnormal data is selected out for further investigation to find out inconsistency of design and implementation.

4. Experiments

The experiment platform is a embedded system.

Hardware is a touch screen with CPU ARM9 2440 400MHz and displaying acceleration chip SM501. The system memory size is 32M. WinCE 4.0 is the installed OS which is a pre-empty schedule system. A HMI project is to be tested. Its main function is to monitor attached control nodes information in real time. Data collected from distributed nodes are used for presenting simulation in center touch screen. Every monitored scene can be switched on from one to another when needed.

The primary displaying module of the system can be divided into three parts. (1) Initialization module to do some common works such as memory preparing, protocol initialization and drawing context setting. (2) Data collection from all net nodes being monitored and decoding according to protocol. (3) Animation and displaying corresponding scene of node information when needed. These parts still can be further divided into smaller modules. Main trouble of the system is that it is too slow to response when switching from one scene to another. How to evaluate the impaction to the inefficiency of the system is rather a more difficult problem than what it looks like from the apparent. There are two obstacles must be tackled for the traditional software testing method. One is that pre-empty strategy makes the lowest priority displaying module more difficult to trace. Another one is that users’ interference induces more uncertainty to the system response time.

There are six scene sighs to be monitored in this project. Different execution time was spent to animate those scenes that contribute much for switching time. Next figures show the irrational execution time of the animation module indicating further investigation is needed.

In figure 3, left vertical coordination is to specify switching time needed for displaying corresponding scene sight. Right vertical coordination is to show drawing object numbers in displaying scenery. Obviously, animation time of the oil transferring is abnormal for its switching time is not accord with its contained object count. It spent almost twice time cost as any other scenery. By applying linear regression, it can be figured out two coefficient are α = 400, and β = 1.298.

Switching time diagram

0

200

400

600

800

1000

1200

1400

1600

1800

2000

oil trans. dryer sys. dryer room heating sys.cleaner sys. purify sys.

ms

0

50

100

150

200

250

300

350

Switch Time

Drawing Object Number

After more careful investigation, an implementation

defect was found out. The drawing cut region is not properly set when animation. Then, more unnecessary redrawing time was spent to do this work. Because such a defect is context related, all except at the oil transferring scene sight can work well. Next figure is the results after defect revised.

Switching time diagram

0

100

200

300

400

500

600

700

800

900

1000

oil trans. dryer sys. dryer room heating sys. cleaner sys. purify sys.

ms

0

50

100

150

200

250

300

350

Switch time

Drawing Object Number

After removal of this bug in drawing component,

figure 4 is more reasonable. Switch time cost is in accord with the drawing object numbers. Certainly, switching time is not strictly conformance to drawing objects numbers because different objects may need different drawing time. But it is still a very good indicator for time cost. Now, all the switching time is confined in 1000ms, and the operator can get a rather more comfortable response time.

5. Conclusions

In summary, we have proposed a time performance testing model for embedded software testing. The proposed approach takes advantage of comparability of execution time driven by equivalent test cases. By comparing or analyzing the abnormal execution time

Fig. 3 Switch Time and object numbers of different animation windows

Fig.4 Switch Time after implementation revising

246


cost with regression method, it is helpful to find out the deficiency of design or defects introduced by implementation. Future works include finding out more effective method for subdividing the software components, and combining more independent input variables for directing time performance testing. Acknowledgement

This work was supported by the Sichuan province science and technology tackle key problems project under grant No 05GG0321, the National scientific and technological middle or small sized enterprise innovative fund under grant No 06C26225101730. References [1] Glenford J. Myers, The Art of Software Testing, John

Wiley & Sons, Inc., 1979. [2] Stafford J A, Richardson D J, Wolf A L, “Architecture-

level Dependence Analysis for Software Systems”, International Journal of Software Engineering and Knowledge Engineering, Vol. 11, No. 4, pp. 431-453, August 2001.

[3] Mary Shaw, David Garlan., Software Architecture: perspectives on an emerging discipline, Prentice Hall, Inc., 1996.

[4] Rosenblum D S, “Adequate Testing of Component-Based Software”, Department of Information and Computer Science University of California, Irvine, CA 92697-3425, August 1997.

[5] Ren-Jie Zheng, Computer Software Testing Technology, Tsinghua University Press, 1991.

[6] Xiao-Dong Wang, The Design and Analysis of Algorithms, Tsinghua University Press, 2003.

[7] Jorgensen P C, A Craftsman’s Approach, CRC Press, Inc., 2002.

[8] Kenneth C. Louden, Compiler Construction: Principles and Practice, Thomson Learning, 2002.

[9] [IEEE Std 610.12-1990] IEEE Standard Glossary of Software Engineering Terminology Standards, 1999.

[10] Christoph C.Michael, “Genetic Algorithms for Dynamic Test Data Generation”, Proceedings of the 12th international conference on Automated Software Engineering, pp.307-308, 1997.

[11] Hu Jin, Zhi-Shu Li, “Using Genetic Algorithm to Generating Test Cases Automatically and Path-Oriented”, Computer Engineering, Vol.33, No.3, pp.21-23, 2007.

[12] Jia-wei Han, Micheline Kamber, Data Mining: Concepts and Techniques, Higher Education Press, 2001.

247
