considering execution environment resilience: a white-box approach

Considering Execution Environment Resilience:A White-Box Approach

Stefan Klikovits1,2, David PY Lawrence1,3,Manuel Gonzalez-Berges2, Didier Buchs1

1Université de Genève, Carouge, Switzerland2CERN, Geneva, Switzerland

3Honeywell International Sarl., Rolle, Switzerland

Tuesday 8th September, 2015

ConsideringExecution

EnvironmentResilience:

A White-BoxApproach

S.Klikovits,D. Lawrence,M. Gonzalez-

Berges,D. Buchs

What is this all about?

How toI generate test cases with little user interactionI on a large scaleI unit test level

2 / 21



A White-BoxApproach


Berges,D. Buchs

Welcome to CERN

3 / 21



A White-BoxApproach


Berges,D. Buchs

Welcome to CERN

I LHC, experiments, infrastructure (e.g. power grid)I large-scale, widespread, complex systemsI many types of hard- and softwareI > 100 subsystems,

10,000s of devices,1,000,000s of parameters

I thousands of physicists/engineers/workershigh employee turnover

high reliability and resilience expectations

3 / 21



A White-BoxApproach


Berges,D. Buchs

How do we supervise it?

I framework built on top for HEP domainI proprietary scripting language: Control (CTRL)

4 / 21



A White-BoxApproach


Berges,D. Buchs

So where is the problem?

I until recently no automated unit test executionI frequent changes in execution environmentI (mostly) manual verificationI big expenses (time) on QA side

⇒ increase test coverage

5 / 21



A White-BoxApproach


Berges,D. Buchs

Testing

1 f ( x ){2 i f GLOBAL_VAR:3 re tu rn dbGet ( x )4 e l s e :5 re tu rn −16 }

f(x)

1 t e s t_ f ( ){2 dbSet ( " t e s t " ,5 ) // p r epa r e3 GLOBAL_VAR = True4 x = f ( " t e s t " ) // ac t5 a s s e r t ( x == 5) // a s s e r t6 }

Test case for f(x)

6 / 21



A White-BoxApproach


Berges,D. Buchs

What do we want?

code test cases

Iterative Test Case System (ITEC)I regression testingI consider dependenciesI automatic test case generation (ATCG)I build on existing research & toolsI generate unit & component tests

7 / 21



A White-BoxApproach


Berges,D. Buchs

Automated Test Case Generation

I source code basedI black-box (function signature) vs.

white-box (function body)

8 / 21



A White-BoxApproach


Berges,D. Buchs

Semi-purification

I replace dependencies with parameters

1 f ( x ){2 i f GLOBAL_VAR :3 re tu rn dbGet(x)4 e l s e :5 re tu rn −16 }

A non-pure function

1 f_sp ( x , a ,b){2 i f a :3 re tu rn b4 e l s e :5 re tu rn −16 }

Semi-purified f(x)

aaaa

9 / 21



A White-BoxApproach


Berges,D. Buchs

Semi-purification


1 f ( x ){2 i f GLOBAL_VAR :3 re tu rn dbGet(x)4 e l s e :5 re tu rn −16 }

A non-pure function

1 f_sp ( x , a ,b){2 i f a :3 re tu rn b4 e l s e :5 re tu rn −16 }

Semi-purified f(x)

1 test_f_sp ( ){2 x = f ( " t e s t " , True , 5 ) // ac t3 a s s e r t ( x == 5) // a s s e r t4 }

Test case

9 / 21



A White-BoxApproach


Berges,D. Buchs

Semi-purification (cont.)


1 func t i onA ( x ){2 a = func t i onB ( x )3 re tu rn a4 }56 func t i onB ( x ){7 b = GLOBAL_VAR8 b++9 re tu rn b

10 }

Function with SRC

1 funct ionA_sp ( x , y){2 a = func t i onB ( x , y)3 re tu rn a4 }56 funct ionB_sp ( x , y){7 b = y8 b++9 re tu rn b

10 }

Semi-purified w. SRC

10 / 21



A White-BoxApproach


Berges,D. Buchs

Semi-purification: Concept

I code contains dependenciesI global variables, data base values, subroutine calls,

other resources

I manual way: test doubles (mocks, stubs, fakes, . . . )[ME06]

I remove dependenciesI based on localization [SW03, SK13]I input parameters instead of dependenciesI use any ATCG (black- and white-box)

11 / 21



A White-BoxApproach


Berges,D. Buchs

“Shortcut”

f (P){D} f (P ∪ PD){}

〈TI ,TS〉〈TI ∪ TID〉

SP

SP−1

AT

CG

Figure: Test case generation schema

I P . . . ParametersI D . . . Dependencies

I TI . . . Test InputI TS . . . Test Setup Routine

12 / 21



A White-BoxApproach


Berges,D. Buchs

Identified Technical Challenges

I Shared SubroutinesI LoopsI Concurrency

13 / 21



A White-BoxApproach


Berges,D. Buchs

SP: Technical Challenges (cont.)

I Shared subroutine dependencies

1 var SPEED_VAR = 12 ad ju s tSpeed ( ){3 x = getTheSpeed ( )4 i f x < 10 :5 doubleTheSpeed ( )6 }

CUT

1 getTheSpeed ( ){2 re tu rn SPEED_VAR3 }

Subroutine 1

1 doubleTheSpeed ( ){2 speed = SPEED_VAR3 SPEED_VAR = speed ∗24 }

Subroutine 2

I BUT: a and b replace the same dependency

14 / 21



A White-BoxApproach


Berges,D. Buchs



12 ad ju s tSpeed (a){3 x = getTheSpeed (a)4 i f x < 10 :5 doubleTheSpeed ( )6 }

CUT

1 getTheSpeed (a){2 re tu rn a // SPEED_VAR3 }

Subroutine 1

1 doubleTheSpeed ( ){2 speed = SPEED_VAR3 SPEED_VAR = speed ∗24 }

Subroutine 2


14 / 21



A White-BoxApproach


Berges,D. Buchs



12 ad ju s tSpeed (a, b){3 x = getTheSpeed (a)4 i f x < 10 :5 doubleTheSpeed (b)6 }

CUT

1 getTheSpeed (a){2 re tu rn a // SPEED_VAR3 }

Subroutine 1

1 doubleTheSpeed (b){2 speed = b // SPEED_VAR3 b = speed ∗24 }

Subroutine 2


14 / 21



A White-BoxApproach


Berges,D. Buchs

SP: Technical Challenges (cont.)I Loops

1 s l e e pUn t i l R e a d y ( ){23 whi le dbGet(notReadyDP) :4 s l e e p (5 ) // s l e e p f o r 5 seconds56 }

A semi-purified loop

Test Cases:I a: [False]⇒ loop not executedI a: [True, True, . . . , False] ⇒ loop execution

Open questions:I how long should the list be?I how to modify correctly?I test modified code or w. threads?

15 / 21



A White-BoxApproach


Berges,D. Buchs


1 s l e e pUn t i l R e a d y (a){ // a =bool23 whi le a : // replaces dbGet(notReadyDP)4 s l e e p (5 ) // s l e e p f o r 5 seconds56 }


Test Cases:I a: False ⇒ loop not executedI a: True ⇒ endless loop

I a: [False]⇒ loop not executedI a: [True, True, . . . , False] ⇒ loop execution


15 / 21



A White-BoxApproach


Berges,D. Buchs


1 s l e e pUn t i l R e a d y (a){ [bool]2 i = 03 whi le a[i] : // replaces dbGet(notReadyDP)4 s l e e p (5 ) // s l e e p f o r 5 seconds5 i++6 }


Test Cases:I a: [False]⇒ loop not executedI a: [True, True, . . . , False] ⇒ loop execution


15 / 21



A White-BoxApproach


Berges,D. Buchs

SP: Technical Challenges (cont.)I Concurrency

managerCtrl UI

manager

DeviceDrivers

Distmanager

DBmanager

EVmanager

Figure: WinCC OA’s manager concept

16 / 21



A White-BoxApproach


Berges,D. Buchs


I access to ALL data pointsI 100+ sub-systemsI discover dirty read/write scenarios (adjustSpeed())

16 / 21



A White-BoxApproach


Berges,D. Buchs

ITEC implementation

IDE SP engine

CTRLtest gen.

testdriver

code

SPdata

test cases

resu

lts

ITEC

ATCG

SP code

testinput

s

ATCG

source codetranslator

test inputtranslator

SP CTRL

CTRLtest inputs

SP tool code

testinpu

ts

TI generator

Figure: ITEC workflow

17 / 21



A White-BoxApproach


Berges,D. Buchs

Conclusion

Semi-purificationI remove dependenciesI facilitate unit tests generationI use black-box techniques on all code

18 / 21



A White-BoxApproach


Berges,D. Buchs

Future works

What’s next?I connect to test framework & roll-outI intrinsic domain informationI compare different ATCGI research into technical challenges

19 / 21



A White-BoxApproach


Berges,D. Buchs

References

[ME06] Meszaros, G.:XUnit Test Patterns: Refactoring Test Code, Chapter 23.Test Double Patterns, pages 521–590.Prentice Hall PTR, Upper Saddle River, NJ, USA, (2006)

[SC03] Sward, R. E., Chamillard, A. T.:Re-engineering Global Variables in Ada.In: Proc. 2004 ACM SIGAda international conference onAda, pp. 29–34, ACM, New York, (2003).

[SK13] Sankaranarayanan, H., Kulkarni, P.:Source-to-Source Refactoring and Elimination of GlobalVariables in C Programs.In: Journal of Software Engineering and Applications,Vol. 6 No. 5, pp. 264–273, (2013).

20 / 21

Considering Execution Environment Resilience:A White-Box Approach

Stefan Klikovits1,2, David PY Lawrence1,3,Manuel Gonzalez-Berges2, Didier Buchs1

1Université de Genève, Carouge, Switzerland2CERN, Geneva, Switzerland

3Honeywell International Sarl., Rolle, Switzerland

Tuesday 8th September, 2015