*An Integrated Self-Testing Framework for Autonomic Computing Systems

Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, Peter J. ClarkeSchool of Computing and Information Sciences

Florida International University

07/14/2007

FIU-SCIS Departmental Colloquium

* To appear in Issue 8 2007 of the Journal of Computers * Supported in part by the National Science Foundation under grants IIS-0552555 and HRD-0317692

2

Introduction o Continual growth in size and complexity of

computing systems has led to a need for support tasks to be shifted from people to technology

o Movement towards self-managing systems that can dynamically configure, heal, protect, and optimize themselves – Autonomic Computing.

o There is a lack of techniques to dynamically validate such systems.

o How can we be sure that AC systems behave correctly after a change at runtime?

3

Introduction (cont’d)o This work presents a methodology which

dynamically validates changes resulting from self-management in autonomic systems.

o Applies the concepts of autonomic managers, knowledge sources and manual management facilities to testing activities.

o Provides two general strategies based on system constraints and feasibility of use.

o Elaborates on a prototype that uses a generic design for autonomic systems that supports runtime testing.

4

Outlineo Backgroundo Overview of Testing Approacho Test Managerso Test Support Componentso Challengeso Prototypeo Related Worko Conclusion & Future Work

5

Autonomic Computing (1)o IBM’s solution to the problems of integrating and

managing highly complex computing systemsoHuman Body – ANS regulates vital bodily functions

without conscious human involvement (homeostasis)o Applied to Computing Systems – automation of low-

level tasks, and high-level goal specification.oMain Characteristics

Self-Configuration: configuring existing / new componentsSelf-Optimization: balancing workloads, tuning efficiencySelf-Protection: safeguarding from attacks or failuresSelf-Healing: diagnosing and repairing problems

6

Autonomic Computing (2)

Layered Architecture of Autonomic ComputingSource: An Architectural Blueprint for Autonomic Computing, Fourth Ed., IBM Corporation, June 2006.

o Management console to facilitate human activity

o Coordinates Touchpoint AMs (within or across)

o Manages resources directly through touchpoint layer

o Implements sensor and effector interface for MR

o H/W or S/W entities being managed

7

Autonomic Computing (3)o Monitor: collects state

information from MRo Analyze: determines if

and when self-* is neededo Plan: formulates a plan to

address change requesto Execute: executes the

change plan on the MRo Knowledge: stores data

shared by the MAPE functionsMAPE Structure of Autonomic ManagersSource: An Architectural Blueprint for Autonomic Computing, Fourth Ed., IBM Corporation, June 2006.

8

Software Testing - Categorieso Two broad types of testing:

o Blackbox – specification-based, focuses on functionality, i.e., inputs → expected outputs.

o Whitebox – implementation-based, focuses on whether or not the program has been thoroughly (adequately) exercised.

o Regression Testing – determines whether or not modifications to software have introduced new errors into previously tested code.o Retest-all – retest the entire test suiteo Selective – only retest a strict subset

9

Software Testing - Automationo Automating the testing process involves:

1) Designing test cases2) Developing test scripts3) Setting up a test harness for automatically:

Setting up the test environment Executing test cases Logging the test results Evaluating the test log

o What happens when the post-test evaluation passes or fails?

o Whitebox – implementation-based, focuses on whether or not the program has been thoroughly (adequately) exercised.

o Regression Testing – determines whether or not modifications to software have introduced new errors into previously tested code.o Retest-all – retest the entire test suiteo Selective – only retest a strict subset

10

Safe AdaptationZhang et al. (WADS 2004)

Source: J. Zhang, B. H. C. Cheng, Z. Yang, and P. K. McKinley. Enabling safe dynamic component-based software adaptation. In WADS, pages 194–211, 2004.

11

Testing Approach – Overviewo Idea: Develop an implicit self-test characteristic

for autonomic computing systemso Integrate activities of a self-testing framework

into the workflow autonomic managers (AMs)o Test Managers interface with the autonomic

system and coordinate the testing activityo Framework consistent with grand vision of ACo Based on two strategies:

Safe Adaptation with Validation Replication with Validation

12

Testing Approach - Architecture

Integrated Self-Testing Framework for AC Systems Extends the dynamic test model by King et. al, “Towards Self-Testing in Autonomic Computing Systems”Based on two general approaches: Safe Adaptation with Validation, and Replication with Validation

13

Test Managers (TMs)o Extend the concept of autonomic managers to

testing activities, i.e., Self-Testing via MAPE o Responsible for:

Test CoordinationTest Planning and ExecutionTest Suite ManagementPre- and Post-test SetupPost-test EvaluationStorage of Test Artifacts

14

High-Level Test Coordination1a,1b) Monitors OAM & TKS for state changes

2) Uploads new validation policy to Touchpoint TM

3) Touchpoint TM needs support tools to be setup

4) Invokes effector of ATS to configure support tool

5) Support tool configured successfully for TTM

6) TTM notified that it can begin test execution

Orchestrating TM Interactions Manages multiple components in the self-testing framework for high-level test coordination

15

Low-Level Testing Tasks

Touchpoint TM Interactions Performs testing on managed resources of the autonomic system via two control loops (a) and (b)

16

Test Support Componentso Auxiliary Test Services (ATS) allows TMs to

configure external or third-party testing tools, code profilers, performance analyzers, etc.

o Provides mechanisms for accessing information in the test knowledge sources. updating validation policies, test suites, test logs/histories

o Implements facilities for administrative functions related to manual test management, interactive test management, defect tracking, scenarios

o Administrative functions will be integrated with the console of the autonomic system

17

Challengeso V&V techniques for offline systems not scalable;

and still heavily dependent on human tester.o Drawbacks magnified for adaptive systems.o Testing adaptive nature requires dynamic:

Regression test case selectionTest suite analysis to:

– generate new test cases when necessary.– identify test cases that are no longer applicable.

o Testing autonomic systems in the presence of unforeseen conditions

18

Overcoming Challengeso Using proposed framework as the focal point for

research directions to support QA in AC systemso Some Directions:

Use of formal specifications to generate test sequences, test oracles, and test data

Executable formal specifications also support visualization systems (grand challenge of AC)

Mechanisms for policy-based risk analysis and trust can support testing in unforeseen circumstances

Risk-based strategies for regression test case selection

19

Prototype - Featureso Autonomic Container – Data structure with

autonomic capabilities and implicit self-testo Version 1 – Stack with self-configuration and self-

test. 80% full, reconfigure by increasing capacityo Version 2 – Implemented as a remote stack and

added self-protection. Users stack exceptions > 3, protect by disabling user

o Implemented Replication with Validation Validation required 100% pass rate for test cases,

75% for both branch and statement coverage Test suite: 24 test cases, using boundary,

random, and equivalence partitioning.

20

Prototype – Setup Environmento Developed in Java 5.0 using Eclipse 3.3 SDK

with required test support plugins and libraries.o Design Tools:

StarUML – class, activity, package diagramsOribeXML – creating and modifying XML policies

o Test Support Tools:JUnit – a Java unit testing tool from the xUnit family

of testing frameworksCobertura – a Java code coverage analysis tool

that calculates the percentage of source code exercised by unit tests.

21

Prototype – Top-Level Design

Top-Level Design of Autonomic Container

22

Prototype – Manager Design

Generic Design of Autonomic Managers

23

Prototype – Policy Design

24

Prototype – Self-Test Classes

Minimal Class Diagram of Self-Test Subsystem

25

Prototype – Evaluationo Used a mutation testing technique to evaluate

the fault detecting ability of the prototypeo Strategy: simulate faulty change requests to the

managed resource (stack) under SC and SP:SC – created mutant stack by modifying the

resize method to cause decreased capacitySP – created mutant stack by altering disable

account method to enable the usero Analyzed the results of executing the self-testing

framework on original stack and the mutants

26

Prototype – Resultso Correct change scenarios:

o Incorrect change scenarios:Two TC failures each, coverage omitted

o Mutation analysis produced favorable results as testing would have prevented potentially harmful changes to the managed resource

Feature TC Pass Rate Branch StatementSelf-Config 100% 85% 100%

Self-Protect 100% 88% 100%

27

Prototype – Lessons Learnedo Provided us with insight on the scope of the self-

testing subsystem w.r.t responsibilities.o Synchronization of AMs and intelligent control

loops are a major challenge.o Limitations:

Dynamic test planning (current: static lookup)

Need to implement Safe Adaptation with Validation Strategy

Code-based changes

28

Related Worko Towards Self-Testing in Autonomic

Computing Systems, King et al., ISADS ‘07.o A Self-Testing Autonomic Container, Stevens et

al., ACMSE ‘07.

o Synthesizing assertions from observed behavior, Denaro et al., ACC ’05. Embeds assertions into the communication

infrastructure.Assertions are checked at runtime.

o Making components self-testable, Le Troan et al., TOOLS ’99.

29

Conclusion and Future Worko Proposed a framework that dynamically

validate change requests in AC systems.o Approach extends the current structure of AC

systems to include self-testing.o Supports two validation strategies.o Developed a prototype to show their feasibility.o Future work calls for:

Extending the capabilities of the prototype Implementing safe adaptation with validation Evaluating efficiency of the two approaches

30

Summer ’07 and Ongoing Worko REU Program 2007 – Alain E. Ramirez and

Barbara Morales; Autonomic Job Schedulero A Generic O-O Design to Support Self-Testable

Autonomic Systems (SAC ‘07)o Student Paper to ACMSE 2008

o Fall 2007 – Gonzalo Argote-Garciao Embedding Formal Specifications into Autonomic

Components

o ? – Alain E. Ramirezo Applying Design Patterns to AC systems

31

Acknowledgements

Djuradj BabichJonatan Alava

Ronald StevensBrittany Parsons

Mitul PatelDr. S.M. Sadjadi

32

Thank You

Questions?

This work has been supported in part by the National Science Foundation under grant IIS-0552555