efficient systematic testing for dynamically updatable software

Download Efficient Systematic Testing for Dynamically Updatable Software

Post on 23-Feb-2016

20 views

Category:

Documents

0 download

Embed Size (px)

DESCRIPTION

Efficient Systematic Testing for Dynamically Updatable Software. Christopher M. Hayden, Eric A. Hardisty , Michael Hicks, Jeffrey S. Foster University of Maryland, College Park. Dynamic Software Updating (DSU). Performing updates to software at runtime has clear benefits: - PowerPoint PPT Presentation

TRANSCRIPT

Slide 1

Efficient Systematic Testing for Dynamically Updatable SoftwareChristopher M. Hayden, Eric A. Hardisty, Michael Hicks, Jeffrey S. Foster

University of Maryland, College Park

1Dynamic Software Updating (DSU)Performing updates to software at runtime has clear benefits:Increased software availabilityNo need to terminate active connections / computation but can we trust updated software?Critical to ensure updates are safe22Our ContributionsVerification of DSU through testing:Testing ProcedureTest Minimization AlgorithmExperimentsEffectiveness of MinimizationEmpirical study of Update Safety / Safety Checks33DSU SafetyDSU creates the opportunity for new sources of bugs:Faulty state transformationUnsafe update timingSafety Checks restrict when updates may be appliedActiveness Safety / Con-freeness Safety

44Activeness Safety (AS)AS prevents updates to active codeIn this example, no patch updating main or foo is allowed:5main() { foo();

baz(); }foo() { bar();}5Con-freeness Safety (CFS)CFS (Stoyle, et al 05) allows updates to active code only when type safety can be ensuredIn this example, no patch updating the signature of baz or bar is allowed:

6main() { foo();

baz(); }foo() { bar();}6DSU TestingSafety Checks offer limited guarantees:CFS and AS ensure type-safe execution AS ensure that you never return to old code following an updateNeither of these properties ensure safe update timingWe propose testing to verify the correctness of allowed update points:Use existing suite of application system testsEnsure that updating anywhere during the execution of those tests results in an execution that passes the test.77Testing ProcedureApproach:Instrument application to trace update pointsExecute system test and gather initial traceFor each update point in the initial trace, perform an update test: force an update at that point while executing the system testPotential Update PointsTrace Start88Testing ProcedureApproach:Instrument application to trace update pointsExecute system test and gather initial traceFor each update point in the initial trace, perform an update test: force an update at that point while executing the system testinitial trace99Testing ProcedureApproach:Instrument application to trace update pointsExecute system test and gather initial traceFor each update point in the initial trace, perform an update test: force an update at that point while executing the system testinitial traceupdate tests1010Update Test MinimizationProgram traces may have thousands or millions of update pointsMany update tests have the same behavior for a given patchwe can eliminate redundant tests

baz() {}Patch Avoid main() {

foo();

bar();

baz();}Version 0foo() {}bar() {}baz() {}Patch BAll update points yield same behaviorAll update points yield distinct behavior1111Minimization AlgorithmExecution events are traced if they have the potential to conflict with a patchA event conflicts with a patch p if applying p before the event might produce a different result than applying p after the eventExample: function calls, global variable accessesTrace the execution of a test T on P0Iterate through the trace noting the last update point each time we reach a conflicting trace elementRun only the identified update tests Tnp1212Experimental Results1313Experimental SetupBased testing infrastructure on top of the Ginseng DSU system (Neamtiu, et al):Modified to support tracing and updating at pre-selected update pointsInsertion of explicit update points before each function call to approximate more liberal systemsDisabled safety checking (CFS) for experimentsTested 3 years of patches to OpenSSH and vsftpd (only report OpenSSH in this talk)1414Program Modifications15foo() { while (1) { // main loop

update();

extract { ... // main loop body } } extract { ... // after main Loop }}Identify Long-running loopsAdd a Manually Selected Update PointPerform Loop Body ExtractionPerformContinuation Extraction15Experiments: Update Test SuiteHow many update tests must be run to test real-world updates to real-world applications?How effective is minimization at eliminating redundant tests?16Our first experimental results are a look at the test suite sizes that are produced by our testing procedure and minimization algorithm.16Update Test Suite Size: OpenSSH#D to next versionReductionSigFunTypeAll PointsActiveness-Safe Points03985580,871g31,791(95%)35,314g3,027(91%)1060705,322g1,795(~100%)587,578g1,717(~100%)2523811638,720g63,011(90%)20,902g2,353(89%)30180772,198g4,324(99%)638,803g3,775(99%)41317210773,086g27,399(96%)21,343g1,564(93%)50241878,235g17,398(98%)111,950g1,723(98%)6625710879,668g47,092(95%)44,278g2,139(95%)7417912918,717g89,601(90%)100,854g4,141(96%)80723973,364g34,293(96%)61,724g2,070(97%)9101577933,514g52,356(94%)61,051g2,891(95%)Total8,053,695g369,060(95%)1,683,797g25,400(98%)1717Empirical Study of Update SafetyHow many failures occur when applying updates arbitrarily?How many failures occur when applying updates subject only to the AS and CFS safety checks?

1818Safety: OpenSSH19D to next versionAll PointsCFS PointsAS PointsUpdateSigFunTypeFailedTotalFailedTotalFailedTotal0398519,715580,871068,044035,31410600705,3220705,3220587,5782523811306,965683,7201,68875,307420,902301800772,1980772,1980638,80341317210565,681773,086609110,63338021,3435024110,703878,2350130,0000111,9506625710163,333879,66844,46196,18311044,278741791211,380918,717180,0701100,854807233973,3640261,885061,7249101577357,919933,51424121,337061,051Total1,435,6998,053,69546,7832,420,9794951,683,79719void handle_upload_common() { ret = do_file_recv();}

void do_file_recv() { // receive file if (ret == SUCCESS) write(226, OK.); return ret;}Version 0void handle_upload_common() { ret = do_file_recv(); if (ret == SUCCESS) write(226, OK.);}

void do_file_recv () { // receive file return ret;}Version 1 (patch)Unsafe Timing:Version Inconsistency (vsftpd)2020Unsafe Timing:Version Inconsistencyvoid foo() { bar(); baz();}

void bar() { }

void baz() { dig(); }Version 0void foo() { bar(); baz();}

void bar() { dig(); }

void baz() { }Version 1 (patch)21Manually Selected Update Points22D to next versionSafety#TestsSigFunTypeReductionFailedTotal0753985566g566(0%)0566175060630g592(6%)0630276523811568g568(0%)05683910180783g770(2%)07834911317210782g782(0%)078251040241860g841(2%)08606104625710859g859(0%)08597104417912850g850(0%)085081050723868g823(5%)08689104101577833g833(0%)0833Total7,599g7,484(2%)07,59922SummaryWe have argued that verification is necessary to prevent unsafe updatesProvided empirical evidence that AS/CFS cannot prevent all unsafe updatesWe have presented an approach for testing dynamic updatesWe have presented and evaluated a minimization strategy to make update testing more practical2323Discussion QuestionsGiven that AS cannot ensure correctness (both in theory and in practice), should DSU implementations continue to rely on it?What standards for verification should be required of DSU system benchmarks?Are there other assumptions of DSU that are appropriate for empirical evaluation?2424

Recommended

View more >