software reliability

Software ReliabilitySoftware Validation

Defensive ProgrammingSoftware Analysis Tools

Conclusion

Software ReliabilityJournee des Technologies de l’Information et de la

Communication

Baptiste WichtEIA-FR

June 3, 2014

Baptiste Wicht EIA-FR Software Reliability



Conclusion

EIA-FR




Conclusion

Table of Contents

1 Software Reliability2 Software Validation3 Defensive Programming4 Software Analysis Tools5 Conclusion




Conclusion

DefinitionsExamplesFactsSolutions

Contents

1 Software ReliabilityDefinitionsExamplesFactsSolutions

2 Software Validation3 Defensive Programming4 Software Analysis Tools5 Conclusion




Conclusion


Definitions

DefinitionProbability that a software will work properly in a specifiedenvironment for a given amount of time.

What makes a program reliable ?No bugMust work under any possible conditionMust comply to the specifications




Conclusion


Difficulties

properly, specified environment, given amount of timeNo specification is perfectSpecifications can changeSoftware is complex




Conclusion


Measuring reliability

Difficult problemHard to specifySoftware is complicated to quantify

Models are availableMore than 200 models existNo single model completely represent reliabilityBasic idea: Reliability is function of the number of defects




Conclusion


Examples

1991: Telephone outage in CaliforniaThree lines of code changedseveral-million lines of code program⇒ ≈ 200’000 phone calls lost

1991: Patriot Missile fails to intercept Scud MissileTime stored in tenths of a second, multiplied by 1/10 to getseconds24 bit fixed-point register, 1/10 has 0.000000095 errorAfter 100 hours, error of 0.34 seconds=> 28 dead




Conclusion


Examples (cont.d)

1996: Ariane 5 explodes after 37 seconds of flight16bit integer overflowOverflow detected ⇒ useless corrections ⇒ othermalfunctions ⇒ self-destructionSystem taken from Ariane 4, not used in Ariane 5⇒ 500M $ lost

2014: Heartbleed (OpenSSL)Missing bound checkAllows more data to be read than should be allowedIntroduced in 2011⇒ Millions of website vulnerable




Conclusion


Facts

Perfect software does not exist

The bigger the program ⇒ the more bugs

working today 6= working tomorrow

small change may have big impact




Conclusion


Solutions

Software ValidationSoftware Analysis ToolsDefensive Programming




Conclusion

Unit TestingIntegration TestingSoftware ValidationConclusion

Contents

1 Software Reliability2 Software Validation


3 Defensive Programming4 Software Analysis Tools5 Conclusion




Conclusion


Unit Testing

DefinitionSet of tests to verify that a small unit of code is correct

Automated testsShould be run after each change to the program and underproduction-like environmentMonitor code coverage




Conclusion


Integration Testing

DefinitionSet of tests to verify that a set of units forming a component iscorrect

Can be automated or performed by humanIf human-performed, can be very costlyShould be performed as much as possible




Conclusion


Software Validation

DefinitionTests to ensure that the complete system meets its specifications

Was the right software built ?Was the software built right ?Generally performed by Q/A specialists




Conclusion


Conclusion

No part of the code should be neglectedNo change should be neglectedSmall modules easily covered by a unit test should always beunit testedUnit/Integration tests should always be used to avoidregressionUnit testing should be done as part of Continuous IntegrationAutomated tests are less costly that human-performed tests




Conclusion

Language FeaturesContract ProgrammingConclusion

Contents

1 Software Reliability2 Software Validation3 Defensive Programming


4 Software Analysis Tools5 Conclusion




Conclusion


Defensive Programming

DefinitionMake a program more reliable by ensuring that code is used in thecorrect way and it does what it is supposed to do.

Protection against failures and faultWritten as codeAccurate information to the source of the errorVerified at compile-time or runtime




Conclusion


Language Features

Different languages provide different level of security

int array [100];int b = array [100]; //What should happen ?

C/C++: UndefinedJava/Python: Runtime error

Generally: lower level ⇒ harder to debug




Conclusion


Assertions

DefinitionPredicates (true-false statements) inserted into code indicatingwhat the developer thinks should be true at a certain point.

Indicates that a certain condition must be true

Typically aborts execution if condition not metMay be replaced by exceptionsIs generally disabled in productionShould not have any side effect




Conclusion


Assertions (cont.d)

double log(int x){assert(x > 0);//...

}

template <typename T>class Z {

static_assert(sizeof(T) > 2, "Invalid␣Type");//...

}




Conclusion


Assertions (cont.d)

ProsAvailable in almost every languagesVery simpleGenerally useful to check input parameters

ConsOnly verified at runtime

Some languages supports compile-time assertionsOnly local to a functionCan make the code more complexCan be disabled




Conclusion


Contract Programming

Contract of the code specified into the programPre/postconditions, Invariants, ...Can be implemented with assertionsCoupled with powerful static analysis, some errors can befound without running the program




Conclusion


Conclusion

ProsCatch errors with nice error messagesStored with the code

ConsMake the code more complexMay introduce new errorsMay incur a runtime costDependent on the developer




Conclusion

CompilersToolsExercisesSonarQubeConclusion

Contents

1 Software Reliability2 Software Validation3 Defensive Programming4 Software Analysis Tools


5 Conclusion




Conclusion


Software Analysis Tools

Many available toolsAutomatic search for bugs, performance improvements, styleproblemsTwo types:

Static Analysis: Source code levelDynamic Analysis: Runtime level




Conclusion


Compilers

Every modern compiler produces warningsOnly when askedMay produce too much warnings

Often forgottenRecommendation

New project: As much as possible and -WerrorExisting project: Fix existing warnings and enable warnings oneby oneCLang/GCC: -Wall -Wextra




Conclusion


cppcheck

Static Analysis for C/C++ programsFree software, GPL licenseCross-platformActive project, latest stable release 10 May 2014Available on every mainline distributionContains a large set of checkersIntegrated in several IDE (Eclipse, Visual Studio,Code::Blocks, CodeLite...)Generate text, XML and HTML reports




Conclusion


CLang Static Analyzer

Static Analysis for C/C++ and Objective-C programsFree software, MIT licenseDeveloped as part of the LLVM/CLang projectSlower than compilationStill young projectSeveral checkers implementedCan be extended with checker pluginsGenerate text and HTML reports




Conclusion


Valgrind

Framework for dynamic analysis toolsFree Software, GPL LicenseWork on Linux and Mac OS XMost known for Memcheck toolRuns the program into a virtual machineProgram runs slower (4-5 times slower with None tool)Generate text reports




Conclusion


Tools

Memcheck: Memory debuggerDefault and most used toolVery large performance lost (20 to 40 times slower)

Massif: Heap profilerHelgrind: Detect race conditionsCachegrind: Cache profilerCallgrind: Call-graph analyzerexp-sgcheck: Experimental tool for memory overruns




Conclusion


Exercises

Some exercisesMachines ready with UbuntuLaunch terminal

> cd examples




Conclusion


Memory Leak

int main (){int* a = new int (4);// Somethinga = new int (5);// Something elsedelete a;

}

> g++ leak.cpp> valgrind ./a.out




Conclusion


Memory Leak

==15386== HEAP SUMMARY:==15386== in use at exit: 4 bytes in 1 blocks==15386== total heap usage: 2 allocs, 1 frees, 8 bytes allocated==15386====15386== LEAK SUMMARY:==15386== definitely lost: 4 bytes in 1 blocks==15386== indirectly lost: 0 bytes in 0 blocks==15386== possibly lost: 0 bytes in 0 blocks==15386== still reachable: 0 bytes in 0 blocks==15386== suppressed: 0 bytes in 0 block

4 bytes have been lostNo information on location

> valgrind --leak -check=full ./a.out




Conclusion


Memory Leak

==6434== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1==6434== at 0x4028812: operator new(unsigned long) (vg_replace_malloc.c:319)==6434== by 0x400641: main (in /home/wichtounet/dev/analysis-examples/a.out)

The source of the allocation is now shownOnly binary location, no source location

> g++ -g leak.cpp> valgrind --leak -check=full ./a.out




Conclusion


Memory Leak

==9604== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1==9604== at 0x4028812: operator new(unsigned long) (vg_replace_malloc.c:319)==9604== by 0x400641: main (leak.cpp:2)

Source location (leak.cpp:2) is now completeValgrind will only show the sources of allocated leaked memorySometimes it may still be complicated to find why not freed




Conclusion


Heap Overrun

int main (){int* heap = new int [5];

heap [5] = 42;

delete [] heap;

return 0;}

> g++ -g heap_overrun.cpp> valgrind ./a.out




Conclusion


Heap Overrun

==16922== Invalid write of size 4==16922== at 0x40064E: main (heap_overrun.cpp:4)==16922== Address 0x50c4054 is 0 bytes after a block of size 20 alloc’d==16922== at 0x4028F40: operator new[](unsigned long) (vg_replace_malloc.c:384)==16922== by 0x400641: main (heap_overrun.cpp:2)

Source location of the invalid write (heap overrun.cpp:4)Source location of the allocation (heap overrun.cpp:2)

> valgrind --leak -check=full ./a.out




Conclusion


Stack Overrun

int main (){int i, a[10];

for (i = 0; i <= 10; i++)a[i] = 42;

return 0;}

> g++ -g stack_overrun.cpp> valgrind ./a.out




Conclusion


Stack Overrun

==20444== HEAP SUMMARY:==20444== in use at exit: 0 bytes in 0 blocks==20444== total heap usage: 0 allocs, 0 frees, 0 bytes allocated==20444====20444== All heap blocks were freed -- no leaks are possible

Overrun is not detectedSize of memory block is detected by malloc implementation

> valgrind --tool=exp -sgcheck ./a.out




Conclusion


Stack Overrun

==24703== Invalid write of size 4==24703== at 0x400572: main (stack_overrun.cpp:5)==24703== Address 0xfff0000c8 expected vs actual:==24703== Expected: stack array "a" of size 40 in this frame==24703== Actual: unknown==24703== Actual: is 0 after Expected

Overrun is detected!

> valgrind --tool=exp -sgcheck ./a.out




Conclusion


Stack Overrun 2

int main (){int stack [5];

stack [5] = 0;

return 0;}

> g++ -g stack_overrun_2.cpp> valgrind --tool=exp -sgcheck ./a.out




Conclusion


Stack Overrun 2

==28007== exp-sgcheck, a stack and global array overrun detector==28007== NOTE: This is an Experimental-Class Valgrind Tool==28007== Copyright (C) 2003-2013, and GNU GPL’d, by OpenWorks Ltd et al.==28007== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info==28007== Command: ./a.out==28007====28007====28007== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0

Overrun not detected!Need a valid access before the invalid one

> cppcheck stack_overrun_2.cpp




Conclusion


Stack Overrun 2

Checking stack_overrun_2.cpp...[stack_overrun_2.cpp:4]: (error) Array ’stack[5]’ accessed at index 5, which is out of bounds.

Overrun detected!Tools put together are very powerful




Conclusion


Complex Memory Leak

int main (){int* array [42];

for(int i = 0; i < 42; ++i){array[i] = new int;

}// Somethingfor(int i = 0; i < 41; ++i){

delete array[i];}

}

> cppcheck --enable=all leak_2.cpp




Conclusion


Complex Memory Leak

Checking leak_2.cpp...Checking usage of global functions..

cppcheck static analysis not powerful enough

> g++ -g leak_2.cpp> valgrind --leak -check=full ./a.out

==12868== 4 bytes in 1 blocks are definitely lost in loss record 1 of 1==12868== at 0x4C2C167: operator new(unsigned long) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)==12868== by 0x400739: main (leak_2.cpp:7)




Conclusion


Yet Another Leak

int main (){unsigned int* mem =

(unsigned int*) malloc (4);if(mem)

return 1;*mem = 0xdeadbeaf;free(mem);return 0;

}




Conclusion


Yet Another Leak

> clang --analyze leak_3.cpp

leak_3.cpp:7:16: warning: Potential leak of memory pointed to by ’mem’return 1;

ˆleak_3.cpp:8:10: warning: Dereference of null pointer (loaded from variable ’mem’)

*mem = 0xdeadbeaf;˜˜˜ ˆ

2 warnings generated

> c++ analyzer leak_3.cpp

leak_3.cpp:7:16: warning: Potential leak of memory pointed to by ’mem’return 1;

ˆleak_3.cpp:8:10: warning: Dereference of null pointer (loaded from variable ’mem’)

*mem = 0xdeadbeaf;˜˜˜ ˆ

2 warnings generated.




Conclusion


Yet Another Leak

> scan -build clang++ leak_3.cpp

scan-build: Using ’/usr/bin/clang’ for static analysisleak_3.cpp:7:16: warning: Potential leak of memory pointed to by ’mem’

return 1;ˆ

leak_3.cpp:8:10: warning: Dereference of null pointer (loaded from variable ’mem’)*mem = 0xdeadbeaf;˜˜˜ ˆ

2 warnings generated.scan-build: 2 bugs found.scan-build: Run ’scan-view /tmp/scan-build-2014-06-02-103126-14357-1’ to examine bug reports.

> scan -view /tmp/scan -build -...




Conclusion


Yet Another Leak

> scan -build -enable -checker alpha make all> scan -view /tmp/scan -build -...




Conclusion


SonarQube

Web platform for software quality managementMulti-project, Multi-languageOpen Source and free

Some plugins are commercial (e.g. C++)Centralized view for code source quality metrics

LOC, Cyclomatic ComplexityWarnings, style errorsUnit Test Results, Code CoverageDocumentation CoverageDuplications

Views by components (package, folders, classes, ...)




Conclusion


Demo

Online demo: http://nemo.sonarqube.org/


http://nemo.sonarqube.org/



Conclusion


Conclusion

Numerous other tools exist:Python: PylintJava: FindBugs, PMD, CheckstyleJavascript: JSLint, JSHintPowerful commercial tools: PVS-Studio, Coverity, ......

Tools helps in finding existing very quicklyVery powerful when coupled with continuous integration




Conclusion

Contents

1 Software Reliability2 Software Validation3 Defensive Programming4 Software Analysis Tools5 Conclusion




Conclusion

Conclusion

Build reliable software is hardSoftware testing is very importantNumerous tools are available

No tool will ever find all bugsSome tools are overlapping




Conclusion

Recommendations

No change should be ignored when testingTests should be run in production-like environmentContinuous Integration and TestingMake use of the existing analysis features at each levelNo tool is perfect ⇒ Several tools can be used together




Conclusion

Questions

Questions ?
