introduction to automated tests -...
TRANSCRIPT
Introduction to Automated Tests
Paulo Cheque
Summer 2009
License: Creative Commons: Attribution-Share Alike 3.0 Unportedhttp://creativecommons.org/licenses/by-sa/3.0/
2
About● Language● Types of tests● Introduction to test frameworks
3
Many communities
● A lot of terminology and dialects:● Agile communities● Formal communities● QA communities● Tools● Clients
4
Standards Specifications● BS 79252:1998. Software Component Testing.● DO178B:1992. Software Considerations in Airborne Systems
and Equipment● Certification, Requirements and Technical Concepts for
Aviation (RTCA SC167).● IEEE 610.12:1990. Standard Glossary of Software Engineering
Terminology.● IEEE 829:1998. Standard for Software Test Documentation.● IEEE 1008:1993. Standard for Software Unit Testing.● IEEE 1012:1986. Standard for Verification and Validation
Plans● ...
5
Jargons
• Defect/Mistake/Error/Failure• Verification• Validation• Inspection• Debugging• ...
6
Terms
• Blackbox/White/Gray/Glass• Functional Tests• Structural Tests
• Test• Test case• Test suite
7
Symbols
• SUT/AUT: System/Application Under Test• TFD/POUT: TestFirst Development / Plain Old
Unit Test• TAD: TestAfter Development• TDD: TestDriven Development / Design• BDD/EDD/STDD: Behaviour/Example/Story
Test Driven Development• TDDD: TestDriven Design Databases
8
Double● Dummy: A simple object to make a test executable● Fake: Lightweight object that contains a false
implementation● Stub: Object with data to be used in tests● Mock: Object with an appropriate interface that
contains data and behavior that will be used in tests● Spy: Object to capture indirect calls
9
Type of Automated Tests
● Unit● Integration● Acceptance● Interface● Smoke● Sanity● Mutation● ...
● Performance● Load● Stress● Longevity● Security● Installation● ...
10
Unit test
● Unit: Class/Module or Method/Function● Unit or “miniintegration”
● Test a feature, not an implementation● Basic but very important test● Solid test● Avoid debugging
11
Integration tests
● In general: Test that involve more than one unit
● Verify errors in the relationship between modules, components or layers that may work fine individually
12
Acceptance
● Test a feature from the user's point of view● Assurance for a client that software works
perfectly and correctly
● Clients, users and developers may (must) work together to write tests● Implementation details hidden● Client language – Domain Specific Language
13
Interface Tests● User Interface
● Console● GUI● WUI
● => Acceptance Tests● => Usability● => Layout Tests
14
Smoke Tests
● Rapid, extensive and superficial tests
● After software installation
● Example:● Visit a Web page and verify that 404 text is not
visible
15
Sanity Tests
● A smoke test
● Appropriate to find huge errors in algorithms
● Use particular technique of an algorithm to identify possible errors● Theorems● Mathematical properties
16
Mutation tests
● Test a test● “Tests may show some errors, but not its
absence” Dijkstra● Infinite possibility of errors?
● Competent developer● Mutant: SUT with few changes● Run a suite of tests under mutants● Analysis of results: Mutants and Tests
17
Performance tests● Evaluate response time of a specific module● Profilers help finding bottlenecks● Unit test suites without mocks may be used
like profilers (this is not the main goal of those suites)
● PS: Don't evaluate computational complexity
18
Load tests● Simulate a lot of users and requests
● Simultaneous or inside a time interval
● Test● InfraStructure● Hardware● Network bandwidth● Database and servers
19
Stress tests● Maximum load test● Identify maximum amount of:
● users● requests● data
● Limitations of an environment
20
Longevity tests● Test to verify performance and correction
within a large period of time● Find problems with memory leakage● Find cache errors
21
Other types...● Tests of:
● Installation: Verify if all components are interconnected properly without hardware incompatibility
● Recovery: Appropriated reaction after errors● Configuration: Different configurations and
distinct environments (portability)● ...
● Alfa / Beta versions
22
Security tests● Useful for applications exposed to bad
intentioned users● Web, banks ...
● Test:● Permissions errors● Errors in model layer (SQLInjection ...)● Bugs in servers● Fragility to attacks like DoS (Stress tests)
23
Pertinent information● Machine: CPU, Memory, HD, Swap ...● OS: Name/Distribution, Version, Kernel, Locale ...● Servers: NameVersion, Pertinent files,
● Tomcat 5.5, Jboss 4.0.4, (/etc/init.d/jboss)● Database: NameVersion, Machine, Schema● Network: IP, Firewall, Port, DNS ...● Dependencies: NameVersion
● JDK 1.5.0_06, Ruby 1.8● Product: Name, Version, installation data, logs
24
Unit tests
• Main• XUnit frameworks
– Test case– Verification– Exceptions– Results– Report
25
Main
public class MainExample {
public static void main(String[] args) {if(!new Double(Math.sqrt(1)).isNaN())
throw new RuntimeException("Ops!");// Orassert new Double(Math.sqrt(1)).isNaN() == true;
}
}
26
xUnit● Article:
http://junit.sourceforge.net/doc/cookstour/cookstour.htm
27
Test case
● Conventions● public void testSomeMethodName() { ... }
● Annotations@Test public void someMethodName() { ... }
● Configurations● xml...
28
Verification● AssertTrue / AssertFalse● AssertEquals● AssertSame● Fail● Expected Exceptions
● Friendly messages
● Hamcrest: Matchers + AssertThat
29
Demonstration
● C● Java● Scala● Groovy
30
Setup e Teardown
● Test cases must be independent:● From each other● From number of times it has been executed● From external factors
● Setup: Prepare environment to test ● Teardown: Clean environment for other
tests
31
Results
● Skip● Fail● Error● Success
32
Reports
● Language● APIDocs
● Tools● StackTrace● Logs● Screenshots
● Metrics
33
Some tools● Unit tests:
● CxxTest (C++): http://cxxtest.sourceforge.net● CUnit: http://cunit.sourceforge.net● JUnit (Java): http://www.junit.org● DUnit (Delphi): http://dunit.sourceforge.net● VBUnit (Visual Basic): http://www.vbunit.com● TestNG (Java): http://testng.org● RSpec (Ruby): http://rspec.info/
● http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks
34
+some tools● Mock Objects:
● SevenMock (Java): http://sevenmock.sourceforge.net● EasyMock (Java): http://www.easymock.org/● JMock (Java): http://www.jmock.org● Rhino.Mocks (.NET):
http://www.ayende.com/projects/rhinomocks.aspx● SMock (Smalltalk):
http://www.macta.f2s.com/Thoughts/smock.html● Mockpp (C++): http://mockpp.sourceforge.net● GoogleMock (C++): http://code.google.com/p/googlemock
35
+some tools● User Interface tests for Desktop:
● Fest: http://fest.easytesting.org/swing● Jemmy: http://jemmy.netbeans.org● Marathon: http://www.marathontesting.com
● Web interface tests:● Selenium: http://www.openqa.org/selenium● Watir: http://wtr.rubyforge.org
● Acceptance tests: ● Fit: http://fit.c2.com
36
+some tools● Mutation tests:
● Jabuti (USP São Carlos): http://jabuti.incubadora.fapesp.br
● Jester: http://jester.sourceforge.net● Heckle: https://rubyforge.org/projects/seattlerb
● Performance / Load tests:● JMeter: http://jakarta.apache.org/jmeter
● Others: ● JPDFUnit: http://jpdfunit.sourceforge.net
37
Some links http://www.testing.com http://www.opensourcetesting.org http://xunitpatterns.com http://www.mockobjects.com http://javasource.net/opensource/testingtools http://www.junit.org http://www.agilcoop.org.br
38
Some Books Gerard Meszaros, “xUnit Test Patterns, Refactoring Test Code”,
AddisonWesley, 2007 Robert V. Binder, “Testing ObjectOriented Systems”, Addison
Wesley Professional, 1999 L. Crispin, T. House, “Testing Extreme Programming”, Addison
Wesley, 2005 R. Mugridge, W. Cunningham, “Fit for Developing Software”,
Prentice Hall, 2006 M. Delamaro, J. Maldonado, M. Jino, “Introdução ao Teste de
Software”, Campus, 2007
39
Contact
http://www.agilcoop.org.brhttp://ccsl.ime.usp.br
http://qualipso.org
[email protected]@agilcoop.org.br
License: Creative Commons: Attribution-Share Alike 3.0 Unportedhttp://creativecommons.org/licenses/by-sa/3.0/