t# @ agile tour montreal 2009 en

©2009 Pretty Objects Computers inc. All Rights Reserved.

Agile Tour Montréal 2009

Ludovic [email protected]

www.prettyOBJECTS.com

Well tested or not?That is the question!



Agenda

Pretty Objects – Visual T# Survey A good test is… Conclusion



Pretty Objects

• Mission: help OO developers (1993)• Three service offers:

– Training(1993)o In partnership with CRIMo Trained over 10,000 participants

– Consultation – Coaching (1994)o Object oriented – automated unit testso Speciality: Microsoft.NET

– Development of development tools(1995)o POCMock: VS addin for whitebox testing (2003)o T#: .NET language to simplify writing unit tests (2009)



Visual T#

• 2005: Idea to simplify writing tests, in particularly when using mocks

• 2005: T# documentation• 2006: Beginning of development• 2007: C# compiler done (over 8,000 tests)

Beginning of Visual Studio integrationComplete rewrite of T# documentation

• 2008: T# compiler done (over 12,000 tests),Visual Studio integration still going!

• 05/2009: T# v1.0 (almost 18,000 tests)• 09/2009: T# v1.5 (almost 20,000 tests)• 10/2009: T# v1.6 (over 22.000 tests)



Agenda

Pretty Objects – Visual T#Survey A good test is… Conclusion



Survey

• Who knows about unit testing theory?• Who develops unit tests?• Who uses NUnit? MSTests? Other?• Who has written over:

– 100 tests?– 1,000 tests?– 10,000 tests?



Agenda

Pretty Objects – Visual T#SurveyA good test is… Conclusion



An existing test!

No test = the worst of tests! Unless it is… … the test which verifies the contrary of what we want!

Test = a will to test– Is the good will to test sufficient?

Test written after the code– « Does this test actually tests something? »

Test written before the code– Think about the method’s objective prior to creating it



An immutable test

Never fix a bug… Write a test which proves the existence of the bug Correct the code to make the test pass Verify that the bug has disappeared from the original

context

Never change a test… Unless the business objective or need has changed In case of a new bug: duplicate the test and change the

copy



A test focused on the essential

Generalize whenever possible– Define general contexts for your tests– Separate tests in many classes (because the contexts

are different): o Constructorso Class declarations (statics)o Instance declarations (non-statics)

The context creates an instance

Limit code length– Create high-level Assert… methodsÞ Writing, maintaining and refactoring becomes much

easier



A test focused on the essential

protected void AssertLinked( Invoice invoice, params object[] products ){ Assert.IsTrue(products.Length % 2 == 0, "Bad test!"); Assert.AreEquals( products.Count / 2, invoice.Count ); for( int i = 0; i < invoice.Count; i += 2 ) { Item item = invoice[ i / 2 ]; Assert.AreSame( products[i], item.Product ); Assert.AreEqual( products[i + 1], item.Quantity ); }}[TestMethod]public void Purchase(){ … AssertLinked( invoice, product1, 2, product2, 4 );}



A test with a clear objective

Think about the objective prior to coding it!– « What does it test? »

o Which declaration? Of which class?

– « Under which situation? »o States which influences the execution of the tested

declaration? An instance’s state A class’ state Other instances/class’ states The environment’s state: files, DB…

o What are the parameters during the invocation?



A test with a clear objective

Non formalized objectives– « What does it test? »: name of a method!– « Under which situation? »: name of a method! Not easily manageable! Rests on the developer’s discipline



T#: a test with a clear objective!

Formalized objectives– « What does it test? »: indicate the tested declaration– « Under which situation? »: make use of criteria Knowledge of what is being tested and by which tests Compiler validation Missing tests notification at compilation Execution of tests from your tested declaration



A test with a clear objective?

[TestClass]public class ProductTest{ [TestMethod] public void PriceChange_SomeRandomValue() { … }

[TestMethod] public void PriceChange_MinimumValue() { … }}



T#: a test with a clear objective!

testclass for Product{ test Price set when MinIncluded.IsAboveMin { … }

test Price set when MinIncluded.IsMin { … }}

// The compiler will warn you of a missing// test for MinIncluded.BelowMinCase



Tests covering all cases

Determine the test cases for the declaration– Normal cases (such as IsAboveMin, IsMin)– Error cases (such as BelowMinCase)

Détermine the cases for each parameter

Write a test for each possible combination– Combine valid cases– Isolate error cases



Tests covering all cases

Test coverage– Technical, not logical!

o Doesn’t indicate missing tests, only code that hasn’t been used by the tests

o Doesn’t properly cover all cases if( M1() && M2() )

with M1() always true, M2() variable

– Must run all the tests to know which ones are missing– If the coverage isn’t complete: there’s a problem– If the coverage is complete: it doesn’t prove anything!



T#: Tests covering all cases!

Using tests criteria– Definition of normal cases (IsAboveMin, IsMin)– Definition of error cases (BelowMinCase)– Logical definitions, not technical ones

Write a test per possible combination– Define case expressions for each test

o Use the following operators &&, ||, => and ()Þ The compiler will warn you about the missing tests




criteria MinIncludedCriteria{ IsAboveMin, IsMin, [Error] BelowMinCase}

criteria NotifyPropertyChangedCriteria{ HasNoSubscriber, HasSubscribersSetSameValue, HasSubscribersSetOtherValue}




test Price set when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasNoSubscriber{ … }

test Price set when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscribersSetOtherValue{ … }

test Price set when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscribersSetSameValue{ … }

test Price set when MinIncluded.BelowMinCase { … }



A test with a clear implementation

A couple of lines of code– « What does it test? » « Under which situation? »– « For what outcome? »

Separate it in 3 distinctive parts:– Execution: « What does it test? »

o Write a single line of code– Preparation: « Under which situation? »

o Write the most necessary code in the tests’ contexto Validate the situation at the end of each preparation

– Verification: « For what outcome? »o Correspond the assertions before and after the execution

The effect of the tests code becomes evident



A test with a clear implementation?

Context in two methods– A method before running the test– Another one after running the test Instance variables instead of local ones Impossible to protect the code

o No try…catch or finally blocks. Impossible to repeat tests



T#:a test with a clear implementation!

Context defined in only one method– runtest keyword used like a placeholder for the test Local variables instead of instance variables Possibility to protect the code

o try…catch or finally around runtest Easy repetition of tests

o Use of multiple runtesto Loop over runtesto Define a new situation before each call to runtest




[TestClass]public class ProductTest{ private Product p;

[TestInitialize] public void Setup() { p = new Product( "T#", 123 ); }

…



T#:A test with a clear implementation!

testclass for Product{ private Product p;

testcontext { p = new Product( "T#", 123 ); runtest; }

…




3 parts well distinctive?– With comments in the code Discipline of each developer

Assertions by methods– Lots of different Assert…– Not always natural

o Expected value as first parametero No automatic casting of types (MSTests)




Keyword: runtest– Execution: identified by the runtest keyword– Preparation: before runtest– Verification: after runtest Validation done by the compiler

Keyword: assert– Followed by a boolean expression– Whichever order of expected and actual values– Automatically generated error messages More natural to use




[TestMethod]public void PriceChange_SomeValue_NoSubscribers(){ // Preparation Assert.AreNotEqual( 234.0, p.Price );

// Execution p.Price = 234;

// Verification Assert.AreEqual( 234.0, p.Price );}




test Price set when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasNoSubscriber{ assert p.Price != 234;

runtest p.Price = 234;

assert p.Price == 234;}



A test verifying all effects

Verify all wanted effects– Each test has assertions for every wanted effect– Relative vs absolute assertions

o Relative: « 1 more than before » Clearly shows the change of value More code, more complex, less maintenance

o Absolute: « now equals to 1 » Ensures fixed values Less code, more simpler, more maintenance

Verify all unwanted effects– Each test has assertions for every unwanted effects



A test verifying all effects?

Verify all wanted effects– One Assert. … for each wanted effect– More complex scenarios: events

o TONS of codeÞ Objective of the test is flooded in the code

– Sometimes not well tests: exceptionso Makes use of attributesÞ Test passes if any line of code raises an exception!Þ Impossible to add further assertions



T#: a test verifying all effects!

Verify all wanted effects– One assert keyword used for any verification

o assert <expression booléenne>o assert [!] raised <event>

1 single line of code!o assert thrown <exception>

1 single line of code! Assertion for the « Execution » part only Other possible assertions

o Automatic generated message (more detailed)




[TestMethod]public void PriceChange_SaveValue_HasSubscriber(){ // Preparation p.PropertyChanged += PropChanged; eventRaised = false;

// Execution p.Price = p.Price;

// Verification Assert.IsFalse( eventRaised );}private bool eventRaised;private void PropChanged( object sender, EventArgs e ){ eventRaised = true;}




test Price set when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscriberSetSameValue{ runtest p.Price = p.Price;

assert !raised p.PropertyChanged;}




[TestMethod]public void PriceChange_SomeValue_HasSubscriber(){ // Preparation p.PropertyChanged += PropChanged; eventRaised = false; Assert.AreNotEqual( 234.0, p.Price );

// Execution p.Price = 234;

// Verification Assert.IsTrue( eventRaised ); Assert.AreEqual( 234.0, p.Price );}




test Price set when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscriberSetOtherValue{ assert p.Price != 234;


assert raised p.PropertyChanged; assert p.Price == 234;}




[ExpectedException(typeof(ArgumentOutOfRangeException)]public void PriceChange_ImpossibleValue(){ // Preparation // If the preparation code throws an expected // exception, the test passes!!! Assert.AreNotEqual( -1.0, p.Price );

// Execution p.Price = -1;

// Verification // Instruction never reached (else: problem!) // Impossible to add further assertions // If reached and throws expected exception, // the test passes!!!}




test Price set when MinIncluded.BelowMinCase { // If the preparation code throws the expected //exception, the test fails assert p.Price != -1;

runtest p.Price = -1;

assert thrown ArgumentOutOfRangeException; // Add as many assertions as needed // If the verification code throws the expected // exception, the test fails}




Verify all wanted effects…cont’d…– Relative assertions in 1 single line of code

o assert changed <expression> assert changed collection.Count++ assert changed collection.Count += 1 assert changed collection.Count =

collection.Count + 1




[TestClass]public class InvoiceTest{ private Invoice invoice; private Product product; … [TestMethod] public void Add_New_MultipleProducts_Test() { // Preparation int totalBefore = invoice.Total; Assert.AreEqual( 0, invoice.Count ); // Execution invoice.Add( product, 2 ); // Verification Assert.AreEqual( 1, invoice.Count ); Assert.AreEqual( totalBefore + 2 * product.Price, invoice.Total );}




testclass for Invoice{ private Invoice invoice; private Product product; … test Add( Product p, int qty ) when p: ValidReference.IsNotNull, qty: MinIncluded.IsAboveMin, Collection.IsEmpty && CollectionItem.IsNotInCollection{ assert invoice.Count == 0;

runtest invoice.Add( product, 2 );

assert changed invoice.Count++; assert changed invoice.Total += 2 * product.Price;}




Verify what you do not wantÞ Lots of code…blurry

Þ Code in « Preparation » to saveÞ Code in « Verification » to compare

Þ Objective of the test is flooded in the code




Verify what you do not want– Always the same keyword: assert !changed

o For only public propertiesassert !changed obj.*;

o For all variablesassert !changed obj.-*;

– No changes except…: …except…o Some properties (variables)

assert !changed obj.* except Property;o An absolute assertion

assert !… except Property == 1;o A relative assertion

assert !… except changed Property++;




[TestMethod]public void PriceChange_ImpossibleValue(){ // Preparation double actualPrice = p.Price; Assert.AreNotEqual( -1.0, actualPrice ); // Execution try { p.Price = -1; // Verification Assert.Fail("ArgumentOutOfRangeException

expected!"); } catch( ArgumentOutOfRangeException ) { Assert.AreEqual( actualPrice, p.Price ); }}




test Price set when MinIncluded.BelowMinCase { assert p.Price != -1;

runtest p.Price = -1;

assert thrown ArgumentOutOfRangeException; assert !changed p.*; // Verifies the constancy of Price, Number, // Description, etc.}




test Price set when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscriberSetOtherValue{ assert p.Price != 234;


assert raised p.PropertyChanged; assert !changed p.* except Price == 234;}



A test not altering the tested code

Prefer « black box » tests– Test only available services– Not the internal organization of classes

Not always thinkable– Simple invocation of the service, but complex to handle

o A compiler has only one method:Compile(sourceFiles, destinationFiles)

– Test hidden parts of higher level

Changing code so that it can be testable is…detestable!



• Test hidden parts Don’t test them

o Doesn’t correspond to our need!o Who is going to use those parts? For what outcome?

Use meta-programming in the testo typeof(…).GetMethod(…).Invoke(…)Þ Blurs the test codeÞ Harder to read and understandÞ Error at runtime instead of compile time

– Hide the meta inside extension methods in the test moduleo Less bad solution!

A test not altering the tested code?



T#: a test not altering the tested code!

• Test hidden parts .- operator

o As simple as the . operatoro Access any declaration, even private oneso Access any type, even internal oneso Doesn’t change the tested codeo More effective than reflection



A test easy to manage

Pass the tests one by one– If you’re writing many tests, ignore all but one– Then make another one pass…

Instantly find tests regarding a declaration

Easily find what’s going on when a test fails



A test easy to manage?

Find the tests– 1 class to test = many test classes– 1 declaration to test = many test methods

Find out what’s wrong– 3 states for a test:

o Ignored: temporarily on holdo Correct: it passes, all is good

Uh…really?o Failed: the code is not good

Which code? The tested code? The test code?



T#: a test easy to manage!

Find the tests– Organize the tests according to the tested code– Execute the tests from the tested code

Find out what’s wrong– 4 states for a test:

o Ignored: temporarily on holdo Correct: it passes, all is goodo Invalid: the test code is not good (for sure)o Failed: the tested code is not good (probable)



T#: a test easy to manage!



Agenda

Pretty Objects – Visual T#SurveyA good test is…Conclusion



Conclusion

Visual T# is free!

http://forum.prettyobjects.com/forum2-t-download.aspxhttp://fr.wikipedia.org/wiki/Visual_T_Sharp

What are you waiting for to write good tests?



Thanks! :)

Pretty Objects Computers inc.550 Sherbrooke Ouest, suite 100Montreal, H3A 1B9TÉL.: 514-840-1253FAX: [email protected]

Conclusion

t# @ agile tour montreal 2009 en

Documents