secret unit testing tools

The secret unit testing tools no one has ever told you about

Dror Helper | blog.drorhelper.com | @dhelper

Join the conversation on Twitter: @DevWeek // #DW2016 // #DevWeek

http://blog.drorhelper.com/


About.MEConsultant @CodeValue

Developing software (professionally) since 2002Mocking code since 2008Clean coder & Test Driven Developer

OzCode (a.k.a “Magical debugging”) Evangelist

Blogger: http://blog.drorhelper.com




But it was not always like that1st Attempt Failed!

2nd Attempt Failed!!!

New job + UT + Mentor Success


Why should I care about tools?


Background: unit testing tools


Well known unit testing tools

Build Failed!


Server

Dev Machine

Source ControlBuild Server

Test Runner

Code Coverage

Build Agent

Unit Testing Framework

Isolation Framework

?


xUnit test framework

Test Suite

Fixture

Test Case

Test Case

Test Case

Test Case

Test Case

FixtureTest Case

Test Case

Fixture

Test Case

Test Case

Test Case

public class BeforeAndAfter { [SetUp] public void Initialize() { }

[TearDown] public void Cleanup() { }

[Test] public void test1() {

}

[Test] public void test2() { }}

Test Fixture


Mocking FrameworksUnit test

Code under test

DependencyFake object(s)


What Mocking framework can do for you?• Create Fake objects

• Set behavior on fake objects

• Verify method was called

• And more...


[Test]public void Calculate_ReturnTwoValidNumbers_ServerCalled(){ IDataAccess fakeDataAccess = A.Fake<IDataAccess>(); A.CallTo(() => fakeDataAccess.GetData(A<string>.Ignored))

.Returns(new Tuple<int, int>(2, 3));

var fakeCalculatorService = A.Fake<ICalculatorService>();

var cut = new DistrobutedCalculator(fakeDataAccess, fakeCalculatorService);

cut.Calculate();

A.CallTo(() => fakeCalculatorService.Add(2,3)).MustHaveHappened();}

These tools do not help us write good unit tests

In fact, sometimes they prevent us from

writing good unit tests!


Definition: unit testsAutomated piece of code that invokes a unit of work in the system and thenchecks a single assumption about thebehavior of that unit of work

[Roy Osherove, The Art Of Unit Testing]


Unit test structure[Test]public void MyTest(){

}


No guidance Fragile tests Stop unit testing


What about AAA?[Test]public async void GetUserFromUrl() { var clientFake = A.Fake<IJsonClient>(); A.CallTo(() => clientFake.HttpGetUncompressedAsync(A<string>.Ignored))

.Returns(Task.FromResult(JsonResult));

var userRepository = new UserRepository(clientFake);

var user = await userRepository.GetUser(11361);

var expected = new User { Id=11361, DisplayName = "Dror Helper", ImageUrl=DefaultAvatar, Reputation=13904 };

Assert.That(user, Is.EqualTo(expected));}

Arrange

Act

Assert


Problem solved? Hardly!• Where to start?• How to test existing code?• What about test structure?• Integration tests vs. unit tests• What is a “unit of work”?


Test setup (Arrange)


Arrange issues[TestMethod]public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { var user1 = new User {ImageUrl = "http://dummy.jpg", Reputation = 10}; var user2 = new User {ImageUrl = "http://dummy.jpg", Reputation = 10};

var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); var viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository));

await viewModel.LoadUser(); await viewModel.LoadUser();

var result = await InvokeAsync(() => ((SolidColorBrush)viewModel.ReputationTrend).Color); Assert.AreEqual(Colors.White, result);}


Solution: Setup/TearDownprivate UserDetailsViewModel _viewModel;

[TestInitialize]public async Task InitilizeUserViewModel() { var user1 = new User { ImageUrl = "http://dummy.jpg", Reputation = 10 }; var user2 = new User { ImageUrl = "http://dummy.jpg", Reputation = 10 };

var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); _viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository));}

[TestMethod]public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { await _viewModel.LoadUser(); await _viewModel.LoadUser();

var result = InvokeAsync(() => ((SolidColorBrush)_viewModel.ReputationTrend).Color); Assert.AreEqual(Colors.White, result);}


Why you shouldn’t use Setup in unit testsprivate UserDetailsViewModel _viewModel;

[TestInitialize]public async Task InitilizeUserViewModel() { var user1 = new User { ImageUrl = "http://dummy.jpg", Reputation = 10 }; var user2 = new User { ImageUrl = "http://dummy.jpg", Reputation = 10 };

var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); _viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository));}

[TestMethod]public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { await _viewModel.LoadUser(); await _viewModel.LoadUser();

var result = InvokeAsync(() => ((SolidColorBrush)_viewModel.ReputationTrend).Color); Assert.AreEqual(Colors.White, result);}


Solution: Extract to methods[TestMethod]public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { var user1 = CreateUser(reputation: 10); var user2 = CreateUser(reputation: 10);

var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); var viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository));


var result = await InvokeAsync(() => ((SolidColorBrush)viewModel.ReputationTrend).Color); Assert.AreEqual(Colors.White, result);}


Over extraction[TestMethod]public async Task LoadUser_ReputationStaysTheSame() { var viewModel = InitializeSystem(10, 10);


CheckColor(Colors.White, viewModel);}


The problem with factoriesprivate User CreateUser(int reputation) { return new User { ImageUrl = "http://dummy.jpg", Reputation = reputation };}

private User CreateUser(int reputation, string imageUrl) { return new User { ImageUrl = imageUrl, Reputation = reputation };}


Builder patternclass UserBuilder{ private int _id, _reputation; private string _displayName, _imageUrl;

public UserBuilder() { _id = 1; _displayName = "dummy"; _imageUrl = "http://dummy.jpg"; }

User Build() { return new User { Id = _id, DisplayName = _displayName, ImageUrl = _imageUrl, Reputation = _reputation }; }}


Builder pattern (cont.)class UserBuilder{ ... public UserBuilder WithName(string displayName) { _displayName = displayName;

return this; }

public UserBuilder WithReputation(int reputation) { _reputation = reputation;

return this; } ...}

var user1 = new UserBuilder() .WithReputation(10) .Build();


Tool: AutoMocking ContainersTest Container SUT

http://blog.ploeh.dk/2013/03/11/auto-mocking-container/

New()

Configure

CreateCreate SUT

Act




Automocking with AutoFixture[Fact]public void YellIfTouchHotIron() { var fixture = new Fixture().Customize(new AutoFakeItEasyCustomization());

Fake<IMouth> fakeMouth = fixture.Freeze<Fake<IMouth>>();

Fake<IHand> fakeHand = fixture.Freeze<Fake<IHand>>(); A.CallTo(() => fakeHand.FakedObject.TouchIron(A<Iron>._)).Throws<BurnException>();

var brain = fixture.Create<Brain>();

brain.TouchIron(new Iron {IsHot = true});

A.CallTo(() => fakeMouth.FakedObject.Yell()).MustHaveHappened();}


Automocking with Typemock Isolator[TestMethod]public void FakeAllDependencies_ChangeBehavior(){ var real = Isolate.Fake.Dependencies<ClassUnderTest>(); var fake = Isolate.GetFake<Dependency>(real); Isolate.WhenCalled(() => fake.Multiplier).WillReturn(2); var result = real.Calculate(1, 2); Assert.AreEqual(6, result);}


Problem: Complex inputs• Big objects• Deep objects• Need for precision• Lack of knowledge


Solution: use trivial inputsSometimes missing the point

Not always enough for “real test”


Solution: SerializationSupported in most programming languages (XML, json).

• Need development testing delayed• Production code change indefinitely delayed


Tool: Export using OzCode


Verification (Assert)


Can you spot the problem?[TestMethod] public void PerformSomeActionReturns42() { var myClass = ... bool initOk = myClass.Initialize();

var result = myClass.PerformSomeAction(); Assert.IsTrue(initOk); Assert.AreEqual(42, result);}

http://stackoverflow.com/q/26400537/11361




Can you spot the problem?[TestMethod] public void TestPasswordComplexity() { var result = _UserManager.ChangePasswordAsync(_TestUser.Id, "Password123!", "1!").Result; //Changes the password. Assert.IsFalse(result.Succeeded);

result = _UserManager.ChangePasswordAsync(_TestUser.Id, "Password123!", "123456789").Result; //Changes the password. Assert.IsFalse(result.Succeeded);

result = _UserManager.ChangePasswordAsync(_TestUser.Id, "Password123!", "123456789!").Result; //Changes the password. Assert.IsFalse(result.Succeeded);

result = _UserManager.ChangePasswordAsync(_TestUser.Id, "Password123!", "abcdefghijk").Result; //Changes the password. Assert.IsFalse(result.Succeeded);

result = _UserManager.ChangePasswordAsync(_TestUser.Id, "Password123!", "abcdefghijK1!").Result; //Changes the password. Assert.IsTrue(result.Succeeded); }





How many Assert(s) per test?

One Assert Per Test!

Two Assert == Two Tests Usually ???

”(…the Code is more) what you'd call guidelines

than actual rules”


Sometimes multiple asserts make sense[TestMethod]public void CompareTwoAsserts(){ var actual = GetNextMessage();

Assert.AreEqual(1, actual.Id); Assert.AreEqual("str-1", actual.Content);}


public class AssertAll{ public static void Execute(params Action[] assertionsToRun) { var errorMessages = new List<exception>(); foreach (var action in assertionsToRun) { try { action.Invoke(); } catch (Exception exc) { errorMessages.Add(exc); } } if(errorMessages.Any()) { var separator = string.Format("{0}{0}", Environment.NewLine); string errorMessage = string.Join(separator, errorMessages); Assert.Fail(string.Format("The following conditions failed:{0}{1}", Environment.NewLine, errorMessage)); } }}

http://blog.drorhelper.com/2011/02/multiple-asserts-done-right.html




Using AssertAll[TestMethod]public void CompareTwoAsserts(){ var actual = CreateMessage();

AssertAll.Execute(() => Assert.AreEqual(1, actual.Id),

() => Assert.AreEqual("str-1", actual.Content);}


Some frameworks are catching up!

https://github.com/nunit/docs/wiki/Multiple-Asserts-Spec




Ever had issues choosing the right Assert?• IsTrue vs. AreEqual• Parameter ordering confusion• StringAssert/CollectionAssert

It’s all about proper error messages


Tool: 3rd party assertion libraries

Better error messagesReadabilityMultiple asserts*

×Additional dependency×Limited UT framework support×System.Object “SPAMED” by extension messages


Shouldly[Fact]public void AddTest(){ var calculator = new Calculator(); var result = calculator.Add(2, 3); Assert.Equal(6, result);}

[Fact]public void AddTest_Shouldly(){ var calculator = new Calculator(); var result = calculator.Add(2, 3); result.ShouldBe(6);}

https://github.com/shouldly/shouldly

Shouldly.ShouldAssertException

result should be6 but was5

Xunit.Sdk.EqualException

Assert.Equal() FailureExpected: 6Actual: 5




Shouldly[Fact]public void GetDivisorsTest(){ var calculator = new Calculator(); var result = calculator.GetDivisors(20); Assert.Equal(new[] {2,3,5,7}, result);}

[Fact]public void GetDivisorsTest_Shouldly(){ var calculator = new Calculator(); var result = calculator.GetDivisors(20); result.ShouldBe(new[] { 2, 3, 5, 7 });}


Shouldly.ShouldAssertException

result should be[2, 3, 5, 7] but was[2, 4, 5, 10] difference[2, *4*, 5, *10*]

Xunit.Sdk.EqualException

Assert.Equal() FailureExpected: Int32[] [2, 3, 5, 7]Actual: WhereEnumerableIterator<Int32> [2, 4, 5, 10]




FluentAssertions[Fact]public void CompareTwoObjects(){ var customer1 = new Customer("cust-1", "John Doe"); var customer2 = new Customer("cust-2", "John Doe");

customer1.ShouldBeEquivalentTo(customer2, o => o.Excluding(customer => customer.Id));}

http://www.fluentassertions.com/




AssertHelper[Test]public void CheckCompare(){ var myClass = new MyClass(); Expect.That(() => myClass.ReturnFive() == 10);} [Test]public void CheckTrue(){ var myClass = new MyClass(); Expect.That(() => myClass.ReturnFalse() == true);}

[Test]public void StringStartsWith() { var s1 = "1234567890"; Expect.That(() => s1.StartsWith("456"));}

[Test]public void CollectionContains(){ var c1 = new[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 }; Expect.That(() => c1.Contains(41));}

https://github.com/dhelper/AssertHelper

https://github.com/dhelper/AssertHelper


Test organization


Test structure issues• What to call the test?• AAA is not mandatory• What should I test?• How to avoid unreadable, complicated tests?

- Unit testing framework provide no structure


The BDD approach

Specifications Step Definitions


Specifications == focused testFeature: Addition

In order to avoid silly mistakesAs a math idiotI want to be told the sum of two numbers

Scenario: Add two numbersGiven I have entered 50 into the calculatorAnd I have entered 70 into the calculatorWhen I press addThen the result should be 120 on the screen


BDD Example: SpecFlow

http://www.specflow.org/




Tool: BDDfy[TestClass]public class CardHasBeenDisabled { private Card _card; private Atm _subject;

void GivenTheCardIsDisabled() {_card = new Card(false, 100);_subject = new Atm(100);

}

void WhenTheAccountHolderRequestsMoney() {_subject.RequestMoney(_card, 20);

}

void ThenTheAtmShouldRetainTheCard() {Assert.IsTrue(_subject.CardIsRetained);

}

void AndTheAtmShouldSayTheCardHasBeenRetained() {Assert.AreEqual(DisplayMessage.CardIsRetained,

_subject.Message); }

[TestMethod] public void Execute() { this.BDDfy(); }}


Test execution


Everybody needs a CI serverUnit tests without a CI server are a waste of time - if you're running all of the tests all of the time locally you're a better man then I am


Tool: Continuous testing

DotCover

Typemock Runner

nCrunch


The right tools will help you write good tests

Arrange

Builder Pattern

AutoMocking Containers

Export

Assert

Shouldly

FluentAssertions

AssertHelper

Test Structure

BDDfy

Continuous Testing

Typemock Runner

DotCover

nCrunch


Thank youDror Helper | @dhelper | http://blog.drorhelper.com



secret unit testing tools

Software