Craftsmanship in software strives for high, predictable, repeatable and affordable quality. This is the reason why we care about design, code reviews and of course testing. Test Driven Development helps us create a flexible implementation that meets the expectations of our clients in terms of features, performance and quality. As professionals we have to master as many tools as possible to reach our goal of steadily providing value.
The above is the reason why I am exploring Property Based Testing. I like to think about it as "Let the machine do the heavy lifting for you" or as John Huges says "Don't write tests, generate them".
I am not claiming that you should stop writing unit tests. I find that when driving the design Outside-in hence declaring the relations between my components, unit tests are the tool to use. Sometimes when I want to discover deeper properties of my domain, I am faced with features that have more edge cases than my brain could handle. In such cases, generating a wide array of inputs to test my program is appealing. This is where I think property testing becomes a viable tool.
If you are asking "How can I set up a development environment in .NET that would allow me to experiment and work with this tool?" the following tutorial is for you.
Comment out the content in FsCheckAddin.fs file to allow NCrunch to run the tests
//[<NUnitAddin(Description = "FsCheck addin")>] //type FsCheckAddin() = // interface IAddin with // override x.Install host = // let tcBuilder = new FsCheckTestCaseBuilder() // host.GetExtensionPoint("TestCaseBuilders").Install(tcBuilder) // true
In your test file add the following
module Properties = open NUnit.Framework open FsCheck
Add this test to enable NCrunch
//Needed to enable NCrunch [<Ignore>][<Test>] let ignoreMe() = ()
Create a static class whose methods are the properties you want to test. For this example I'll test an incorrect implementation of a list - the reverse of a list is always equal to the original.
type ListProperties = // Note: should fail static member reverseIsAsTheOriginal (xs:int list) = List.rev xs = xs
Add the following test to verify the property defined in the
[<Test>] let verifyAll () = Check.QuickThrowOnFailureAll<ListProperties>()
Enable NCrunch following NCRUNCH > Enable NCrunch from the main menu and let the engine catch the error
You should read a message similar to
--- Checking ListProperties --- System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ----> System.Exception : ListProperties.ReverseOfReverseIsAsTheOriginal-Falsifiable, after 6 tests (3 shrinks) (StdGen (1165808905,296086422)): Original: [-1; 2; -2] Shrunk: [1; 0]
This shows that the property was falsifiable (as expected) but let's try to gather more informations to address the issue
Change the test so that it provides a verbose output when failing
[<Test>] let verifyAll () = Check.VerboseThrowOnFailureAll<ListProperties>()
Re-run the tests through Resharper Runner and read the output
--- Checking ListProperties --- 0: [-2] 1: [-2; -2] 2:  3: [1; 1] 4: [-2] 5: [-3; 1; 5; -3; 6; 2; 6; -2] shrink: [1; 5; -3; 6; 2; 6; -2] shrink: [5; -3; 6; 2; 6; -2] shrink: [-3; 6; 2; 6; -2] shrink: [6; 2; 6; -2] shrink: [2; 6; -2] shrink: [6; -2] shrink: [6; 2] shrink: [6; 0] shrink: [3; 0] shrink: [2; 0] shrink: [1; 0] System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ----> System.Exception : ListProperties.ReverseOfReverseIsAsTheOriginal-Falsifiable, after 6 tests (11 shrinks) (StdGen (1681135586,296086423)): Original: [-3; 1; 5; -3; 6; 2; 6; -2] Shrunk: [1; 0]
The output shows the "shrinking process" from the sixth input (identified by 5:) original value
[-3; 1; 5; -3; 6; 2; 6; -2] to the "shrunk" (reduced) input
[1; 0] that still falsifies (makes it fail) the property
Let's correct the property as the reverse of the reverse of a list is equal to the original
type ListProperties = static member ReverseOfReverseIsAsTheOriginal (xs:int list) = List.rev (List.rev xs) = xs
Re-run the tests that should be all green by now and read the output. You'll notice it's very verbose as expected and shows all the inputs used to verify the property.
Software is our passion.
We are software craftspeople. We build well-crafted software for our clients, we help developers to get better at their craft through training, coaching and mentoring, and we help companies get better at delivering software.