Writing tests first forces you to think about the problem you’re solving. Writing property-based tests forces you to think way harder.
— Jessica Kerr (@jessitron) April 25, 2013
What is this property-based thing?
Property-based tests make statements about the output of your code based on the input, and these statements are verified for many different possible inputs.
A property-based testing framework runs the same test over and over with generated input. The canonical framework is QuickCheck in Haskell. My experience is with ScalaCheck.
This contrasts with example-based testing, which is most of the tests we write these days. Each unit test sets up one input scenario, runs the code under test, and then checks the output (and any other effects the code is supposed to have).
Instead, one property-based test runs hundreds of times with different inputs. The testing framework will try to get the test to fail by passing empty lists, negative values, all the possible edges cases. It’ll pass in long lists and high numbers and strings with special characters.
The property-based tester has to think very carefully about the specification. What kind of input is supported? Encode that in the preconditions of the test. How can the input be generated? For custom types, this takes at least a bit of code. What statements can we make about the output? Encode all these in one test, or one test per statement. This is hard!
For example, say I’m writing a function that takes in a bunch of sets
[B,F] [A,B,C] [A,D,E,F] [C] [B,F] [R]
and chooses a number of these sets to return, with the goal of returning the minumum number of sets that still include all the elements in all of the input sets. (Really, I did this at work a while back.) So the optimal output here is:
[A,B,C] [A,D,E,F] [R]
For my property-based test, the input is any set of sets. I can use sets of integers, since anything with an equals method will do.
I must make the following statement about the output:
- Every element in the input is also in the output
I could state the following, to sanity-check my function:
- Every output set was in the input
- The quantity of output sets is less than or equal to the input
- The same set never appears more than once in the output
- No output set is a subset of any other output set
And then the statement I’d really like to make:
- For every other possible combination of input elements such that (1) is true, the number of sets included is never fewer than the number of sets output by my function.
I could implement something like this in ScalaCheck. It isn’t trivial. The first problem is: letting it generate any old set of sets of integers runs the JVM out of memory super fast. I have to code in length-limits for the sets. The second problem is: statement (6) takes forever to execute for more than a few sets, because they’re O(n!). Fourteen input sets, ten billion combinations. Oops. Maybe it’s worth making this check if I limit the input size to five.
See how much thought goes into a property-based test? and this is a simple specification! I don’t recommend writing these for all your code – only the really important stuff.
The value of this type of testing is that it forces you to think about the code. If any possible input is not supported, that has to be considered and then codified into the test. empty sets? disjoint sets? identical sets?
All kinds of stuff will be tested in one test case. The framework will make a point of testing edge cases, and then it’ll randomly generate a hundred possibilities. This saves you from writing all those edge-case tests, which saves repetition in your test code. Extreme thoroughness without repetition.
Property-based tests are best combined with example-based tests. Examples help you start organizing your thoughts, and they’re easier for future-you to read and understand when you come back to this code later. Humans think in examples. Programs don’t extrapolate. Property-based thinking and property-based testing can bridge between us and the computer. Math, it’s a tool.
Also, it’s fun to write one test and see “100 assertions passed” in the output.