0

Property based tests have been added to Bitcoin Core.

This Dev article describes property based tests as randomized unit tests that in some cases could replace unit tests.

"Generally speaking, property-based tests require only a few lines of code (like unit tests), but unlike unit tests they test a different set of inputs each time. Because of this, you end up covering more domain space with roughly the same amount of test code."

Is this a good summary? Could we phase out some of the unit tests in Bitcoin Core and replace them with property based tests?

1

I think this is a fair summary. I would hesitate to phase out unit tests, rather supplement them with property based tests. In a library I maintain called bitcoin-s we use them in tandem.

Concepts

There are two primary concepts in property based testing systems.

The first one is a Generator. The point of a generator is to generate values you want to run through your tests. Think of a generator as specify the boundaries of what random values you want to be generated for your test.

The second concept is the test itself. In english, property based tests say that

"Given a generator X, i want to generate random values from X and run them through my test case. My assertion should always hold true for the value generated from my generator".

An example

I'm going to pull an example from this PR i've had open for a long time now. If you want to see some more extensive tests I recommend taking a lookg at it.

The "property" we are going to assert is that

"If we generate two random private keys in bitcoin, they should NOT be the same"

If this property fails, we have an indicatin that something is probably wrong with our pseudo random number generator.

Here is what that test case looks like in C++ with bitcoin core:

/** Check CKey uniqueness */
RC_BOOST_PROP(key_uniqueness, (const CKey& key1, const CKey& key2))
{
    RC_ASSERT(!(key1 == key2));
}

That test case can be run N number of times (by default 10 in the rapidcheck library.

The next logical question would be "where do key1 and key2 come from"? The answer: They come from the generator! The beauty of property based testing is leverage the framework to do the generation of values for you.

Here is what the generator for CKey looks like:

/** Generator for a new CKey */
template <>
struct Arbitrary<CKey> {
    static Gen<CKey> arbitrary()
    {
        return rc::gen::map<int>([](int x) {
            CKey key;
            key.MakeNewKey(true);
            return key;
        });
    };
};

This gives rapidcheck the ability to generate arbitrary valid CKeys to pass in as a parameter to the properties that we are asserting holds true.

You can see more complex generators and assertions related to Script in that PR too if you want to see non trivial generators/assertions.

Summary

It's really useful to have multiple runs of a test case with newly generated values every time. Typically in software testing, you write a unit test asserting that your code works as expected for one or two values. This is a really limited approach to testing as you want your code to work for a range of values. With generators, you can get the computer to generate a range of valid values that your property should assert true on.

I think this is particularly useful for consensus systems like bitcoin where we need to reason about a wide range of values being fed as input into the consensus system. Property based testing is a great tool to use to offload some of that cognitive load when reasoning about corner cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.