Understanding Property-Based Testing: Beyond Simple Abstractions

Property-Based Testing (PBT) is often discussed in abstract terms, focusing on properties that define correctness and generators that define the domain. However, the simplicity of this definition doesn't capture the complexity of implementing PBT frameworks. A property is a universally quantified computation that must hold for all inputs, often modeled as a boolean function in programming languages. For example, the property \l -> reverse (reverse l) == l asserts that double reversal returns the original list.

Preconditions complicate this model, as they determine if an input is valid. The implication operator (==>) is used to handle preconditions, ensuring properties are only tested when conditions are met. For instance, in a database context, a precondition might check if a table exists before testing if inserted values can be selected back. Generators play a crucial role in PBT by creating random inputs, but they must be dependent on the system under test to ensure meaningful results.

A dependent generator can create valid inputs by considering existing data, like generating table names from a list of tables in a database. This approach avoids precondition failures by ensuring inputs are valid by construction. The conceptual API for running tests involves generating inputs, checking properties, and handling precondition failures.

QuickCheck, a popular PBT tool, supports this style by allowing properties to be written with generators inline, using forAll combinators. This method integrates generation and assertion, simplifying the process. While not revolutionary, this approach highlights the need for PBT frameworks to consider the interplay between properties and generators.

The article critiques existing PBT libraries like QuickCheck and Proptest for their abstraction boundaries, which may not accommodate the interleaving of generation and computation. It suggests that libraries should allow for more expressive capabilities, as seen in newer tools like Hegel, which integrates generation with test cases. This exploration aims to refine the expressiveness of PBT libraries, inviting further discussion on the topic.