Property-Based Testing
Property-based testing checks general rules about software against many generated inputs. When a rule fails, the testing library searches for a smaller counterexample that makes the defect easier to understand and reproduce.
itSoftware engineering | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Intro
Property-Based Testing
Property-based testing checks a general statement about software against many generated examples. You describe a property that should hold over a range of valid inputs. A testing library generates inputs, runs the property, and reports a counterexample when the statement is false.
An example-based test starts with a value chosen by a person. It might assert that sorting [3, 1, 2] returns [1, 2, 3]. A property-based test starts with a relationship. It can assert that sorting any generated list preserves its length, produces an ordered result, and preserves the same elements. The two styles complement each other. Named examples document important cases, while generated examples explore variations that were not written by hand.
The execution pipeline
A property-based run connects five parts:
- Property: an executable statement about behavior.
- Generator: a description of the input space and how values are produced.
- Runner: repeated execution of the property with generated values.
- Shrinker: a search for a smaller input that still fails.
- Failure report: the reduced counterexample and information needed to reproduce it.
The runner asks each generator for a value. It passes that value to the property. A passing result leads to another generated value. A failing result starts shrinking. The shrinker proposes related values that are smaller according to the library's ordering. It keeps a candidate only when the property still fails. The final report contains a reduced counterexample rather than the first, potentially large failure.
Generation and shrinking form one design problem. A generator must produce valid and useful data. Its shrinking behavior must preserve constraints that matter to the property. Filtering a broad generator can spend most runs rejecting invalid values. It can also make a failure hard to shrink. Constructing valid values directly usually gives the runner more useful executions and the shrinker a clearer route toward a small counterexample.
What makes a useful property
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://research.chalmers.se/en/publication/237427
Supports
- Definition of properties as executable functions checked against generated input
- Custom generators, distribution observation, and limitations of random testing
- September 2000 QuickCheck timeline milestone
- Quiz answers about the defining property-based structure
- https://hackage.haskell.org/package/QuickCheck
Supports
- Current QuickCheck generator, property, distribution, and monadic-testing capabilities
- Open-source licensing and free availability for Landscape placement
- https://hypothesis.works/articles/what-is-property-based-testing/
Supports
- Distinction between essential property-based testing behavior and library-specific mechanisms
- Relationship between property-based testing and fuzzing
- Property patterns and quiz answer about idempotence
- https://hypothesis.readthedocs.io/en/latest/reference/strategies.html
Supports
- Strategy composition, dependent data, recursive generation, shrinking, and reproducible pseudorandom state
- Generator design and reproduction guidance
- Quiz answers about recursive generators and replay
- https://hypothesis.readthedocs.io/en/latest/reference/api.html
Supports
- Explicit, replay, generate, target, shrink, and explain phases
- Minimal failing examples and replay behavior
- https://hypothesis.readthedocs.io/en/latest/stateful.html
Supports
- Rules, bundles, state-machine action sequences, and reduced failing programs
- Stateful architecture and quiz answer
- https://hypothesis.readthedocs.io/en/latest/compatibility.html
Supports
- Requirement for deterministic outcomes and deterministic data generation
- Quiz answer about nondeterminism during shrinking
- https://hypothesis.works/articles/how-hypothesis-works/
Supports
- Unified byte-stream representation for generation, replay, and shrinking
- Hypothesis 3.0 engine timeline milestone in 2016
- https://fast-check.dev/docs/introduction/what-is-property-based-testing/
Supports
- Differences between example and property tests
- Random generation, multiple runs, and counterexample shrinking
- Test-runner integration and quiz answers about the core pipeline
- https://fscheck.github.io/FsCheck/TestData.html
Supports
- Generator, shrinker, and Arbitrary relationships
- Constructive generation, type-derived data, and shrinking behavior
- FsCheck Landscape placement and quiz answer about shrinking
- https://fscheck.github.io/FsCheck/Properties.html
Supports
- Preconditions, quantified properties, custom generators, and avoiding filtered cases
- Quiz answers about direct generation and discard rates
- https://jqwik.net/docs/current/user-guide
Supports
- Properties, arbitraries, assumptions, shrinking, edge cases, statistics, and stateful actions
- Generator distribution and discard diagnostics
- jqwik Landscape placement and quiz answers about valid domains
- https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf
Supports
- Systematic patterns for writing independent properties of pure functions
- Model, invariant, and metamorphic property design
- Quiz answers about circular oracles and differential disagreement
- https://proper-testing.github.io/
Supports
- Erlang property generation, targeted testing, and model-based use
- PropEr Landscape placement
- https://pure.york.ac.uk/portal/en/publications/smallcheck-and-lazy-smallcheck-automatic-exhaustive-testing-for-s-2/
Supports
- Bounded exhaustive generation as an alternative to random sampling
- 2008 SmallCheck timeline milestone
- https://proper-testing.github.io/publications.html
Supports
- September 2011 PropEr types and specifications publication
- PropEr timeline milestone
- https://arxiv.org/abs/1409.0393
Supports
- Property-based design of information-flow abstract machines and counterexamples for seeded defects
- September 2014 timeline milestone
- https://proper-testing.github.io/papers/issta2017.pdf
Supports
- Search objectives and optimization in targeted property-based testing
- Generator-distribution guidance and July 2017 timeline milestone
- https://arxiv.org/abs/1808.01520
Supports
- Branching-process analysis of recursive generator constructor counts
- August 2018 timeline milestone
- https://joss.theoj.org/papers/10.21105/joss.01891
Supports
- Hypothesis generation and test-case-reduction architecture
- November 2019 timeline milestone
- https://arxiv.org/abs/2404.16062
Supports
- Parallel property execution and deterministic versus greedy parallel shrinking
- April 2024 timeline milestone
- https://github.com/sindresorhus/awesome
Supports
- Discovery path to the Awesome Testing list
- https://github.com/TheJambo/awesome-testing
Supports
- Scope decision that the general list excludes unit-test resources and requires a more specific list
- https://github.com/cleder/awesome-python-testing
Supports
- Discovery of Atheris, HypoFuzz, hypothesis-crosshair, and icontract-hypothesis
- https://pypi.org/project/atheris/
Supports
- Coverage-guided fuzzing of Python and CPython extensions
- Atheris Awesome Links rationale
- https://hypofuzz.com/docs/
Supports
- Adaptive coverage-guided execution of Hypothesis tests and shared failure database
- HypoFuzz Awesome Links rationale
- https://hypothesis.readthedocs.io/en/latest/extensions.html
Supports
- CrossHair solver backend, HypoFuzz backend, failure database, and normal shrinking integration
- Hypothesis CrossHair Awesome Links rationale
- https://pypi.org/project/icontract-hypothesis/
Supports
- Strategy inference from preconditions and generated test tooling
- icontract-hypothesis Awesome Links rationale
- https://hypothesis.readthedocs.io/
Supports
- Python strategy, example-database, shrinking, and stateful-testing capabilities
- Open-source licensing and free availability for Landscape placement
- https://fast-check.dev/
Supports
- JavaScript and TypeScript arbitraries, shrinking, and test-runner integration
- Open-source licensing and free availability for Landscape placement
- https://scalacheck.org/
Supports
- Scala and Java properties, generated parameters, standalone operation, and framework integration
- ScalaCheck Landscape placement
- https://fscheck.github.io/FsCheck/
Supports
- F sharp and C sharp properties, generators, shrinkers, and framework integration
- BSD licensing and free availability for Landscape placement
- https://github.com/hedgehogqa/haskell-hedgehog
Supports
- Integrated generation and shrinking for composed Haskell generators
- Open-source licensing and free availability for Landscape placement
- https://proptest-rs.github.io/proptest/
Supports
- Per-value Rust strategies, shrinking, failure persistence, recursive data, and state-machine support
- Open-source licensing and free availability for Landscape placement