Vinícius Campos – Vinícius Campos

Property-based testing

Sat, 26 Sep 2026 00:00:00 -0300

When we think about testing we usually remember the classic test pyramid, with unit tests at the bottom and e2e tests at the top, but there is much more around testing and testing strategies than just that.

There are MANY different kinds of tests like load tests, performance tests, regression tests, and there are also property-based tests, but what are property-based tests?

Property-based tests are basic tests that, as the name suggests, are based on a given property, but what’s a property? A property can be seen as a function f(x) -> boolean, so given an input it outputs true or false. For example, imagine that we’re creating a sorting algorithm and we want to validate its implementation with tests. A sorting algorithm has many properties (conditions) that it must meet, but we can list at least two: for ALL consecutive x,y in the sorted list, x <= y, and the other property is that len(sorted(list)) == len(list).

How it works

Property-based testing has 3 main concepts, a generator, a property and a runner. A property is a function that must ALWAYS return true for any input to the code under test. A generator is the code responsible for generating, given a definition, an arbitrary number of inputs (usually 100, but it’s configurable). The runner, as its name suggests, is responsible for validating the generated inputs against the properties defined until one fails - it does more than that, as it then tries to shrink it to the smallest example that breaks the property.

Why use it

The main reason for using it is the way of thinking about tests. Usually we think about arbitrary test inputs for the function, and this has bias. When we start to think about properties we shift our way of thought to try to extract properties inherent to the code that we’re trying to test. So we shift from thinking about which tests to write to writing code that generates tests from probability distributions. More inputs get tested, which makes it more likely to find edge cases before they reach the end users.

Finding properties

The properties can be as custom as you want them to be, but there are some common patterns already, and we can highlight 4:

1. Round-trip

g(f(x)) == x -> common use case is serialization/deserialization

2. Invariant

Something that never changes, the len(list) == len(sorted(list)) for example

3. Idempotency

f(f(x)) == f(x). This property is essential in distributed systems and when we’re handling money operations.

4. Oracle

f(x) == g(x). This is very useful when we are refactoring some code or making changes. The same idea powers migrations in shadow mode.

Examples with Hypothesis

In Python, the most used library for property-based testing is Hypothesis. It plays the generator and the runner roles, so we only need to describe the inputs with strategies and write the property as a regular test. It works with pytest out of the box:

pip install hypothesis pytest

Round-trip

For serialization, we can generate any JSON-like value and check that encoding and decoding gives us the same value back. st.recursive builds nested lists and dictionaries from the basic types:

import json

from hypothesis import given
from hypothesis import strategies as st

json_values = st.recursive(
 st.none() | st.booleans() | st.integers() | st.text(),
 lambda children: st.lists(children) | st.dictionaries(st.text(), children),
)


@given(json_values)
def test_json_round_trip(value):
 assert json.loads(json.dumps(value)) == value

json_values is the generator, the assert is the property, and @given hands it to the runner.

Try adding st.floats() to the base types: Hypothesis will usually find nan, and since nan != nan, the property fails (and json.dumps writes it as NaN, which isn’t even valid JSON). That’s exactly the kind of edge case we wouldn’t pick by hand.

Invariant

Now a sorting function that has a bug: it uses a set under the hood, so it drops duplicated values.

def my_sort(xs):
 return sorted(set(xs)) # bug: drops duplicates


@given(st.lists(st.integers()))
def test_sort_keeps_length(xs):
 assert len(my_sort(xs)) == len(xs)

Running pytest fails with:

E assert 1 == 2
E + where 1 = len([0])
E + where [0] = my_sort([0, 0])
E + and 2 = len([0, 0])
E Failing test case: test_sort_keeps_length(
E xs=[0, 0],
E )

The first list that broke the property was probably something bigger and noisier, but Hypothesis shrank it to [0, 0], the smallest input that shows the bug: two equal values.

The length alone doesn’t prove that a function sorts, though. For that, we check that each element is less than or equal to the next one, and that the result has exactly the same elements as the input:

from collections import Counter


@given(st.lists(st.integers()))
def test_sort_is_ordered(xs):
 result = my_sort(xs)
 assert all(a <= b for a, b in zip(result, result[1:]))
 assert Counter(result) == Counter(xs)

The order check passes, but the Counter check fails on [0, 0] too, since the duplicate is gone. Once my_sort returns sorted(xs), both tests pass.

Idempotency

For a charge, retrying with the same order_id must leave the ledger in the same state as charging once:

def charge(ledger, order_id, amount):
 if order_id not in ledger:
 ledger[order_id] = amount
 return ledger


@given(st.uuids(), st.integers(min_value=1), st.integers(min_value=1, max_value=10))
def test_charge_is_idempotent(order_id, amount, retries):
 once = charge({}, order_id, amount)

 ledger = {}
 for _ in range(retries):
 ledger = charge(ledger, order_id, amount)

 assert ledger == once

Oracle

When rewriting a function, the old implementation becomes the source of truth for the new one. Here the new version looks simpler, but it forgot that the legacy one never returns a negative price:

def legacy_discount(price):
 if price <= 0:
 return 0
 return round(price * 0.9, 2)


def new_discount(price):
 return round(price * 0.9, 2) # forgot the negative case


@given(st.floats(min_value=-1_000, max_value=1_000, allow_nan=False))
def test_new_discount_matches_legacy(price):
 assert new_discount(price) == legacy_discount(price)

E assert -0.9 == 0
E + where -0.9 = new_discount(-1.0)
E + and 0 = legacy_discount(-1.0)
E Failing test case: test_new_discount_matches_legacy(
E price=-1.0,
E )

We didn’t write a single test case with a negative price, and it was found anyway.

Settings and replay

By default, Hypothesis runs 100 examples per test. We can raise it for the properties that matter the most:

from hypothesis import given, settings


@settings(max_examples=500)
@given(st.floats(min_value=-1_000, max_value=1_000, allow_nan=False))
def test_new_discount_matches_legacy(price):
 assert new_discount(price) == legacy_discount(price)

It also saves the failing examples in a local database and replays them first on the next run, so a bug it found once doesn’t disappear on the next random run.

Wrapping up

Property-based tests don’t replace the tests we already have. They add another way of looking at the same code. Instead of asking “which inputs should I test?”, we ask “what must always be true?”, and let the generator explore inputs we wouldn’t think of. To start:

Look for round-trips: anything we encode, we should be able to decode.
Look for invariants: what doesn’t change after the function runs.
Look for idempotency: operations that can be retried, especially with money.
Use the old code as an oracle when refactoring or migrating.
Trust the shrinking: the smallest failing example usually points straight to the bug.

A good first step is picking a function that already has a few example-based tests and asking which property those examples were trying to check.

The dual-write problem

Wed, 23 Sep 2026 00:00:00 -0300

When we build a checkout flow, it’s common to save the order, charge the customer with a payment provider, and then update the order with the result. It looks simple, but it hides a classic distributed systems problem: the dual-write problem.

It consists of having more than one system we write to in a given flow. This could be a DB and a third-party provider, but also other systems on the same cloud infrastructure, like a DB and a cache. Since these systems can’t share a single transaction, there’s no way to guarantee that all writes succeed or none do, so any failure between them can leave us in an inconsistent state.

Going back to our checkout flow, let’s say an API handles it in three steps:

Insert the order in our DB as PENDING.
Charge the customer with the payment provider.
Update the order in our DB to PAID.

The API can die between any of these steps, or the charge can time out. Pick a scenario below to see the end state of each system, and keep the toggle on Naive flow for now.

Pick a scenario to run the flow.

What each failure leaves behind

When all three steps succeed, both systems agree: the order is PAID and the customer was charged once. Let’s start with the worst failure.

When the process dies after step 2, the customer was charged: the money left their account or was charged to their credit card. The business, on the other hand, has no visibility of that payment, and the order stays PENDING. It’s a double bad experience for the customer: they paid, and nothing happens. Worse, they’ll probably try again, and a new order means a new charge.

Dying after step 1 and before step 2 wouldn’t be a problem by itself: the order is PENDING and nobody was charged. But how could we differentiate it from a timeout when calling the provider? It would be essentially the same. From our database, all three failures look identical, so we have to treat every PENDING order as possibly charged.

Making the flow heal itself

Since we can’t tell whether a PENDING order was charged, we don’t try to guess. We make sure every order reaches a final state, PAID or FAILED, by retrying the flow until it gets a definitive answer, and we make those retries safe.

Instead of running the three steps inside the API request, the client generates the order_id (a UUID, for example) when the checkout starts and sends it to the API. The API publishes a message with it to a queue and returns, and a worker consumes the message and runs the three steps. Publishing is a single write, so the API itself has no dual-write problem: if the publish fails, nothing happened, and the client can retry. Since a retry carries the same order_id, it doesn’t create a new order. The whole flow looks like this:

%%{init: {"sequence": {"width": 120, "actorMargin": 24, "mirrorActors": false}}}%%
sequenceDiagram
participant API
participant Queue
participant Worker
participant DB as Orders DB
participant Provider as Payment provider
participant Webhook as Webhook handler
API->>Queue: publish order_id
Queue->>Worker: deliver
Worker->>DB: insert PENDING if absent
Worker->>Provider: charge (key: order_id)
Provider-->>Worker: result
alt succeeded
Worker->>DB: PAID if PENDING
else declined
Worker->>DB: FAILED if PENDING
end
Worker->>Queue: ACK
Provider-)Webhook: payment result
Webhook->>DB: PAID/FAILED if PENDING

A few details make this work:

The message is only ACKed after the flow ends. If the worker crashes, it can’t ACK, so the queue redelivers the message and a worker retries the whole flow.
Every step must be safe to run again. On redelivery the worker starts from step 1: the insert is skipped if the order already exists, and if the order is already PAID or FAILED, the worker just ACKs. The charge uses the order_id as the idempotency key, so if the provider already saw that key, it returns the result of the first request instead of charging the customer again.
Definitive answers end the flow. If the charge succeeds, the worker marks the order as PAID; if it’s declined, as FAILED. Only unknown outcomes, like a crash, a timeout, or a 5xx, are retried.
The webhook is a safety net. It arrives at any time after the charge and covers the window where the customer was charged but the order is still PENDING.
The order_id also goes in the charge metadata. The webhook only carries the idempotency key when our request triggered the event directly, while the metadata travels with the payment itself, so the webhook handler can always find the order.
Updates are conditional. The worker and the webhook can race, so both only move the order forward if it’s still PENDING. Whoever arrives second does nothing.

Now switch the widget above to Self-healing flow and replay the failure scenarios to see each one heal.

Once retries are exhausted, the message goes to a dead-letter queue (DLQ). While it sits there, the webhook still fixes the order if the customer was charged, and since the flow is idempotent, it’s safe to reprocess the message later. What’s left is monitoring: alert on the DLQ, and redrive before its retention expires, otherwise the message and the record of the pending work disappear.

Redriving has one catch: idempotency keys also expire at the provider (Stripe can remove them once they’re 24 hours old). If the message is older than that, the retry could create a second charge. So before processing an old message, search the provider by the order_id in the metadata (Stripe’s Search API supports metadata['order_id']), and only charge if there’s no successful payment for it; if there is one, mark the order as PAID. Search results aren’t real-time, which is fine here, since by then the payment is long indexed.

Wrapping up

Every PENDING order is possibly charged, so the flow has to keep going until it gets a definitive answer. What makes that safe:

ACK only after the flow ends.
Use the same order_id everywhere: generated by the client, as the idempotency key, and in the metadata.
Retry unknown outcomes, and end the flow on definitive ones.
Use the webhook as a safety net.
Only move an order forward if it’s still PENDING.
Monitor the DLQ.

This works because the API makes a single write: it only publishes a message. If your API needs to write to the DB and publish a message, look at the transactional outbox pattern. The outbox also delivers messages at least once, so the consumer still has to be idempotent.

Whenever we do one or more write operations on our system, having idempotency is a must in order to guarantee replayability, and help us out on doing reconciliation without needing to infer anything.

Devcontainer

Sun, 25 Jan 2026 00:00:00 -0300

In the past we’d have the problem of artifact mismatch between development and production. Containerization mitigated this, because we could build a Docker image once and promote it from environment to environment. The code would run as expected regardless of the host, since it lived inside the container.

Even so, we still develop and run the solution locally on our main machines, and that brings issues like incompatibility. A global library version can conflict with the one required by the project. It also adds another attack surface, because we might run malicious code without noticing it if a dependency gets poisoned (supply chain attack).

There are solutions that help mitigate this, and one of them is using a devcontainer. With it, we not only create the final image of the project, but we keep the whole development cycle inside a container. That isolates it from our OS, avoids unwanted incompatibilities, and prevents it from accessing sensitive data on our machine, like credentials.

This matters even more now with AI coding assistants like Codex, Claude Code, and OpenCode. They could potentially access our root repositories (especially if we run in yolo mode) and create catastrophic scenarios, like deleting important files and folders.

As a simple example, we can create a devcontainer in the repository and add OpenCode to it, so the AI agent won’t have access to anything outside the repository folder, since the mounted volume contains only the repository files. It’s important to note that the Docker container still has access to the network, so if we want to make it 100% isolated we should restrict that as well, but that’s not our case.

To configure it, create a .devcontainer/ folder in the repository root directory and add the devcontainer.json and Dockerfile files to it.

The devcontainer.json specifies the name of the sandbox devcontainer environment, how it can be built, the user, and the workspaceFolder, which guarantees it will only have access to this mounted volume, not to all OS folders. Here’s a minimal example.

{
 "name": "opencode-sandbox-alpine",
 "build": { "dockerfile": "Dockerfile" },
 "remoteUser": "dev",
 "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
 "runArgs": [
 "--cap-drop=ALL",
 "--security-opt=no-new-privileges",
 "--pids-limit=256"
 ]
}

The Dockerfile just represents the development environment, so it must install the necessary libraries to do development, including any additional AI coding agent like Claude Code, Codex, and OpenCode.

FROM alpine:3.20

# Basics + Node + npm
RUN apk add --no-cache bash ca-certificates git nodejs npm

# Install opencode (adjust package name if yours differs)
RUN npm i -g opencode-ai

# Non-root user
RUN adduser -D -u 1000 dev
USER dev

WORKDIR /workspaces
CMD ["bash"]

Next, install the devcontainers/cli in our system and use it to build and enter the development container as shown below.

npm i -g @devcontainers/cli
devcontainer up --workspace-folder .
devcontainer exec --workspace-folder . bash

Once it’s running, you’ll have a simple development environment where you can make all the changes safely inside a container that doesn’t have access to other files, avoiding the risk of corrupting any OS related files, for example.