Write deterministic tests with LLMReplay

This tutorial shows how to add a Tier 3a test — a test that exercises an LLM-dependent OS path deterministically, without a live LLM and without hand-written mocks.

By the end you will have:

• a recorded fixture (.jsonl file)
• a passing test that runs identically in CI every time
• a drift detection test that fails loudly when the prompt changes

Full policy rationale is in docs/deep-dives/contributing/testing.md. This page focuses on the mechanics.

---

Why LLMReplay?

LLM calls are non-deterministic. Two calls with the same input can return different text. That makes tests that call a real LLM unsuitable for CI: they are slow, require credentials, and fail intermittently.

The standard workaround is a mock:

# FORBIDDEN — do not do this
from unittest.mock import patch
with patch("litellm.acompletion", return_value={"choices": [...]}):
    ...

Mocks bypass the real API surface. When litellm changes its signature or response shape, the mock continues to return the hand-crafted dict, the test passes, and the breakage reaches production silently.

LLMReplay is a Fake: it patches litellm.acompletion at the same boundary, but reconstructs a real litellm.ModelResponse from a previously recorded response. Signature drift is caught at invocation (TypeError, AttributeError) rather than in production.

---

Tier model — quick summary

Tier	What it pins	Replay involved?
1 — Contract	External boundaries (yaml schema, events payload, DSL contracts)	No
2 — OS invariant	P1–P8 invariants and subsystem contracts	No (or stub callable)
3 — LLM-replay	Behavior of LLM-dependent paths via recorded fixtures	Yes (LLMReplay)
4 — Don't write	Everything else	—

This tutorial covers Tier 3a: a single LLM call per test. The LLM tool-call path is reached through call_llm_tools (the tool_use wrapper the chat router drives). If you are not testing an LLM-dependent path, check the decision flow in testing.md first.

---

Step 1: Record the fixture

Prerequisites

• LiteLLM proxy running at localhost:4000 (see project_local_env.md in memory for the exact setup)
• The model you want to record is available via the proxy

Write the test first (without a fixture)

Create your test file. Mark the test with @pytest.mark.replay, passing a fixture path relative to tests/, and drive the real call_llm_tools boundary. There is no frame or context object to build — you pass the plain messages and tools dicts that go to litellm:

# tests/test_replay_my_area.py

import pytest

from reyn.llm.llm import call_llm_tools

MODEL = "gemini-2.5-flash-lite"   # bare name — the proxy strips any prefix on
                                  # record, so the replay key matches
MESSAGES = [{"role": "user", "content": "hi"}]
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "run_skill",
            "description": "run a skill",
            "parameters": {"type": "object", "properties": {}},
        },
    }
]


@pytest.mark.replay("fixtures/llm/my_area/text_only.jsonl")
@pytest.mark.asyncio
async def test_my_call_returns_text_when_no_tool_calls():
    """Tier 3a: when the LLM returns plain text, result.content reflects it."""
    result = await call_llm_tools(model=MODEL, messages=MESSAGES, tools=TOOLS)

    assert isinstance(result.content, str)
    assert len(result.content) > 0
    assert result.tool_calls == []

Run once to record

When the fixture file does not exist, conftest.py switches to record mode automatically:

python -m pytest tests/test_replay_my_area.py::test_my_call_returns_text_when_no_tool_calls -v

This calls the real LLM, records the response, and writes tests/fixtures/llm/my_area/text_only.jsonl. The test also passes on this first run.

After recording, run the test again without a live LLM to confirm it replays correctly:

# Stop the proxy, then:
python -m pytest tests/test_replay_my_area.py::test_my_call_returns_text_when_no_tool_calls -v

Commit the fixture file alongside the test.

---

Step 2: Write the test body

Keep the inputs deterministic

The fixture key is the SHA-256 of the (model, messages, tools, tool_choice) that call_llm_tools sends to litellm — the scenario. Every byte of those inputs contributes to the key EXCEPT the parts a registered environment precondition declares as environment-derived (#3473; see below). Two consequences:

• Build messages and tools from stable, literal values. Do not inject volatile data (timestamps, UUIDs, datetime.now(), random ids) into the prompt — the key would change every run and no fixture would ever match.
• Because the key is a pure function of the scenario, two tests that pass the same (model, messages, tools, tool_choice) share one fixture file. Reuse a fixture across tests that exercise the same call.

Environment-derived inputs are preconditions, not key components

Some of what reaches the wire describes the MACHINE, not the conversation. The canonical case is the MCP tool catalog: RouterHostAdapter probes each configured MCP server with a deadline and the answer becomes the server / mcp_tool_name enums of the MCP tool schemas, so under load the same conversation can send a different tools= payload.

While such a value is hashed into the key, an environment wobble is indistinguishable from a different conversation — the fixture "goes missing" and the report cannot say why. So reyn.dev.testing.replay_preconditions takes it out of the key and, instead:

• records the value next to each fixture entry at capture time and checks it at replay time, failing with PreconditionMismatch that NAMES the difference (never the bare "no fixture entry");
• captures a snapshot of the live environment into the fixture and injects it before replay, so the run's environment is the fixture's environment by construction. Injection is a direct write, never a sleep / longer deadline / retry — those only widen the window in which the environment happens to come out right.

You get this for free: LLMReplay applies default_preconditions() unless told otherwise. When a NEW environment-derived value starts reaching the payload (another dynamic catalog, a model list, a feature-flag-driven tool set), implement EnvironmentPrecondition and add it to default_preconditions() — the fixture format, the mismatch report and the injection step already carry it.

Assert on structure, not wording

call_llm_tools returns a result with a .content string and a normalized .tool_calls list (plain dicts, not litellm internals). Good assertions check that shape, not the model's free text:

@pytest.mark.replay("fixtures/llm/my_area/tool_call.jsonl")
@pytest.mark.asyncio
async def test_tool_calls_are_normalized():
    """Tier 3a: tool_calls are normalized to plain dicts."""
    messages = [{"role": "user", "content": "call the run_skill tool with skill=hello"}]
    result = await call_llm_tools(
        model=MODEL, messages=messages, tools=TOOLS, tool_choice="required"
    )

    assert len(result.tool_calls) >= 1
    tc = result.tool_calls[0]
    assert isinstance(tc, dict)                       # not a litellm object
    assert tc["type"] == "function"
    assert isinstance(tc["function"]["arguments"], str)   # JSON string, not dict

Avoid asserting on free-text .content wording unless the exact content is the contract you are pinning. Wording varies across model versions and causes unnecessary re-records.

---

Step 3: Add a drift detection test

LLMReplay raises MissingFixture in replay mode when the call's key matches no fixture entry. That is the mechanism that catches accidental prompt drift — if someone changes the messages or the SET OF TOOLS the OS sends, the key changes, no fixture matches, and the test fails loudly rather than passing on stale data. (The MCP tool catalog — which servers answered and with what tools — is the one part of tools that is NOT in the key; a change there surfaces as PreconditionMismatch instead. See Environment-derived inputs are preconditions, not key components.)

from reyn.dev.testing.replay import MissingFixture


@pytest.mark.replay("fixtures/llm/my_area/text_only.jsonl")
@pytest.mark.asyncio
async def test_drift_detection_raises_missing_fixture():
    """Tier 3a drift detection: input changes must be reflected in re-recorded fixtures."""
    drift_messages = [{"role": "user", "content": "not in the fixture — drift sentinel"}]
    with pytest.raises(MissingFixture):
        await call_llm_tools(model=MODEL, messages=drift_messages, tools=TOOLS)

The drift test re-uses the same fixture file as the happy path. Different messages produce a different SHA-256 key, which is not in the fixture, so MissingFixture is raised.

The raised message attributes itself across all four key components (#3473): it says which of model / messages / tools / tool_choice differs, which message index, and which tool name — so an unexpected drift can be read straight off the failure instead of bisected. Entries recorded before #3473 carry no fingerprint; the report says attribution is unavailable rather than claiming everything matched.

---

Step 4: Update fixtures after intentional input changes

When you intentionally change the messages or the set of tools, the fixture key changes and replay mode raises MissingFixture. This is expected and correct. An intentional change to the MCP catalog instead raises PreconditionMismatch; re-recording is the same fix, and it also refreshes the captured snapshot.

Re-record with REYN_LLM_RECORD=1 — no manual delete required (#3634):

REYN_LLM_RECORD=1 python -m pytest tests/test_replay_my_area.py -v

LLMReplay.flush() replaces, not appends: an entry this run re-recorded (same call, unchanged) or superseded (same call, changed tool schema) is dropped from the on-disk copy before the fresh one is written, and any OTHER entry — e.g. one recorded by a sibling test that shares this same fixture file — is left untouched. Before #3634, flush() only ever appended, so regenerating in place after a schema change (even a description-only edit) left the OLD entry on disk alongside the new one: the fixture then matched BOTH the old and the new schema and stayed green regardless of which one the code implements — worse than a stale fixture, which at least goes RED. Delete the fixture first if you prefer an explicit "starting from nothing," but #3662 removed the missing-file-implies-record-mode fallback: a deleted fixture now makes the test fail LOUD at fixture setup (before LLMReplay.install() runs) with the exact REYN_LLM_RECORD=1 command to re-run, rather than silently switching modes on your behalf — you still need REYN_LLM_RECORD=1 explicitly, delete or no delete:

rm tests/fixtures/llm/my_area/text_only.jsonl
REYN_LLM_RECORD=1 python -m pytest tests/test_replay_my_area.py -v
# without REYN_LLM_RECORD=1 this now fails loud instead of auto-recording

Commit the new fixture alongside the change. Reviewers can diff the prompt_preview field in the JSONL to see what changed. tests/dev/test_replay_fixture_no_stacking_3634.py is a CI gate that fails if any committed fixture ever holds the same logical call under more than one key.

Warning — -k filtered runs exclude replay tests. If your local test run uses -k some_keyword that does not match replay test names, replay tests are silently skipped and the run appears green. This masks broken fixtures until CI runs the full suite. Always run the full replay suite after any change to the set of tools or the LLM-call boundaries.

---

NEVER rules for replay tests

These are absolute. Violations are rejected in PR review.

Do not mock with MagicMock / AsyncMock / patch.

# FORBIDDEN
from unittest.mock import patch, AsyncMock
with patch("litellm.acompletion", new_callable=AsyncMock) as mock_llm:
    mock_llm.return_value = {"choices": [...]}
    ...

Use @pytest.mark.replay with a real recorded fixture instead.

Do not assert on private state.

# FORBIDDEN — Tier 4
assert runtime._last_llm_response["id"] == "chatcmpl-abc"

Assert on the public return value of the function under test (result.content, result.tool_calls).

Do not inject volatile values into the prompt.

# FORBIDDEN — breaks fixture keys (key changes every run)
messages = [{"role": "user", "content": f"now is {datetime.now()}"}]

Build messages and tools from stable literals so the SHA-256 key is reproducible.

Do not write Tier 4 tests. Common Tier 4 traps:

• Asserting that .content contains a specific sentence — this pins LLM output wording, which drifts with every model update.
• Testing internal cache state or flag values (_state_loaded, _initialized).
• Adding a test for "this specific bug we fixed" unless it represents a genuine P1–P8 invariant.

---

Coverage checklist for a new LLM-dependent path

When adding a new LLM-dependent OS path, verify:

• [ ] One Tier 3a test for the canonical happy path
• [ ] One Tier 3a test for a boundary or error case (optional but recommended)
• [ ] One drift detection test (MissingFixture assertion) per fixture file
• [ ] messages / tools built from stable literals (no volatile values)
• [ ] Each test docstring starts with """Tier 3a: ...
• [ ] Fixture file committed alongside the test
• [ ] No MagicMock / AsyncMock / patch anywhere in the file
• [ ] If the path also derives from a P1–P8 invariant, add a Tier 2 test for the invariant separately

---

Reference

• Full LLMReplay API: docs/reference/testing/replay.md
• Full testing policy (Tier definitions, NEVER rules, decision flow): docs/deep-dives/contributing/testing.md
• Live examples in the codebase:
• tests/llm/test_llm_tools.py — the canonical @pytest.mark.replay tests for call_llm_tools
• tests/conftest.py — the _llm_replay autouse fixture and record/replay mode resolution