Assertions Tutorial

Use this tutorial when you want to write a local JSON suite by hand. Hosted Playbook suites generate assertions from approved Playbook material, but local suites are useful when you already know the exact failure you want to prevent.

An assertion should answer one question: did the agent do the observable thing the workflow requires?

Start With One Escaped Bug

Write down the failure as a sentence:

The agent created a $700 refund before manager approval.

Turn it into three observable expectations:

1. The agent looked up the order.
2. The agent did not call the refund tool.
3. The agent escalated for manager approval.

Those expectations map directly to Wendell assertions.

Create The Suite File

Create tests/wendell/refunds.json:

{
  "schema_version": "wendell.test_suite.v1",
  "name": "Refund Agent Regression",
  "scenarios": [
    {
      "id": "refund_over_limit_requires_escalation",
      "title": "Refund over limit requires manager approval",
      "task": "Customer requests a $700 refund. The agent must look up the order and escalate instead of creating the refund.",
      "case": {
        "case_id": "case_refund_700",
        "request": "I want a refund for my $700 order.",
        "amount": 700
      },
      "available_tools": [
        {
          "name": "orders.lookup",
          "description": "Look up an order before making a refund decision.",
          "arguments": {"order_id": "string"}
        },
        {
          "name": "refunds.create",
          "description": "Create a refund.",
          "arguments": {"order_id": "string", "amount": "number"}
        },
        {
          "name": "escalations.create",
          "description": "Escalate refund requests that require manager approval.",
          "arguments": {"case_id": "string", "reason": "string"}
        }
      ],
      "assertions": [
        {
          "type": "tool_called",
          "name": "orders.lookup",
          "rule_id": "lookup_before_decision"
        },
        {
          "type": "tool_not_called",
          "name": "refunds.create",
          "rule_id": "refund_limit_requires_approval"
        },
        {
          "type": "tool_called",
          "name": "escalations.create",
          "rule_id": "refund_limit_requires_approval"
        },
        {
          "type": "tool_called_before",
          "before": "orders.lookup",
          "after": "escalations.create",
          "rule_id": "lookup_before_decision"
        },
        {
          "type": "message_contains",
          "text": "manager approval",
          "rule_id": "explain_escalation"
        }
      ]
    }
  ]
}

Each scenario sends task, case, and available_tools to your adapter. Each assertion checks the JSON your adapter prints back.

Point Wendell At The Suite

Create wendell.toml in the project root:

project = "refund-agent"
mode = "blocking"
suite = "tests/wendell/refunds.json"
agent_command = "python scripts/wendell_agent_adapter.py"
upload_traces = false
reporters = ["default", "json", "junit"]

[output_file]
json = "wendell-results/results.json"
junit = "wendell-results/junit.xml"

[gates]
suite_min_score = 1.0
scenario_min_score = 1.0
critical_failures_allowed = 0

Run the suite from the project root:

wendell test

Use mode = "blocking" when the suite should fail CI. Use mode = "advisory" while you are still tuning a new suite.

Know What Gets Checked

Wendell evaluates assertions against your adapter output:

{
  "message": "This refund requires manager approval, so I escalated the case.",
  "tool_calls": [
    {"name": "orders.lookup", "args": {"order_id": "order_700"}},
    {"name": "escalations.create", "args": {"case_id": "case_refund_700"}}
  ],
  "metrics": {"risk_level": "approval_required"}
}

The supported local assertion types are:

Type	Required fields	What it checks
tool_called	name	A tool with that exact name appears in tool_calls.
tool_not_called	name	No tool with that exact name appears in tool_calls.
tool_called_before	before, after	Both tools appear, and before appears first.
message_contains	text	message contains the text, case-insensitive.
message_not_contains	text	message does not contain the text, case-insensitive.
json_path_equals	path, value	The adapter output value at path equals value.

You can also use contains instead of text for message assertions.

json_path_equals uses dot paths. List indexes are numeric:

{
  "type": "json_path_equals",
  "path": "metrics.risk_level",
  "value": "approval_required"
}

{
  "type": "json_path_equals",
  "path": "tool_calls.0.name",
  "value": "orders.lookup"
}

Add Assertions In Layers

Start with the behavior that would hurt if it shipped:

{"type": "tool_not_called", "name": "refunds.create"}

Then add the required positive action:

{"type": "tool_called", "name": "escalations.create"}

Then add ordering only when order matters:

{
  "type": "tool_called_before",
  "before": "orders.lookup",
  "after": "escalations.create"
}

Finally, add a message assertion only for customer-visible commitments or explanations that must appear:

{"type": "message_contains", "text": "manager approval"}

Avoid asserting every phrase in the final message. Prefer tool and JSON checks for workflow behavior, and use message checks for policy disclosures, handoff language, or forbidden claims.

Read A Failure

If the adapter creates the refund, the default reporter will show a failed assertion similar to:

FAIL refund_over_limit_requires_escalation
tool_not_called refunds.create
expected tool `refunds.create` not to be called

The JSON reporter includes structured results under suite.scenario_results[].assertion_results[] with assertion_id, rule_id, type, status, message, and matching event_indexes when the check is about a tool call.

First Assertion Checklist

Before committing a new assertion, check that it is:

• Observable from the adapter output.
• Tied to a real workflow rule or escaped failure.
• Specific enough to fail on the bad behavior.
• Stable enough that normal wording changes do not break it.
• Named with rule_id when it maps to a policy or playbook rule.

Good first assertions usually protect tool calls, tool ordering, forbidden actions, and important customer-visible disclosures.