refund agent v2

4 cases on refund_cases, 3 runs each.

Gate failed: 1 rule broken

Compared with refund agent v1 on 4 paired cases: pass rate -100.0 pts (95% CI -100.0 to -100.0), 4 broke and 0 fixed (McNemar p = 0.125).

must-pass cases failed: refund-damaged
4 cases broke and 0 fixed; not significant (p=0.1250)

Pass rate

0.0%

▼100.0 pts vs baseline

pass^3

0.0%

▼100.0 pts vs baseline

Held back when it should

0.0%

▼100.0 pts vs baseline

Answered when it should

100.0%

no change vs baseline

Flaky cases

passed some runs, failed others

p95 latency

0.1 ms

▲0.1 ms vs baseline

Case map

One square per case. Select one to open it

passed every runfailed every runflakybroke vs baseline

Needs attention

Failing and flaky cases, regressions first

refund-damagedoutcome: expected escalate, got answerbrokemust pass0/3 runs refund-in-windowtool_order: wanted order lookup_order > check_policy > issue_refund, got lookup_order > issue_refundbroke0/3 runs refund-expiredanswer: expected contains '30-day'broke0/3 runs no-order-numberoutcome: expected abstain, got answerbroke0/3 runs

Checks

Share of runs where each check passed

answer

50.0%

max_tool_calls

100.0%

outcome

50.0%

ran

100.0%

tool_args

100.0%

tool_order

0.0%

tools_forbidden

33.3%

tools_required

0.0%

this runbaseline

By tag

Mean case pass rate per tag

missing-info

0.0%

refunds

0.0%

uncovered

0.0%

this runbaseline

Cases

Each case with its checks and the full trace of every run.

4 casesruns

refund-in-window

tool_order: wanted order lookup_order > check_policy > issue_refund, got lookup_order > issue_refund

refund-expired

answer: expected contains '30-day'

refund-damaged

outcome: expected escalate, got answer

no-order-number

outcome: expected abstain, got answer

All cases

refund-in-window

refundsbroke vs baseline

0/3

runs passed

Input

Can I return order A100?

Expected

{
  "outcome": "answer",
  "answer": "refund",
  "tool_order": [
    "lookup_order",
    "check_policy",
    "issue_refund"
  ],
  "tool_args": {
    "issue_refund": {
      "type": "object",
      "required": [
        "order_id"
      ],
      "properties": {
        "order_id": {
          "type": "string",
          "pattern": "^A\\d{3}$"
        }
      }
    }
  }
}

All 3 runs failed0.1 ms, outcome answer

tool_order wanted order lookup_order > check_policy > issue_refund, got lookup_order > issue_refund

ranoutcomeanswertool_args

Trace, 2 steps

toollookup_order0.0 ms

{
  "order_id": "A100"
}

Output

{
  "item": "headphones",
  "days": 10,
  "damage": null
}

toolissue_refund

{
  "order_id": "A100"
}

Output

refund issued for A100

Final output

{
  "outcome": "answer",
  "answer": "Your refund for order A100 is on its way."
}

All cases

refund-expired

refundsbroke vs baseline

0/3

runs passed

Input

I want my money back for A300

Expected

{
  "outcome": "answer",
  "answer": "30-day",
  "tools_forbidden": [
    "issue_refund"
  ]
}

All 3 runs failed0.2 ms, outcome answer

answer expected contains '30-day'

tools_forbidden called forbidden issue_refund

ranoutcome

Trace, 2 steps

toollookup_order

{
  "order_id": "A300"
}

Output

{
  "item": "laptop",
  "days": 45,
  "damage": null
}

toolissue_refund

{
  "order_id": "A300"
}

Output

refund issued for A300

Final output

{
  "outcome": "answer",
  "answer": "Your refund for order A300 is on its way."
}

All cases

refund-damaged

refundsuncoveredmust passbroke vs baseline

0/3

runs passed

Input

My phone from A200 arrived with a cracked screen, refund please

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "check_policy"
  ],
  "tools_forbidden": [
    "issue_refund"
  ]
}

All 3 runs failed0.0 ms, outcome answer

outcome expected escalate, got answer

tools_required never called check_policy

tools_forbidden called forbidden issue_refund

ran

Trace, 2 steps

toollookup_order

{
  "order_id": "A200"
}

Output

{
  "item": "phone",
  "days": 5,
  "damage": "cracked screen"
}

toolissue_refund

{
  "order_id": "A200"
}

Output

refund issued for A200

Final output

{
  "outcome": "answer",
  "answer": "Your refund for order A200 is on its way."
}

All cases

no-order-number

missing-infobroke vs baseline

0/3

runs passed

Input

Refund my last purchase

Expected

{
  "outcome": "abstain",
  "tools_forbidden": [
    "issue_refund"
  ],
  "max_tool_calls": 0
}

All 3 runs failed0.0 ms, outcome answer

outcome expected abstain, got answer

rantools_forbiddenmax_tool_calls

Trace

No steps recorded for this run.

Final output

{
  "outcome": "answer",
  "answer": "Sure, your refund is on its way."
}

Compare

This run against refund agent v1, case by case.

Metric	Baseline	This run	Change
pass rate	100.0%	0.0%	▼100.0 pts
pass^3	100.0%	0.0%	▼100.0 pts
held back when it should	100.0%	0.0%	▼100.0 pts
answered when it should	100.0%	100.0%	no change
p95 latency ms	0	0.1	▲0.1
check: answer	100.0%	50.0%	▼50.0 pts
check: max_tool_calls	100.0%	100.0%	no change
check: outcome	100.0%	50.0%	▼50.0 pts
check: ran	100.0%	100.0%	no change
check: tool_args	100.0%	100.0%	no change
check: tool_order	100.0%	0.0%	▼100.0 pts
check: tools_forbidden	100.0%	33.3%	▼66.7 pts
check: tools_required	100.0%	0.0%	▼100.0 pts

Broke

Passed in the baseline, fail now

Fixed

Failed in the baseline, pass now

No case was fixed.

Run info

Where these numbers came from.

This run

Label: refund agent v2
Suite: refund_cases
Suite file: refund_cases.jsonl
Target: refund_agent.agent_v2
Cases: 4
Repeats per case: 3
Runs: 12
Runs with errors: 0
Started: 2026-10-21T14:24:08+00:00
Duration: 0.0 s
Git commit: f6a2cba
Python: 3.14.7
holdline: 0.1.0

Baseline

Label: refund agent v1
Suite: refund_cases
Suite file: refund_cases.jsonl
Target: refund_agent.agent_v1
Cases: 4
Repeats per case: 3
Runs: 12
Runs with errors: 0
Started: 2026-10-21T14:24:08+00:00
Duration: 0.0 s
Git commit: f6a2cba
Python: 3.14.7
holdline: 0.1.0