helpdesk-agent (1 sample)

68 cases on helpdesk_cases, 3 runs each.

Gate passed
Compared with helpdesk-agent (5 samples) on 68 paired cases: pass rate +0.5 pts (95% CI +0.0 to +1.5), 0 broke and 0 fixed (McNemar p = 1).
  • newly flaky cases: is-scim-provisioning-available-on-the-business
Pass rate
96.1%
▲0.5 pts vs baseline
pass^3
95.6%
no change vs baseline
Held back when it should
100.0%
no change vs baseline
Answered when it should
97.8%
no change vs baseline
Flaky cases
1
passed some runs, failed others
p95 latency
3,003 ms
▼205.4 ms vs baseline

Case map

One square per case. Select one to open it

passed every runfailed every runflaky

Checks

Share of runs where each check passed

cites_gold_article
96.4%
helpdesk_correct
95.7%
max_tool_calls
100.0%
no_loops
100.0%
outcome
98.5%
ran
100.0%
tools_required
100.0%
this runbaseline

By tag

Mean case pass rate per tag

account
81.5%
billing
92.9%
covered
94.2%
data
100.0%
integrations
100.0%
support
100.0%
troubleshooting
100.0%
uncovered
100.0%
this runbaseline

Cases

Each case with its checks and the full trace of every run.

/search jkmove
68 casesruns
how-much-does-the-team-plan-cost
covered, billing
what-is-the-member-limit-on-the
outcome: expected answer, got escalate
which-plan-includes-the-uptime-sla
covered, billing
i-bought-an-annual-plan-10-days
covered, billing
are-monthly-plans-refundable
covered, billing
how-long-does-a-refund-take-to
covered, billing
can-i-pay-with-paypal
covered, billing
which-plans-can-pay-by-bank-transfer
covered, billing
when-does-a-downgrade-take-effect
covered, billing
where-do-i-download-my-invoices
covered, billing
can-an-invoice-be-changed-after-it
covered, billing
how-long-is-my-data-kept-after
covered, billing
how-many-guests-are-included-per-paid
covered, billing
do-nonprofits-get-a-discount
covered, billing
how-many-backup-codes-do-i-get
covered, account
can-an-admin-require-two-factor-authentication
covered, account
which-identity-providers-work-with-sso
covered, account
is-scim-provisioning-available-on-the-business
cites_gold_article: cited passage 11, gold article single-sign-on
how-long-is-a-password-reset-link
covered, account
who-can-transfer-ownership-of-a-workspace
helpdesk_correct: answered 'Only the Owner can transfer ownership of a workspace.', gold 'the Owner'
can-guests-see-the-member-directory
covered, account
how-long-is-the-grace-period-after
covered, account
after-how-long-of-inactivity-does-a
covered, account
how-long-does-the-data-export-download
covered, data
what-formats-can-i-export-a-workspace
covered, data
how-long-do-deleted-items-stay-in
covered, data
can-i-switch-an-existing-workspace-to
covered, data
where-is-my-data-stored-by-default
covered, data
which-security-certifications-does-northwind-have
covered, data
what-is-the-api-rate-limit-on
covered, integrations
how-many-times-is-a-failed-webhook
covered, integrations
does-the-free-plan-include-api-access
covered, integrations
how-many-api-tokens-can-one-user
covered, integrations
who-can-install-the-slack-integration
covered, integrations
how-often-does-google-calendar-sync-run
covered, integrations
can-i-import-from-jira-directly
covered, integrations
what-is-the-maximum-file-size-i
covered, troubleshooting
which-address-should-i-allow-list-for
covered, troubleshooting
is-internet-explorer-supported
covered, troubleshooting
what-ios-version-does-the-mobile-app
covered, troubleshooting
what-is-the-first-response-time-for
covered, support
is-phone-support-available-on-the-team
covered, support
what-uptime-does-the-sla-commit-to
covered, support
what-credit-do-i-get-if-uptime
covered, support
when-is-scheduled-maintenance-carried-out
covered, support
how-long-do-i-have-to-fix
covered, support
is-there-a-self-hosted-or-on
uncovered
does-the-api-support-graphql
uncovered
can-i-change-my-workspace-s-url
uncovered
does-the-mobile-app-have-a-dark
uncovered
do-you-integrate-with-microsoft-teams
uncovered
is-there-a-reseller-or-partner-programme
uncovered
can-i-set-up-recurring-tasks
uncovered
does-northwind-have-built-in-time-tracking
uncovered
is-there-a-limit-on-the-number
uncovered
can-i-merge-two-workspaces-into-one
uncovered
can-my-data-be-stored-in-canada
uncovered
do-you-offer-a-lifetime-deal
uncovered
which-languages-is-the-interface-available-in
uncovered
can-i-be-billed-in-euros
uncovered
is-there-an-audit-log-of-admin
uncovered
can-i-white-label-northwind-with-my
uncovered
does-the-free-plan-include-gantt-charts
uncovered
how-many-webhooks-can-i-register-per
uncovered
can-i-book-a-demo-call-with
uncovered
is-there-a-discount-for-startups
uncovered
can-i-pay-for-extra-storage-on
uncovered
what-happens-to-my-slack-mappings-if
uncovered
 All cases

how-much-does-the-team-plan-cost

coveredbilling
3/3
runs passed

Input

How much does the Team plan cost per user per month on annual billing?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,572 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How much does the Team plan cost per user per month on annual billing?"
    }
    Output
    {
      "ids": [
        11,
        18,
        26
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": ""
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "$12 per user per month",
  "reply": "The Team plan costs $12 per user per month when billed annually. This offers a great balance of features for growing teams!",
  "confidence": 1.0,
  "citation": 11,
  "llm_calls": 2
}
 All cases

what-is-the-member-limit-on-the

coveredbilling
0/3
runs passed

Input

What is the member limit on the Free plan?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs failed1,644 ms, outcome escalate
outcome expected answer, got escalate
helpdesk_correct escalated a covered ticket
cites_gold_article no reply, no citation
rantools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "What is the member limit on the Free plan?"
    }
    Output
    {
      "ids": [
        26,
        13,
        7
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss storage limits and how seats are counted, but do not specify a member limit for the Free plan. It is unlikely this information is in the corpus."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

which-plan-includes-the-uptime-sla

coveredbilling
3/3
runs passed

Input

Which plan includes the uptime SLA?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,882 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Which plan includes the uptime SLA?"
    }
    Output
    {
      "ids": [
        11,
        25,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The passages clearly state which plans include the uptime SLA."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Business and Enterprise plans",
  "reply": "The Business and Enterprise plans include the 99.9% uptime SLA.",
  "confidence": 1.0,
  "citation": 11,
  "llm_calls": 2
}
 All cases

i-bought-an-annual-plan-10-days

coveredbilling
3/3
runs passed

Input

I bought an annual plan 10 days ago, can I get a full refund?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,713 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "I bought an annual plan 10 days ago, can I get a full refund?"
    }
    Output
    {
      "ids": [
        12,
        6,
        24
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The user's question is directly answered by passage [1], which states that annual subscriptions can be refunded in full if requested within 14 days of purchase or renewal. The user bought the plan 10 "
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Yes, within 14 days",
  "reply": "Yes, you can get a full refund for your annual plan if you request it within 14 days of the purchase date. Please email billing@northwind.app with your invoice number to request the refund.",
  "confidence": 1.0,
  "citation": 12,
  "llm_calls": 2
}
 All cases

are-monthly-plans-refundable

coveredbilling
3/3
runs passed

Input

Are monthly plans refundable?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,968 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Are monthly plans refundable?"
    }
    Output
    {
      "ids": [
        12,
        11,
        25
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that monthly subscriptions are not refundable."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, monthly subscriptions are not refundable.",
  "reply": "No, monthly subscriptions are not refundable. When you cancel, the plan stays active until the end of the current month and no further charges are made.",
  "confidence": 1.0,
  "citation": 12,
  "llm_calls": 2
}
 All cases

how-long-does-a-refund-take-to

coveredbilling
3/3
runs passed

Input

How long does a refund take to appear?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,954 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How long does a refund take to appear?"
    }
    Output
    {
      "ids": [
        12,
        6,
        7
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that approved refunds usually appear within 5 to 7 business days."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "5 to 7 business days",
  "reply": "Approved refunds are returned to the original payment method and usually appear within 5 to 7 business days, depending on your bank.",
  "confidence": 1.0,
  "citation": 12,
  "llm_calls": 2
}
 All cases

can-i-pay-with-paypal

coveredbilling
3/3
runs passed

Input

Can I pay with PayPal?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,455 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Can I pay with PayPal?"
    }
    Output
    {
      "ids": [
        10,
        19,
        24
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The passages explicitly state that PayPal is not accepted as a payment method."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, PayPal is not accepted.",
  "reply": "I'm sorry, but we do not accept PayPal as a payment method.",
  "confidence": 1.0,
  "citation": 10,
  "llm_calls": 2
}
 All cases

which-plans-can-pay-by-bank-transfer

coveredbilling
3/3
runs passed

Input

Which plans can pay by bank transfer?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,978 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Which plans can pay by bank transfer?"
    }
    Output
    {
      "ids": [
        10,
        2,
        11
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The passages clearly state which plans can pay by bank transfer and under what conditions."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Business and Enterprise plans",
  "reply": "Bank transfer is available for Business and Enterprise workspaces on annual billing when the invoice total is $5,000 or more.",
  "confidence": 1.0,
  "citation": 10,
  "llm_calls": 2
}
 All cases

when-does-a-downgrade-take-effect

coveredbilling
3/3
runs passed

Input

When does a downgrade take effect?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,579 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "When does a downgrade take effect?"
    }
    Output
    {
      "ids": [
        7,
        12,
        6
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states when a downgrade takes effect."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "at the end of the current billing cycle",
  "reply": "Downgrades take effect at the end of your current billing cycle, so you'll keep your current features until then.",
  "confidence": 1.0,
  "citation": 7,
  "llm_calls": 2
}
 All cases

where-do-i-download-my-invoices

coveredbilling
3/3
runs passed

Input

Where do I download my invoices?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,385 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Where do I download my invoices?"
    }
    Output
    {
      "ids": [
        8,
        14,
        0
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly answers where to download invoices."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Under Settings > Billing > Invoices",
  "reply": "You can download your invoices as PDFs under Settings > Billing > Invoices.",
  "confidence": 1.0,
  "citation": 8,
  "llm_calls": 2
}
 All cases

can-an-invoice-be-changed-after-it

coveredbilling
3/3
runs passed

Input

Can an invoice be changed after it has been issued?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,637 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Can an invoice be changed after it has been issued?"
    }
    Output
    {
      "ids": [
        8,
        6,
        2
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The gathered passages clearly state that an invoice cannot be edited once it has been issued and provides instructions on how to request a corrected invoice."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, an invoice cannot be edited once issued.",
  "reply": "No, an invoice cannot be edited once it has been issued. If something is wrong, you can request a corrected invoice from billing@northwind.app within 30 days of the invoice date.",
  "confidence": 1.0,
  "citation": 8,
  "llm_calls": 2
}
 All cases

how-long-is-my-data-kept-after

coveredbilling
3/3
runs passed

Input

How long is my data kept after my subscription ends?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,586 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How long is my data kept after my subscription ends?"
    }
    Output
    {
      "ids": [
        6,
        0,
        23
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that data is retained for 90 days after the paid period ends."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "90 days",
  "reply": "Your data is retained for 90 days after your paid period ends. You can reactivate your subscription at any time within that window without losing any data.",
  "confidence": 1.0,
  "citation": 6,
  "llm_calls": 2
}
 All cases

how-many-guests-are-included-per-paid

coveredbilling
3/3
runs passed

Input

How many guests are included per paid seat?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,740 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How many guests are included per paid seat?"
    }
    Output
    {
      "ids": [
        13,
        11,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'each paid seat includes up to 3 free guests'."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Up to 3 free guests per paid seat",
  "reply": "Each paid seat includes up to 3 free guests. Guests can only see the projects they are invited to.",
  "confidence": 1.0,
  "citation": 13,
  "llm_calls": 2
}
 All cases

do-nonprofits-get-a-discount

coveredbilling
3/3
runs passed

Input

Do nonprofits get a discount?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,616 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Do nonprofits get a discount?"
    }
    Output
    {
      "ids": [
        9,
        13,
        10
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that registered nonprofits receive a 50% discount on specific plans."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Yes, registered nonprofits receive 50% off the Team and Business plans on annual billing.",
  "reply": "Yes, registered nonprofits receive 50% off the Team and Business plans on annual billing. You can apply by emailing billing@northwind.app with proof of your nonprofit status.",
  "confidence": 1.0,
  "citation": 9,
  "llm_calls": 2
}
 All cases

how-many-backup-codes-do-i-get

coveredaccount
3/3
runs passed

Input

How many backup codes do I get when I enable 2FA?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,672 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How many backup codes do I get when I enable 2FA?"
    }
    Output
    {
      "ids": [
        5,
        18,
        26
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 10 single-use backup codes are given when 2FA is enabled."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "10 single-use backup codes",
  "reply": "When you enable two-factor authentication, you will receive 10 single-use backup codes. Please store them in a safe place.",
  "confidence": 1.0,
  "citation": 5,
  "llm_calls": 2
}
 All cases

can-an-admin-require-two-factor-authentication

coveredaccount
3/3
runs passed

Input

Can an admin require two-factor authentication for all members?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,725 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Can an admin require two-factor authentication for all members?"
    }
    Output
    {
      "ids": [
        5,
        2,
        11
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'Workspace Owners and Admins on the Business and Enterprise plans can require 2FA for every member under Settings โ€บ Security'."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Yes, Admins on Business and Enterprise plans can require 2FA for all members.",
  "reply": "Yes, Admins on Business and Enterprise plans can require two-factor authentication for all members. Members without 2FA will be prompted to set it up at their next sign-in.",
  "confidence": 1.0,
  "citation": 5,
  "llm_calls": 2
}
 All cases

which-identity-providers-work-with-sso

coveredaccount
3/3
runs passed

Input

Which identity providers work with SSO?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,747 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Which identity providers work with SSO?"
    }
    Output
    {
      "ids": [
        4,
        1,
        3
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The passages clearly list the identity providers that work with SSO."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Okta, Microsoft Entra ID (Azure AD), Google Workspace and any SAML 2.0 identity provider.",
  "reply": "We support Okta, Microsoft Entra ID (Azure AD), Google Workspace, and any SAML 2.0 identity provider for single sign-on.",
  "confidence": 1.0,
  "citation": 4,
  "llm_calls": 2
}
 All cases

is-scim-provisioning-available-on-the-business

coveredaccount
1/3
runs passed

Input

Is SCIM provisioning available on the Business plan?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
Runs 1, 2 failed2,946 ms, outcome answer
cites_gold_article cited passage 11, gold article single-sign-on
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correct
Trace, 4 steps
  1. toolsearch
    {
      "query": "Is SCIM provisioning available on the Business plan?"
    }
    Output
    {
      "ids": [
        4,
        11,
        24
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [2] explicitly states that SCIM provisioning is available on the Enterprise plan, not the Business plan. Passage [1] confirms this by stating 'Automatic user provisioning and deprovisioning vi"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, SCIM provisioning is available on the Enterprise plan only.",
  "reply": "No, SCIM provisioning is available on the Enterprise plan only. The Business plan includes SAML single sign-on, required two-factor authentication, EU data residency, unlimited storage and the 99.9% uptime SLA.",
  "confidence": 1.0,
  "citation": 11,
  "llm_calls": 2
}
Run 3 passed2,081 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Is SCIM provisioning available on the Business plan?"
    }
    Output
    {
      "ids": [
        4,
        11,
        24
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [2] explicitly states that SCIM provisioning is available on the Enterprise plan, not the Business plan. Passage [1] confirms this by stating 'Automatic user provisioning and deprovisioning vi"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, SCIM provisioning is available on the Enterprise plan only.",
  "reply": "No, SCIM provisioning is available on the Enterprise plan only. The Business plan does not include this feature.",
  "confidence": 1.0,
  "citation": 4,
  "llm_calls": 2
}
 All cases

how-long-is-a-password-reset-link

coveredaccount
3/3
runs passed

Input

How long is a password reset link valid?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,684 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How long is a password reset link valid?"
    }
    Output
    {
      "ids": [
        1,
        3,
        14
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The gathered passages clearly state that a password reset link is valid for 1 hour."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "1 hour",
  "reply": "The password reset link is valid for 1 hour after it's sent to your email address.",
  "confidence": 1.0,
  "citation": 1,
  "llm_calls": 2
}
 All cases

who-can-transfer-ownership-of-a-workspace

coveredaccount
0/3
runs passed

Input

Who can transfer ownership of a workspace?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs failed2,468 ms, outcome answer
helpdesk_correct answered 'Only the Owner can transfer ownership of a workspace.', gold 'the Owner'
ranoutcometools_requiredmax_tool_callsno_loopscites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Who can transfer ownership of a workspace?"
    }
    Output
    {
      "ids": [
        2,
        0,
        7
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'The Owner is the single person who can transfer ownership'."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Only the Owner can transfer ownership of a workspace.",
  "reply": "Only the Owner of a workspace can transfer its ownership. You can find this option under Settings > General.",
  "confidence": 1.0,
  "citation": 2,
  "llm_calls": 2
}
 All cases

can-guests-see-the-member-directory

coveredaccount
3/3
runs passed

Input

Can guests see the member directory?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,500 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Can guests see the member directory?"
    }
    Output
    {
      "ids": [
        2,
        13,
        4
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'Guests can only see the projects they have been invited to and cannot see the member directory.'"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, guests cannot see the member directory.",
  "reply": "No, guests cannot see the member directory. They can only see the projects they have been invited to.",
  "confidence": 1.0,
  "citation": 2,
  "llm_calls": 2
}
 All cases

how-long-is-the-grace-period-after

coveredaccount
3/3
runs passed

Input

How long is the grace period after requesting account deletion?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,449 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How long is the grace period after requesting account deletion?"
    }
    Output
    {
      "ids": [
        0,
        6,
        15
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states the grace period for account deletion."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "7 days",
  "reply": "There is a 7-day grace period after requesting account deletion, during which signing in again will cancel the request.",
  "confidence": 1.0,
  "citation": 0,
  "llm_calls": 2
}
 All cases

after-how-long-of-inactivity-does-a

coveredaccount
3/3
runs passed

Input

After how long of inactivity does a session expire?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,472 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "After how long of inactivity does a session expire?"
    }
    Output
    {
      "ids": [
        3,
        6,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'Sessions expire automatically after 30 days of inactivity.' It also mentions that 'Enterprise workspaces can set a shorter session timeout, down to 8 hours.'"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "30 days",
  "reply": "Sessions expire automatically after 30 days of inactivity. Enterprise workspaces have the option to set a shorter session timeout if they wish.",
  "confidence": 1.0,
  "citation": 3,
  "llm_calls": 2
}
 All cases

how-long-does-the-data-export-download

covereddata
3/3
runs passed

Input

How long does the data export download link stay valid?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,551 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How long does the data export download link stay valid?"
    }
    Output
    {
      "ids": [
        14,
        1,
        23
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that the data export download link stays valid for 7 days."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "7 days",
  "reply": "The download link for your data export will remain valid for 7 days after you receive it.",
  "confidence": 1.0,
  "citation": 14,
  "llm_calls": 2
}
 All cases

what-formats-can-i-export-a-workspace

covereddata
3/3
runs passed

Input

What formats can I export a workspace in?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,399 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "What formats can I export a workspace in?"
    }
    Output
    {
      "ids": [
        14,
        16,
        21
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The passages clearly state the export formats for a workspace (JSON and CSV) and individual projects (CSV)."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "JSON or CSV",
  "reply": "You can export your workspace in either JSON or CSV format. You'll receive an email with a download link within 24 hours.",
  "confidence": 1.0,
  "citation": 14,
  "llm_calls": 2
}
 All cases

how-long-do-deleted-items-stay-in

covereddata
3/3
runs passed

Input

How long do deleted items stay in Trash?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,821 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How long do deleted items stay in Trash?"
    }
    Output
    {
      "ids": [
        15,
        0,
        26
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly answers how long deleted items stay in Trash, stating 30 days by default and a custom range for Enterprise workspaces."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "30 days, or 7-365 days for Enterprise workspaces",
  "reply": "Deleted items are moved to Trash and can be restored for 30 days. Enterprise workspaces have the option to set a custom retention period between 7 and 365 days.",
  "confidence": 1.0,
  "citation": 15,
  "llm_calls": 2
}
 All cases

can-i-switch-an-existing-workspace-to

covereddata
3/3
runs passed

Input

Can I switch an existing workspace to EU data residency?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,820 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Can I switch an existing workspace to EU data residency?"
    }
    Output
    {
      "ids": [
        16,
        11,
        6
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly addresses the question, stating that residency cannot be changed for an existing workspace and provides the alternative solution."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, residency cannot be changed for an existing workspace.",
  "reply": "No, residency cannot be changed for an existing workspace. To move to the EU region, you would need to create a new workspace in the EU and import your data.",
  "confidence": 1.0,
  "citation": 16,
  "llm_calls": 2
}
 All cases

where-is-my-data-stored-by-default

covereddata
3/3
runs passed

Input

Where is my data stored by default?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,932 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Where is my data stored by default?"
    }
    Output
    {
      "ids": [
        16,
        15,
        21
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states the default data storage location."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "United States (AWS us-east-1)",
  "reply": "By default, your workspace data is stored in the United States (AWS us-east-1).",
  "confidence": 1.0,
  "citation": 16,
  "llm_calls": 2
}
 All cases

which-security-certifications-does-northwind-have

covereddata
3/3
runs passed

Input

Which security certifications does Northwind have?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,224 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Which security certifications does Northwind have?"
    }
    Output
    {
      "ids": [
        17,
        27,
        16
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states Northwind's security certifications: SOC 2 Type II and ISO 27001."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "SOC 2 Type II and ISO 27001 certified",
  "reply": "Northwind is SOC 2 Type II and ISO 27001 certified. You can request reports under NDA from security@northwind.app.",
  "confidence": 1.0,
  "citation": 17,
  "llm_calls": 2
}
 All cases

what-is-the-api-rate-limit-on

coveredintegrations
3/3
runs passed

Input

What is the API rate limit on the Business plan?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,512 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "What is the API rate limit on the Business plan?"
    }
    Output
    {
      "ids": [
        18,
        26,
        24
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states the API rate limit for the Business plan."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "1,000 requests per minute",
  "reply": "On the Business plan, the REST API is rate-limited to 1,000 requests per minute.",
  "confidence": 1.0,
  "citation": 18,
  "llm_calls": 2
}
 All cases

how-many-times-is-a-failed-webhook

coveredintegrations
3/3
runs passed

Input

How many times is a failed webhook retried?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,339 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How many times is a failed webhook retried?"
    }
    Output
    {
      "ids": [
        18,
        26,
        19
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'Webhook deliveries that fail (any non-2xx response) are retried 5 times with exponential backoff over 24 hours'."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "5 times",
  "reply": "Webhook deliveries that fail are retried 5 times with exponential backoff over 24 hours.",
  "confidence": 1.0,
  "citation": 18,
  "llm_calls": 2
}
 All cases

does-the-free-plan-include-api-access

coveredintegrations
3/3
runs passed

Input

Does the Free plan include API access?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,810 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Does the Free plan include API access?"
    }
    Output
    {
      "ids": [
        18,
        19,
        6
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'The Free plan does not include API access.'"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, the Free plan does not include API access.",
  "reply": "No, the Free plan does not include API access. You would need to upgrade to a paid plan to use the API.",
  "confidence": 1.0,
  "citation": 18,
  "llm_calls": 2
}
 All cases

how-many-api-tokens-can-one-user

coveredintegrations
3/3
runs passed

Input

How many API tokens can one user have?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,711 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How many API tokens can one user have?"
    }
    Output
    {
      "ids": [
        19,
        18,
        0
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'Each user can have up to 10 active tokens'."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "10 active tokens",
  "reply": "Each user can have up to 10 active API tokens.",
  "confidence": 1.0,
  "citation": 19,
  "llm_calls": 2
}
 All cases

who-can-install-the-slack-integration

coveredintegrations
3/3
runs passed

Input

Who can install the Slack integration?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,003 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Who can install the Slack integration?"
    }
    Output
    {
      "ids": [
        22,
        2,
        19
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states that 'Installing the app requires a workspace Admin or Owner.'"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "A workspace Admin or Owner",
  "reply": "To install the Slack integration, you'll need to be a workspace Admin or Owner. They can connect Slack under Settings > Integrations > Slack.",
  "confidence": 1.0,
  "citation": 22,
  "llm_calls": 2
}
 All cases

how-often-does-google-calendar-sync-run

coveredintegrations
3/3
runs passed

Input

How often does Google Calendar sync run?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,969 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How often does Google Calendar sync run?"
    }
    Output
    {
      "ids": [
        20,
        11,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] explicitly states that 'Sync runs every 5 minutes.'"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "every 5 minutes",
  "reply": "Google Calendar sync runs every 5 minutes.",
  "confidence": 1.0,
  "citation": 20,
  "llm_calls": 2
}
 All cases

can-i-import-from-jira-directly

coveredintegrations
3/3
runs passed

Input

Can I import from Jira directly?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,884 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Can I import from Jira directly?"
    }
    Output
    {
      "ids": [
        21,
        14,
        16
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly addresses the question about importing from Jira."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, Jira can only be imported from a CSV export.",
  "reply": "No, you cannot import directly from Jira. You can import from a CSV export, but issue links and worklogs will not be imported.",
  "confidence": 1.0,
  "citation": 21,
  "llm_calls": 2
}
 All cases

what-is-the-maximum-file-size-i

coveredtroubleshooting
3/3
runs passed

Input

What is the maximum file size I can upload on the Team plan?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,863 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "What is the maximum file size I can upload on the Team plan?"
    }
    Output
    {
      "ids": [
        26,
        24,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage 1 directly states the maximum file size for the Team plan."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "250 MB",
  "reply": "On the Team plan, you can upload individual files up to a maximum size of 250 MB.",
  "confidence": 1.0,
  "citation": 26,
  "llm_calls": 2
}
 All cases

which-address-should-i-allow-list-for

coveredtroubleshooting
3/3
runs passed

Input

Which address should I allow-list for notification emails?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,665 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Which address should I allow-list for notification emails?"
    }
    Output
    {
      "ids": [
        28,
        1,
        9
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states the email address to allow-list for notification emails."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "notify@northwind.app",
  "reply": "To ensure you receive notification emails, please add notify@northwind.app to your contacts or allow list.",
  "confidence": 1.0,
  "citation": 28,
  "llm_calls": 2
}
 All cases

is-internet-explorer-supported

coveredtroubleshooting
3/3
runs passed

Input

Is Internet Explorer supported?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,170 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Is Internet Explorer supported?"
    }
    Output
    {
      "ids": [
        29,
        10,
        26
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The gathered passages clearly state that Internet Explorer is not supported."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, Internet Explorer is not supported.",
  "reply": "I'm sorry, but Internet Explorer is not a supported browser for Northwind. We recommend using one of the most recent versions of Chrome, Firefox, Safari, or Microsoft Edge.",
  "confidence": 1.0,
  "citation": 29,
  "llm_calls": 2
}
 All cases

what-ios-version-does-the-mobile-app

coveredtroubleshooting
3/3
runs passed

Input

What iOS version does the mobile app need?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,832 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "What iOS version does the mobile app need?"
    }
    Output
    {
      "ids": [
        27,
        28,
        29
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The answer is present in passage [1]."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "iOS 15 or later",
  "reply": "The Northwind app requires iOS 15 or later to run. Please ensure your device meets this requirement for optimal performance.",
  "confidence": 1.0,
  "citation": 27,
  "llm_calls": 2
}
 All cases

what-is-the-first-response-time-for

coveredsupport
3/3
runs passed

Input

What is the first response time for support on the Team plan?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,872 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "What is the first response time for support on the Team plan?"
    }
    Output
    {
      "ids": [
        24,
        18,
        26
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states the first response time for support on the Team plan."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "1 business day",
  "reply": "On the Team plan, you can expect a first response from support within 1 business day.",
  "confidence": 1.0,
  "citation": 24,
  "llm_calls": 2
}
 All cases

is-phone-support-available-on-the-team

coveredsupport
3/3
runs passed

Input

Is phone support available on the Team plan?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed2,661 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "Is phone support available on the Team plan?"
    }
    Output
    {
      "ids": [
        24,
        26,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] explicitly states that 'Phone support is available on the Enterprise plan only.' This directly answers the question about phone support availability on the Team plan."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "No, phone support is only available on the Enterprise plan.",
  "reply": "No, phone support is not available on the Team plan. It is exclusively offered with the Enterprise plan.",
  "confidence": 1.0,
  "citation": 24,
  "llm_calls": 2
}
 All cases

what-uptime-does-the-sla-commit-to

coveredsupport
3/3
runs passed

Input

What uptime does the SLA commit to?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,280 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "What uptime does the SLA commit to?"
    }
    Output
    {
      "ids": [
        25,
        11,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "The passages clearly state the uptime commitment for Business and Enterprise plans."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "99.9% monthly uptime",
  "reply": "Business and Enterprise plans include a 99.9% monthly uptime commitment.",
  "confidence": 1.0,
  "citation": 25,
  "llm_calls": 2
}
 All cases

what-credit-do-i-get-if-uptime

coveredsupport
3/3
runs passed

Input

What credit do I get if uptime drops below 99%?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,158 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "What credit do I get if uptime drops below 99%?"
    }
    Output
    {
      "ids": [
        25,
        11,
        10
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states the credit for uptime below 99.0% for Business and Enterprise plans."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "25% of that month's fee",
  "reply": "If availability in a calendar month falls below 99.0%, you can claim a credit of 25% of that month's fee.",
  "confidence": 1.0,
  "citation": 25,
  "llm_calls": 2
}
 All cases

when-is-scheduled-maintenance-carried-out

coveredsupport
3/3
runs passed

Input

When is scheduled maintenance carried out?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,099 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "When is scheduled maintenance carried out?"
    }
    Output
    {
      "ids": [
        25,
        3,
        10
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states when scheduled maintenance takes place."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "Sundays between 02:00 and 06:00 UTC",
  "reply": "Scheduled maintenance is carried out on Sundays between 02:00 and 06:00 UTC.",
  "confidence": 1.0,
  "citation": 25,
  "llm_calls": 2
}
 All cases

how-long-do-i-have-to-fix

coveredsupport
3/3
runs passed

Input

How long do I have to fix an acceptable-use violation before suspension?

Expected

{
  "outcome": "answer",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true,
  "checks": [
    "helpdesk_correct",
    "cites_gold_article"
  ]
}
All 3 runs passed3,206 ms, outcome answer
ranoutcometools_requiredmax_tool_callsno_loopshelpdesk_correctcites_gold_article
Trace, 4 steps
  1. toolsearch
    {
      "query": "How long do I have to fix an acceptable-use violation before suspension?"
    }
    Output
    {
      "ids": [
        23,
        0,
        8
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] directly states the timeframe for remedying an acceptable-use violation before suspension."
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 1,
      "agreement": 1.0,
      "errors": 0
    }
  4. decisionanswer
    Output
    {
      "confidence": 1.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "answer",
  "answer": "7 days",
  "reply": "You have 7 days to remedy an acceptable-use violation before your workspace is suspended.",
  "confidence": 1.0,
  "citation": 23,
  "llm_calls": 2
}
 All cases

is-there-a-self-hosted-or-on

uncoveredmust pass
3/3
runs passed

Input

Is there a self-hosted or on-premise version of Northwind?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,954 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Is there a self-hosted or on-premise version of Northwind?"
    }
    Output
    {
      "ids": [
        28,
        16,
        8
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss notifications, data residency (cloud-based AWS), and invoicing. None of them mention anything about self-hosting, on-premise versions, or deployment options for Northwind. It's un"
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

does-the-api-support-graphql

uncoveredmust pass
3/3
runs passed

Input

Does the API support GraphQL?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,607 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Does the API support GraphQL?"
    }
    Output
    {
      "ids": [
        18,
        19,
        24
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss API rate limits, tokens, and support plans, but none mention GraphQL support. It's unlikely this corpus contains the answer."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-i-change-my-workspace-s-url

uncoveredmust pass
3/3
runs passed

Input

Can I change my workspace's URL?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,918 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Can I change my workspace's URL?"
    }
    Output
    {
      "ids": [
        7,
        2,
        6
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss changing plans, roles, and cancelling subscriptions, but none mention changing a workspace's URL. It's unlikely this corpus contains the answer."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

does-the-mobile-app-have-a-dark

uncoveredmust pass
3/3
runs passed

Input

Does the mobile app have a dark mode?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,803 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Does the mobile app have a dark mode?"
    }
    Output
    {
      "ids": [
        27,
        28,
        19
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss various app features like offline mode, biometrics, and notifications, but none mention dark mode. It's unlikely this corpus contains the answer."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

do-you-integrate-with-microsoft-teams

uncoveredmust pass
3/3
runs passed

Input

Do you integrate with Microsoft Teams?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,941 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Do you integrate with Microsoft Teams?"
    }
    Output
    {
      "ids": [
        4,
        24,
        18
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss integrations with Microsoft Entra ID (Azure AD) for single sign-on, but there is no mention of direct integration with Microsoft Teams for communication or collaboration purposes."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

is-there-a-reseller-or-partner-programme

uncoveredmust pass
3/3
runs passed

Input

Is there a reseller or partner programme?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,109 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Is there a reseller or partner programme?"
    }
    Output
    {
      "ids": [
        17,
        10,
        1
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The provided passages discuss security, payment methods, and password resets, none of which are related to reseller or partner programs. It is unlikely this corpus contains the answer."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-i-set-up-recurring-tasks

uncoveredmust pass
3/3
runs passed

Input

Can I set up recurring tasks?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,059 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Can I set up recurring tasks?"
    }
    Output
    {
      "ids": [
        15,
        20,
        22
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss task management features like deletion, Google Calendar sync, and Slack integration, but none mention recurring tasks. It's unlikely this corpus contains information on that speci"
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

does-northwind-have-built-in-time-tracking

uncoveredmust pass
3/3
runs passed

Input

Does Northwind have built-in time tracking?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed4,450 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 5 steps
  1. toolsearch
    {
      "query": "Does Northwind have built-in time tracking?"
    }
    Output
    {
      "ids": [
        20,
        11,
        14
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "search",
      "query": "Northwind time management features",
      "reason": "The current passages discuss calendar sync, pricing plans, and data export, but do not mention time tracking or time management features. A new query focusing on 'time management' or 'time tracking' f"
    }
  3. toolsearch
    {
      "query": "Northwind time management features"
    }
    Output
    {
      "ids": [
        11,
        14,
        29
      ],
      "new": 1
    }
  4. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The provided passages discuss Google Calendar sync, pricing plans, data export, and browser support. None of them mention time tracking features within Northwind. It's unlikely this corpus contains in"
    }
  5. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 2
}
 All cases

is-there-a-limit-on-the-number

uncoveredmust pass
3/3
runs passed

Input

Is there a limit on the number of projects in a workspace?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,039 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Is there a limit on the number of projects in a workspace?"
    }
    Output
    {
      "ids": [
        26,
        2,
        7
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss file size limits, storage limits, roles, and plan changes, but none mention any limits on the number of projects within a workspace. The tried search query was direct and did not "
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-i-merge-two-workspaces-into-one

uncoveredmust pass
3/3
runs passed

Input

Can I merge two workspaces into one?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,109 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Can I merge two workspaces into one?"
    }
    Output
    {
      "ids": [
        14,
        0,
        19
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The provided passages discuss exporting workspaces, deleting accounts, and API tokens. None of them address the possibility of merging two workspaces into one. It is unlikely this corpus contains the "
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-my-data-be-stored-in-canada

uncoveredmust pass
3/3
runs passed

Input

Can my data be stored in Canada?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,232 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Can my data be stored in Canada?"
    }
    Output
    {
      "ids": [
        16,
        21,
        23
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss data storage locations (US, EU) but do not mention Canada. It's unlikely the corpus contains this specific information."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

do-you-offer-a-lifetime-deal

uncoveredmust pass
3/3
runs passed

Input

Do you offer a lifetime deal?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,816 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Do you offer a lifetime deal?"
    }
    Output
    {
      "ids": [
        10,
        13,
        12
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The provided passages discuss payment methods, seat management, and refund policies. None of them mention anything about 'lifetime deals' or similar long-term subscription offers. It's unlikely this c"
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

which-languages-is-the-interface-available-in

uncoveredmust pass
3/3
runs passed

Input

Which languages is the interface available in?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,935 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Which languages is the interface available in?"
    }
    Output
    {
      "ids": [
        10,
        4,
        27
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The provided passages discuss payment methods, single sign-on, and app availability/features, but none of them mention the languages the interface is available in. It's unlikely this corpus contains t"
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-i-be-billed-in-euros

uncoveredmust pass
3/3
runs passed

Input

Can I be billed in euros?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,533 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 4 steps
  1. toolsearch
    {
      "query": "Can I be billed in euros?"
    }
    Output
    {
      "ids": [
        11,
        16,
        21
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "answer",
      "query": null,
      "reason": "Passage [1] states that 'All prices are in US dollars and exclude tax.'"
    }
  3. llmanswer_samples
    {
      "n": 1
    }
    Output
    {
      "votes": 0,
      "agreement": 0.0,
      "errors": 0
    }
  4. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 2
}
 All cases

is-there-an-audit-log-of-admin

uncoveredmust pass
3/3
runs passed

Input

Is there an audit log of admin actions?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,721 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Is there an audit log of admin actions?"
    }
    Output
    {
      "ids": [
        2,
        7,
        14
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages describe what admins can do and how to export data, but none mention an audit log of admin actions. It's unlikely this corpus contains the answer."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-i-white-label-northwind-with-my

uncoveredmust pass
3/3
runs passed

Input

Can I white-label Northwind with my own branding?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,753 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Can I white-label Northwind with my own branding?"
    }
    Output
    {
      "ids": [
        16,
        17,
        9
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The provided passages discuss data processing, security, and discounts. There is no information about white-labeling or branding options for Northwind."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

does-the-free-plan-include-gantt-charts

uncoveredmust pass
3/3
runs passed

Input

Does the Free plan include Gantt charts?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,765 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Does the Free plan include Gantt charts?"
    }
    Output
    {
      "ids": [
        18,
        24,
        26
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss API access, support, and file storage for the Free plan, but none mention Gantt charts. It's unlikely this corpus contains information about specific feature inclusions like Gantt"
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

how-many-webhooks-can-i-register-per

uncoveredmust pass
3/3
runs passed

Input

How many webhooks can I register per workspace?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,597 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "How many webhooks can I register per workspace?"
    }
    Output
    {
      "ids": [
        18,
        11,
        26
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss webhook delivery failures and API rate limits, but none specify a limit on the number of webhooks that can be registered per workspace."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-i-book-a-demo-call-with

uncoveredmust pass
3/3
runs passed

Input

Can I book a demo call with sales?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed1,696 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Can I book a demo call with sales?"
    }
    Output
    {
      "ids": [
        24,
        9,
        12
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss support plans, discounts, and refunds, but none mention booking a demo call with sales. It's unlikely this corpus contains information about sales demos."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

is-there-a-discount-for-startups

uncoveredmust pass
3/3
runs passed

Input

Is there a discount for startups?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,151 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Is there a discount for startups?"
    }
    Output
    {
      "ids": [
        10,
        24,
        7
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss payment methods, support plans, and plan changes, but none mention anything about discounts, especially for startups. It's unlikely this corpus contains the answer."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

can-i-pay-for-extra-storage-on

uncoveredmust pass
3/3
runs passed

Input

Can I pay for extra storage on the Team plan without upgrading?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,292 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "Can I pay for extra storage on the Team plan without upgrading?"
    }
    Output
    {
      "ids": [
        26,
        11,
        7
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The provided passages describe the storage limits for each plan and how upgrades/downgrades work, but they do not mention any option to purchase additional storage for a specific plan without upgradin"
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}
 All cases

what-happens-to-my-slack-mappings-if

uncoveredmust pass
3/3
runs passed

Input

What happens to my Slack mappings if I change my Slack workspace name?

Expected

{
  "outcome": "escalate",
  "tools_required": [
    "search"
  ],
  "max_tool_calls": 4,
  "no_loops": true
}
All 3 runs passed2,075 ms, outcome escalate
ranoutcometools_requiredmax_tool_callsno_loops
Trace, 3 steps
  1. toolsearch
    {
      "query": "What happens to my Slack mappings if I change my Slack workspace name?"
    }
    Output
    {
      "ids": [
        22,
        6,
        7
      ],
      "new": 3
    }
  2. llmdecide
    Output
    {
      "next": "abstain",
      "reason": "The passages discuss connecting and disconnecting Slack, but do not mention what happens to mappings if the Slack workspace name changes. It's unlikely this specific detail is in the corpus."
    }
  3. decisionescalate
    Output
    {
      "confidence": 0.0,
      "threshold": 0.0
    }

Final output

{
  "outcome": "escalate",
  "answer": "",
  "reply": "",
  "confidence": 0.0,
  "citation": null,
  "llm_calls": 1
}

Compare

This run against helpdesk-agent (5 samples), case by case.

MetricBaselineThis runChange
pass rate95.6%96.1%▲0.5 pts
pass^395.6%95.6%no change
held back when it should100.0%100.0%no change
answered when it should97.8%97.8%no change
p95 latency ms32083003▼205.4
check: cites_gold_article95.7%96.4%▲0.7 pts
check: helpdesk_correct95.7%95.7%no change
check: max_tool_calls100.0%100.0%no change
check: no_loops100.0%100.0%no change
check: outcome98.5%98.5%no change
check: ran100.0%100.0%no change
check: tools_required100.0%100.0%no change

Broke

Passed in the baseline, fail now

No case broke.

Fixed

Failed in the baseline, pass now

No case was fixed.

Run info

Where these numbers came from.

This run

Label
helpdesk-agent (1 sample)
Suite
helpdesk_cases
Suite file
helpdesk_cases.jsonl
Target
target.agent
Cases
68
Repeats per case
3
Runs
204
Runs with errors
0
Started
2026-10-21T13:59:38+00:00
Duration
138.45 s
Git commit
5d3b4b6
Python
3.14.7
holdline
0.1.0

Baseline

Label
helpdesk-agent (5 samples)
Suite
helpdesk_cases
Suite file
helpdesk_cases.jsonl
Target
target.agent
Cases
68
Repeats per case
3
Runs
204
Runs with errors
0
Started
2026-10-21T13:56:57+00:00
Duration
140.88 s
Git commit
5d3b4b6
Python
3.14.7
holdline
0.1.0