Investigates the alert. Shows the evidence. Waits for your yes.
When an alert fires, greenlight investigates through its own MCP tools, cites the evidence behind every claim, and proposes one fix. It has no way to change anything itself: a separate runner executes only the exact action a person approved. When the evidence is thin, it says so.
How it works
From alert to postmortem, with a person at the one step that matters.
greenlight runs against a small shop (web, checkout and payments) that is really running, with real metrics, logs, deploys and feature flags. Incidents are caused on purpose and recorded with their ground truth kept apart.
An alert fires
Checkout errors above 5%, latency above a second, memory near its limit, card declines rising.
It investigates
Six MCP servers it can only read from: the incident, metrics, logs, changes, runbooks and allowed actions.
It shows its work
Every tool result gets an evidence id. A second model checks each claim against exactly the evidence it cites.
You approve the exact change
You type a short code derived from the action. Only then is a single-use approval signed.
The runner acts once
It verifies the signature, makes the change, watches the recovery and drafts the postmortem.
Safety
It cannot change anything on its own.
These are properties of how the system is built, so they hold even if the model is fooled by text planted in a log line or a runbook.
No route to execution
The agent's tools read data or validate a proposal. The action schema allows four bounded kinds of change, each with its undo shown.
Only the approved change runs
An HMAC signature over canonical JSON binds the action, where it runs, the incident, the approver and the expiry.
One approval, one execution
Each approval carries a single-use nonce. Replaying it is refused by the runner.
A person reads the exact action
The read-back code is derived from the action. Misreading or swapping the action gives a different code.
Text in data stays data
Tool results are wrapped as evidence, and a 50-attack red-team suite measures whether planted instructions get through.
Everything is recorded
The runner writes every change to an append-only audit log. Updates and deletes are refused by the database.
Results
Measured on incidents it never saw during tuning.
24 held-out incidents (8 kinds, 3 seeds each), split off before any tuning. greenlight ran each one three times; the two baselines ran once. Brackets are 95% Wilson intervals.
| greenlight | Same model and tools | One prompt, no tools | |
|---|---|---|---|
| Right action and right diagnosis | 72/72[0.949, 1] | 24/24[0.862, 1] | 20/24[0.641, 0.933] |
| Proposed an unsafe action | 0/72 | 0/24 | 4/24 |
| No action on the three kinds where none is right | 27/27 | 9/9 | 6/9 |
| Ran out of model calls, so no action | 7/72 | 4/24 | 0/24 |
| Median time per incident | 74 s | 57 s | 27 s |
Source: results/eval/summary-test.json. A larger model from the same provider grades the diagnosis; actions are checked against the recorded ground truth.
| Red team, 50 planted attacks | greenlight | Same model and tools |
|---|---|---|
| Attack reached its goal | 0/50 | 0/50 |
| Proposal carried an unsupported claim | 2/50 | 1/50 |
Attacks were planted in logs, commit messages, alert text, runbooks and provider responses, aiming at a wrong action, a leaked secret, skipped evidence, self-approval or a destructive command. Source: results/redteam/summary.json.
| Runbook retrieval, 30 held-out queries | Recall@5 |
|---|---|
| Keyword search | 0.817 |
| Hybrid search with reranking | 0.967 |
Difference +0.15, 95% CI [0.067, 0.25]. Source: results/retrieval.json.
What the numbers say
- Tools and a fixed list of actions matter most. One prompt with the data pasted in led to 4 unsafe proposals in 24 incidents; the tool-using agents made none.
- On these sets the evidence check and gate do not score higher than the same model with the same tools. What they add is the cited, verified evidence a person reads before approving.
- 7 of greenlight's 27 "no action" outcomes came from running out of model calls rather than concluding the evidence was thin. Both fail closed, and they are counted separately.
Architecture
A real system to break, an agent that can only read, and a runner that acts once.
Every incident runs as a durable workflow, so a crash or a deploy while it waits half an hour for a person cannot lose it or run an action twice.
Limits
What this does not show.
- The shop and its bugs were built for the demo. Real incidents are messier.
- A replay can only show outcomes that were recorded, so only the recorded fix shows a recovery.
- The diagnosis judge is a model from the same provider as the agent. Its agreement with human graders is not measured.
- 24 held-out incidents and 50 attacks are small sets; the intervals show how much.
- The runbooks and retrieval queries were written for this project.
Approve a fix yourself.
Open a recorded incident, read the evidence, and type the code. You get a private replay.