WarrantYour model says it is right 95% of the time. On your book it was 85%.
That difference is the reason automated decisioning stays stuck in pilot inside regulated firms, and almost nobody can measure it. It needs two things held together: the confidence stated when the decision was made, and what actually happened ninety days later. Warrant is where those two meet.
Open-source core · runs in your environment · every figure on this page is synthetic and is produced by a deliberately flawed stand-in model, not by a vendor's
The measurement nothing else produces
A reliability curve on your own book, not a benchmark.
Bucket every decision by the confidence that was stated at the time. Compare each bucket against what actually happened. If the decider is calibrated, the two agree. Where they do not, you have found the band you were about to automate.
Read the coverage line first. A curve over 93% of the automated band is a claim you can take to a model validation committee. The same curve over 12% is not, and the two numbers always travel together.
Roughly a fifth of these decisions had no answer even at the horizon. They stay outcome-pending and are left out of the curve, because recording a guess is the one thing that would make the whole measurement worthless.
$ warrant calibrate --stream aml --answer disposition \
--where "decision.route == 'auto'" \
--correct-when "outcome.label == 'stayed_closed'"
over decisions where: decision.route == 'auto'
28 decision(s), 28 with confidence, 26 with an outcome (92.9% attached), 26 usable
stated 0.953 vs observed 0.846 ECE 0.1069 MCE 0.1069 Brier 0.1411
band n stated observed gap
0.90-1.00 26 0.953 0.846 -0.107Three layers, one control path
Each does one job, and is forbidden the others.
The boundaries matter more than the features, because they are what let any one layer be replaced without rewriting the system. No model vendor's vocabulary reaches the record, so changing the model changes an adapter and nothing else.
| The decision model | Typed answers with a probability for each, rather than a paragraph and an adjective | Nothing else. Not process state, not retries, not who is allowed to decide |
| Temporal | The case lifecycle: retries, the wait for a human, and a durable timer that survives a quarter | Nothing about what was decided, why, or whether it turned out to be right |
| Warrant | The immutable record, the policy check, the outcome linked months later, the reliability curve, the pack | No business logic and no orchestration. It proves; it does not decide |
One rule holds the whole thing together: there is no code path that calls the model without writing a record. The call and the record are the same operation, not two things a developer has to remember.
The unglamorous half
Somebody has to still be listening ninety days later.
A decision record without an outcome is a log. The realised result — an alert reopened, a loan defaulted, a promise to pay kept — arrives weeks or months afterwards, and lands in a different system belonging to a different team.
So the case stays open. A durable timer set at the horizon where the truth becomes knowable survives your deploys, your restarts and your on-call rotation, then wakes, asks the system of record what happened, and links it. The outcome attaches to the decision without changing it: a new record, never an edit.
This is the least glamorous part of the stack and the hardest to do without. It is also the only part a competitor starting later cannot catch up on, because eighteen months of decisions paired with their outcomes cannot be back-filled.
And some things never resolve themselves
What no confidence score is allowed to override.
Carve-outs belong in the policy, where they can be read and reviewed, not in the prose of a vendor's brochure. These are enforced in code and covered by tests:
- Politically exposed persons never auto-resolve, at any confidence
- Sanctions-adjacent and trade finance never auto-resolve, at any confidence
- A structuring pattern always goes to a person
- If a rule cannot be evaluated at all, the case goes to a person
The failure mode that matters in this kind of work is rarely a missed fraud. It is a case closed by a machine that should have been escalated, and that carries personal liability for a named officer. Which is why a circuit breaker can force a case to a human, and can never force one the other way.
What a committee actually reads
One artefact: the records, the proof they are intact, and what they add up to.
Generated on demand for a date range: every sealed record, the policy versions that were in force, the question sets and their versions, outcome coverage, the reliability curve, and instructions to verify the whole thing offline.
It is checked before it is written. A pack that fails its own verification is never produced, because handing someone evidence that does not verify is worse than handing them nothing.
And it is open-source and extractable without us. An evidence store you cannot get your records out of is one you cannot rely on, and any regulator asking about your exit plan will reach the same conclusion.
Said plainly, on the front page
What the chain proves, and what it does not.
Every record's hash covers its contents and carries the hash of the record before it. Change, remove or reorder one and every link after it breaks. So the pack shows these records have not been altered since they were sealed.
It does not show that whoever operates the store could not have re-sealed the whole chain and produced a different but internally consistent history. That needs signing and a witness outside the operator. Both shipped in Warrant 0.7.0, and a pack states on its own front page which level its chain reached and what that level does not prove. Even signed, an issuer holds its own key: only the witness closes that gap.
An evidence vendor whose operator could rewrite the evidence is not independent evidence. We would rather say that first than have your validation committee find it.
Where this stands
What you can install today.
Everything in this list is released, in warrantai 0.11.0 on PyPI and npm:
- warrant outcomes ingest: attach what actually happened from a file your operations team can produce
- warrant calibrate: Expected Calibration Error and reliability curves per decision type, segment and question-set version
- warrant pack: the records, the chain, the curve and the policy text, verifiable offline
- An adapter for TypeSafe's Jev, behind a model-agnostic decision-model interface
- Typed answers with their full distribution, and the question-set version, on every record
- Question sets as versioned artefacts, with a CI gate on editing a published one in place
- Golden sets: catch a model that changed underneath you, even when nothing flipped
- Effective-dated policies, so a decision is judged by the rules in force when it was made
- Circuit breakers on auto-share and escalation collapse
- Since 0.6.0, the Jev adapter records the probability of the chosen answer, not the vendor's margin between the top two
The SDK, the policy engine, the append-only store, replay, the collector, the offline verifier and the adapters for the Claude Agent SDK, LangGraph and Temporal shipped earlier; the overview lists them. New in 0.7.0:
- Issuer signatures on every sealed record, and checkpoints co-signed by an independent witness
- An adapter for Laya, an open-weight decision model that runs inside your own environment
Still to come:
- Calibration dashboards and drift incidents in the console (in early access)
- SSO and an auditor role, hosted India region
The figures above come from a stand-in model that is deliberately overconfident, running on synthetic alerts. They demonstrate the measurement, and they are not a finding about any real model or any real institution. A reliability curve is only worth reading when it was produced on real decisions with real outcomes, which is exactly why we will not print one here that was not.
If you are automating a decision you will have to defend.
We are looking for one or two teams running a high-volume judgment call they have to justify years later. You bring the decisions and the outcomes; we bring the record, the measurement and the pack. Shadow mode first, deciding nothing, for as long as it takes to see the curve.