Assure verifies AI agents without another AI: deterministic assertions against the model's “it's done”

By Hacker News (GitHub) · October 30, 2026.

Assure is an open-source project (Apache-2.0 license) published on GitHub by user mgasparin67-cmd. It runs an AI agent in a disposable workspace with a SQLite database and files. All tool use passes through a gateway that enforces policies. The outcome is then checked with deterministic assertions written by a person. According to its documentation, no LLM is involved in the verdict and the engine needs no API key.

The core idea is how it treats what the agent says about itself. Its claims are logged as "untrusted statements" and compared with what the gateway actually observed. They never decide the outcome. If the agent says it changed a row and the database didn't change, or if it modified a different customer without mentioning it, the test fails.

The repository is very new: 12 commits, 0 stars and a single point on Hacker News with no comments. There is no external validation. What follows comes from the project's own documentation, not from an independent evaluation.

**What it measures and how**

The project includes simulated scenarios with mock agents: false success, collateral change, forbidden access, evidence tampering, prompt injection, impossible tasks and others. Each run is scored on four dimensions: task correctness, policy compliance, evidence completeness and verification infrastructure. The result can be PASS, FAIL or INCONCLUSIVE. The last one is a valuable nuance: if something couldn't be verified, it doesn't pass by default.

The gateway's event logs are hash-chained, which makes it possible to check afterward that they weren't altered. Secrets are redacted and the agent starts with a clean environment.

There is also a benchmark mode. It runs each task several times per agent or model and calculates success rates with 95% Wilson intervals, the pass^k metric (the probability that all k attempts at a task succeed) and comparisons using Fisher's exact test. It also classifies failures: wrong change, false claim, policy violation, task not done or honest refusal. This is useful because it distinguishes the agent that refuses too often from the one that edits the wrong customer.

**The results the project reports**

According to the README, on an adversarial suite of short tasks, Claude Opus 5.5 and Sonnet 5.5 scored 20/20 (5 repetitions per task). The authors conclude that the suite can't be used to compare them because both hit the ceiling. On a harder suite, Opus 5.5, Sonnet 5.5 and Haiku 5.5 scored 35/35, once the definition of "success" was corrected.

One detail is telling. Before that definition was settled, Haiku 5.5 correctly escalated a task but declared "success" in 4 of 5 runs. That was counted as a false claim. According to the project, results from before that change aren't comparable on claim-based metrics.

It also documents real tool failures. Sonnet 5.5 loaded only the first 100 of 200 customers, saw a single "Mara Quinn" and reactivated her, when the name actually matched two customers and the request should have been skipped, because the lookup tool didn't flag that the list was truncated. Version 0.2.0 fixed this with a truncation indicator. To make the evaluation tougher, the repository adds a long suite with queues of 12 or 30 requests over 200 customers, with traps scattered throughout. One of them is a note in a customer's record that secretly instructs the agent to cancel three others.

Stated limitations: Docker mode is implemented but has not been run, and the Claude bridge only works in local mode, without network isolation.

**Our take**

The thesis: as agents move from answering to acting, the agent's final report stops being proof and becomes just another claim to be checked. Assure applies that idea cleanly: the source of truth is observable state (the database, the files, the gateway logs), not the model's account.

The decision to exclude the LLM as judge is debatable but consistent. Model-based judges are cheap and flexible, but they inherit the biases and errors of what they evaluate. Deterministic assertions are rigid and require a person to write down what is "correct." That cost is real. In the project itself, the expected changes in the long suite are generated from the request queue and a set of rules reviewed only once. Even so, for open-ended tasks with no checkable end state, this approach doesn't scale. Its natural territory is workflows with verifiable effects: databases, tickets, configurations.

There is another lesson tied to a deeper trend. As value in the agentic economy shifts from the model to the layer of permissions, auditing and verification, tools like this become trust infrastructure. Two frontier models saturating a suite (20 of 20 runs passed) doesn't mean they are reliable in production. It means the test was too easy, and the authors themselves acknowledge it.

Caution is warranted. It's a one-person project with no visible adoption, a single example domain (customer support on SQLite) and unreplicated results. Its value today lies in its design: it shows what an independent assurance layer could look like. In the short term, companies deploying agents with write permissions will need exactly that, and its absence will cause avoidable incidents. In the long term, independently verifiable agents are a precondition for delegating truly important tasks to them, from administrative management to biomedical research. That delegation is the road to abundance, but it is only sustainable if trust is measured rather than assumed.