Evaluating agent backends: an open fitness framework

When we launched this blog, we said one of the first things we would publish was the framework we use to evaluate agent backends. This post is that framework. It is the tiering system and scorecard Cloister applies before any backend is routed into consequential work, extracted from our internal documentation and published here in full.

We are publishing it as a proposed open standard, not as marketing. As teams move from a single trusted model to fleets of backends — different models, CLI agents, hosted variants — there is no shared vocabulary for the question that actually matters: not which backend is generally capable, but which backend is fit for which class of work under governance. We would rather contribute the yardstick and let others critique and extend it than keep it internal. Nothing here is specific to Cloister's architecture; the tiers and scorecard work anywhere an orchestrator has to decide which agent to trust with which task.

Two backends can both satisfy the same agent interface and still differ sharply in task success rate, gate artifact quality, structured output reliability, tool-use safety, telemetry completeness, recovery behavior under replay, cost and latency, and how much operator attention they demand. Interface compatibility tells you a backend can be plugged in. It tells you nothing about whether it should be.

The correct unit of evaluation is therefore not the backend on its own. It is the combination of task class, backend, and rubric: how well a specific backend performs a specific class of work against a specific standard. Not all backends belong in all lanes.

One distinction anchors the whole framework. Some properties must stay invariant no matter which backend is running: enforcement of law, policy, and rule constraints; rubric pinning for the lifetime of a task; sensor persistence and replay safety; audit trail and provenance tracking; human checkpoint semantics; and outbox and reconciliation behavior for declared non-idempotent work. If a backend or integration path cannot preserve these, it is not merely lower quality — it is incompatible with governed execution, full stop.

Everything else is expected to vary materially by backend: completion quality, instruction-following under constraint, willingness to produce required artifacts, structured output correctness, tool-call accuracy, latency, token and API cost, health-monitor signatures, and the number of human interventions required. That variance is not an edge case. It is a central feature of operating a heterogeneous backend fleet, and the whole point of evaluation is to make it visible instead of discovering it in production.

We classify every backend into one of three explicit tiers.

Tier 1, Native, is for backends that support structured output directly, propagate operation IDs or idempotency, emit rich telemetry, integrate cleanly with tool-use and provenance boundaries, and support the recovery model without lossy adaptation. These are the primary backends for consequential work: the preferred routing targets, and the only tier eligible for sensitive task classes.

Tier 2, Governed-Compatible, covers backends that satisfy every required governance invariant but may lose optional capabilities such as streaming or rich telemetry, may require adapters for structured output or tool metadata, and recover with more friction or lower observability. These are secondary backends: usable for limited fallback and for moderate-risk task classes, but not the preferred target.

Tier 3, Best-Effort Compatibility, is for backends that are technically pluggable but cannot provide some important capabilities reliably. They work for low-risk or read-only tasks — exploratory work, low-risk analysis — and should never be the default for high-consequence work or trusted for consequential side effects.

The tiers are not a ranking of vendors. The same backend can be Native in one lane and Best-Effort in another. They are a routing discipline: they force the question of fitness to be answered per task class rather than assumed from capability lists.

Every backend under consideration is scored across eleven dimensions. The first group asks whether the backend stays inside the governance envelope: governance compliance (gate pass rate, forbidden-action rejection, artifact completeness) and task performance (success rate, reviewer acceptance, defect rate).

The second group asks how it behaves at the machine boundary and under failure: structured output reliability (schema pass rate, parse failure rate), tool-use safety (tool error rate, denied-call rate, side-effect incidents), recovery behavior under replay, retry, and fallback (duplicate side-effect rate, sensor replay correctness, outbox reconciliation burden), and capability preservation — what is lost on fallback or adapter paths, such as streaming, telemetry, or provenance.

The third group asks whether the backend is observable, economical, and operable: observability (telemetry completeness, trace coverage, audit readability), health profile over time (novelty floor, drift rate, halt frequency, abstention frequency), cost (median tokens, dollars per successful task), latency (p50 and p95 completion time, gate wait time), and operator burden (approvals per task, manual reconciliations, resumes per 100 runs).

Health signals deserve a note of their own, because they are easy to misread. If one backend repeatedly causes convergence warnings, abstentions, gate failures, or manual reconciliations, that may indicate a backend-task mismatch rather than a general problem with the system. Health should be analyzed by partition and by backend, not only at the global level.

We keep a scorecard per lane — a specific task class evaluated against a specific rubric on a specific backend and model variant. Each scorecard records the tier classification, a disposition of primary, fallback-only, or disallowed, and the measured evidence: success rate, gate artifact pass rate, structured output pass rate where the task requires it, recovery incidents, manual reconciliations, median latency, and cost per successful task, plus free-text notes.

A backend should not be routed into a task class unless it can demonstrate, for that class: no violation of the governance invariants, an acceptable task success rate, acceptable gate artifact quality, acceptable structured output reliability if the task requires it, acceptable operation ID propagation for any non-idempotent path, and acceptable telemetry completeness for health monitoring and audit.

Failure in those areas resolves the disposition, not into a vague 'worse' but into one of three explicit outcomes: primary (the preferred backend for the lane), fallback-only (usable only as a constrained backup), or disallowed (not permitted for that task class or rubric). Making the disposition explicit is what turns evaluation results into routing policy.

Start by defining the lane: declare the task class, the rubric, the backend and model or CLI variant, and whether side effects are allowed. Then build a fixed task corpus for the class, one that includes straightforward success cases, ambiguous cases that should trigger gates, cases with policy-required artifacts, cases that should be denied, and recovery cases involving retry, replay, timeout, and fallback. Without a fixed corpus, backend comparisons drift into anecdote.

Run multiple trials and record variance, not just mean performance — a single-run comparison cannot distinguish a one-off failure from a recurring mismatch. Test recovery explicitly by inducing failures on purpose: crash after the agent responds but before commit, sensor timeouts, late human approvals, fallback to a lower-capability backend, pending outbox on recovery, duplicate notification delivery. A backend is not fully evaluated until the recovery story has been exercised.

Finally, evaluate operator experience. Some backends are technically compliant but operationally exhausting. Record how often humans had to intervene, whether the interventions were understandable, whether recovery commands were sufficient, and whether audit records were usable without reading raw logs.

One of the framework's most important rules concerns fallback: fallback is not automatically safer than failure. A weaker backend can increase risk by dropping structured output, emitting worse telemetry, failing to produce required artifacts, interacting poorly with tools, or increasing human intervention frequency. Every fallback edge should be evaluated as its own lane, never assumed valid because both endpoints implement the same interface.

You do not need Cloister to apply any of this. If you route work to more than one model or agent, you can adopt the three tiers as routing vocabulary, build a per-lane scorecard for your own task classes, and write down the invariant-versus-variable distinction for your own stack: which properties must hold no matter which backend runs, and which ones you expect to vary and therefore need to measure. The scorecard's field list above is a starting point; the measures are the part worth debating and adapting to your domain.

We are treating this as a proposal for an open standard rather than a finished one, and we mean that literally: we want critique, alternate rubrics, and scorecards from teams operating fleets we will never see. If you apply the framework, publish your per-lane results or tell us what the tiers get wrong — the fastest way to improve it is to see it used somewhere other than here.

One closing claim, stated as carefully as we can. A governed execution substrate over a heterogeneous backend fleet does not make agents interchangeable. It makes their differences visible, governable, and operationally measurable. That is a stronger and more honest claim than pretending every backend is equivalent once it satisfies an interface — and it is the premise this framework exists to test, lane by lane, with evidence.