Skip to content

Guide

Cookbook

Reference

Concepts

Evaluation

Scoring whether an agent's output is actually good — not just schema-valid. The bar is "the system can gate a critical decision on a judgment call, not just a type check."

How reyn handles it

agent step + schema

There is no dedicated scorer op. Scoring an output against a rubric is ordinary pipeline composition: a pipeline agent step whose schema: names a small schema (e.g. {score: number, reason: string}), followed by a plain transform step that compares the parsed score against a threshold.

pipeline: self_review
steps:
  - agent:
      prompt: "Self-review {ctx.draft} against your own checklist: ... Give a score in [0.0, 1.0] and a short reason."
      schema: Verdict
      output: verdict
  - transform: {value: "ctx.verdict.score >= 0.6", output: passed}
---
schema: Verdict
fields:
  score: {type: number}
  reason: {type: string}

The schema: is the OS's actual contribution here: it constrains the agent's generation (a response_format built from the schema, so the model answers in schema-conforming JSON rather than free text) and validates the parsed result afterwards (belt-and-suspenders — the provider constraint is not blindly trusted). The threshold comparison is a plain if (a transform step), not a bespoke op. Cost is tracked the same way every other agent step's cost is tracked — no separate cost path to wire.

The OS never interprets the checklist content — it is the calling agent's own evaluation criteria, part of the prompt it writes. This is self-review, not objectivity: the same agent (or model family) that produced the draft also writes the checklist and scores against it — useful for catching requirements the checklist names and the draft missed, but not an independent judge. Do not present it as one.

reyn run-once

The non-interactive CLI entry point for running an agent without a live approval prompt (reyn eval was a phase-graph-era command; it was deleted alongside that engine — reyn run-once is its current, live counterpart). Permissions must already be pre-approved before the run starts — e.g. permissions.file.write declared in reyn.yaml grants a specific capability durably (#3924 removed the per-invocation --grant-file-write CLI flag this used to describe) rather than via an interactive prompt. This is what makes a self-review-gated pipeline usable in CI: the scoring loop and the permission model are orthogonal, so a non-interactive run's trust decisions are made once, up front, not re-litigated per invocation.

Where it's still thin

This is one of the constitution's two declared honest thin areas (see CLAUDE.md's Constitution section and docs/concepts/architecture/charter.md, Evaluation row). An agent step + schema is the entire evaluation surface — there is no rubric library, no multi-judge consensus/voting, no built-in eval-suite runner, and no aggregate scoring across a batch of runs. An author who wants any of that composes it themselves out of agent+schema self-review steps plus ordinary pipeline control flow; the OS provides the typed-generation primitive, not an evaluation framework built on top of it.

See also

On this page