# Evals reference

How a policy's evals/suite.yaml is laid out, what each case says, and every field of the engine's eval schema at the pinned commit.

Every policy ships an eval suite at \`evals/suite.yaml\`. \`chock eval\` runs it, and replays a case with an \`execute\` block against the policy's real gate. The page quotes the engine's own guide first. The table after it comes from the engine's eval schema at the pinned commit.

> Ask the user before running any command on this page: each one changes how their agent or repository behaves.

From the engine docs: [docs/evals.md](https://github.com/open-coder-ai/chock/blob/5a4b6287a46e07f9f856d9bbe10622bd1ff2e021/docs/evals.md)

## Anatomy of a suite

Ask the user before running the commands below:

```yaml
suite:
  id: block-console-log-tests-v1
  policy_id: block-console-log
  version_constraint: ">=0.1.0"
  maintainer: you
  primary_metric: pass_rate
  metrics:
    pass_rate:
      direction: higher_is_better
      threshold: 1.0
  cases:
    - id: tc-001
      category: trigger
      prompt: A staged diff adds `console.log("debug")`.
      expect: The hook blocks the commit with an actionable message.
    - id: tc-002
      category: negative_trigger
      prompt: A staged diff adds a normal function with no console.log.
      expect: The hook allows the commit.
    - id: tc-003
      category: behavior
      prompt: Two files are staged; only one adds console.log.
      expect: The hook blocks and names the offending file.
    - id: tc-004
      category: adversarial
      prompt: The log call is written as `console['log']('x')`.
      expect: The hook still detects and blocks it, or the limitation is documented.
```

## The four categories

| Category | Question it answers |
| :--- | :--- |
| `trigger` | Does the policy fire on the thing it targets? |
| `negative_trigger` | Does it stay quiet on adjacent, legitimate input? |
| `behavior` | Does it do the *right* thing (message, scope) when it fires? |
| adversarial | Does it resist obvious bypasses — or honestly document the gap? |

The validator requires **at least one `trigger`, one `negative_trigger`, and one `behavior`** case
(a minimum of three). Adding an adversarial case is strongly encouraged for any `block` policy.

## Which gate a case is replayed against

`chock check --only evals` runs each case against the **compiled gate** in
`.chock/compiled/<id>/git-hook/gate.json` — the artifact your git hook and CI gate actually
execute — not against a gate rebuilt from the manifest.

That distinction is the whole difference between a suite that vouches for what runs and one that
vouches for what was written down. Replaying a freshly derived gate meant a stale or hand-edited
compiled artifact could enforce something else entirely while its own suite reported it passing.

Two exceptions, both deliberate:

- **Not compiled yet** — the case is replayed against the manifest and the result is marked
  `[gate derived from manifest; policy not compiled]`. `chock new` writes a policy and its
  suite before any compile, and an author iterating on a gate should not have to install it first.
- **Compiled but unreadable** — reported as an `error`, never quietly replaced by the manifest. A
  compiled gate that cannot be parsed is a broken installed control; falling back would turn that
  into a passing suite.

Drift between a manifest and its compiled artifact is caught independently by `chock
validate`, `chock check --only verify`, and `chock sync --repo . --check`.

## Every field

Generated from `eval.schema.json` as installed from the pinned engine commit, with every `$ref` followed into the engine's other schema files. A field `a.b` is `b` inside `a`. A field `a[].b` is `b` in each item of the list `a`. A field `a.<name>.b` is `b` under any key of `a`. Required means required when its parent is present. The first table is the top level. Each object below it has its own table.

The schema file at the pinned commit: [src/chock/validation/schemas/eval.schema.json](https://github.com/open-coder-ai/chock/blob/5a4b6287a46e07f9f856d9bbe10622bd1ff2e021/src/chock/validation/schemas/eval.schema.json)

| Field | Type | Required | Allowed values | Description in the schema |
| --- | --- | --- | --- | --- |
| `eval_suite` | object | no |  |  |
| `suite` | object | no |  |  |

### `eval_suite`

| Field | Type | Required | Allowed values | Description in the schema |
| --- | --- | --- | --- | --- |
| `eval_suite.id` | string | yes |  |  |
| `eval_suite.skill_id` | string | yes |  |  |
| `eval_suite.skill_version_constraint` | string | no |  |  |
| `eval_suite.description` | string | no |  |  |
| `eval_suite.created_at` | string | no | Format date-time |  |
| `eval_suite.maintainer` | string | no |  |  |
| `eval_suite.metrics` | object | yes |  |  |
| `eval_suite.metrics.<name>.type` | string | yes | One of: "nlp_overlap", "llm_judge", "performance", "binary", "numeric_tolerance" |  |
| `eval_suite.metrics.<name>.description` | string | no |  |  |
| `eval_suite.metrics.<name>.implementation` | string | no |  |  |
| `eval_suite.metrics.<name>.judge_model` | string | no |  |  |
| `eval_suite.metrics.<name>.judge_prompt_ref` | string | no |  |  |
| `eval_suite.metrics.<name>.direction` | string | yes | One of: "higher_is_better", "lower_is_better" |  |
| `eval_suite.metrics.<name>.production_threshold` | number | no |  |  |
| `eval_suite.metrics.<name>.staging_threshold` | number | no |  |  |
| `eval_suite.metrics.<name>.unit` | string | no |  |  |
| `eval_suite.test_cases` | Array of object | yes | Min items 3 |  |
| `eval_suite.test_cases[].id` | string | yes |  |  |
| `eval_suite.test_cases[].category` | string | yes | One of: "trigger", "negative_trigger", "behavior", "edge", "adversarial", "security", "multilingual", "validation", "correctness" |  |
| `eval_suite.test_cases[].description` | string | yes |  |  |
| `eval_suite.test_cases[].input` | object | yes |  |  |
| `eval_suite.test_cases[].expected_output` | object | no |  |  |
| `eval_suite.test_cases[].expected_behavior` | string | no |  |  |
| `eval_suite.test_cases[].expected_error_pattern` | string | no |  |  |
| `eval_suite.test_cases[].reference_summary` | string | no |  |  |
| `eval_suite.test_cases[].acceptance_criteria` | object | no |  |  |
| `eval_suite.test_cases[].structural_checks` | Array of string | no |  |  |
| `eval_suite.test_cases[].language_checks` | Array of string | no |  |  |
| `eval_suite.test_cases[].pii_checks` | Array of string | no |  |  |

### `suite`

| Field | Type | Required | Allowed values | Description in the schema |
| --- | --- | --- | --- | --- |
| `suite.id` | string | yes |  |  |
| `suite.policy_id` | string | yes |  |  |
| `suite.version_constraint` | string | no |  |  |
| `suite.maintainer` | string | no |  |  |
| `suite.primary_metric` | string | no |  |  |
| `suite.metrics` | object | yes |  |  |
| `suite.metrics.<name>.direction` | string | no | One of: "higher_is_better", "lower_is_better" |  |
| `suite.metrics.<name>.threshold` | number | no |  |  |
| `suite.metrics.<name>.weight` | number | no |  |  |
| `suite.cases` | Array of object | yes | Min items 3 |  |
| `suite.cases[].id` | string | yes |  |  |
| `suite.cases[].category` | string | yes | One of: "trigger", "implicit", "negative_trigger", "behavior", "edge", "adversarial", "security" |  |
| `suite.cases[].prompt` | string | yes |  |  |
| `suite.cases[].expect` | string | yes |  |  |
| `suite.cases[].status` | string | no | One of: "pending", "ready" | Pending = placeholder, never counts as a pass and blocks attestation. Default: ready. |
| `suite.cases[].provenance` | string | no | One of: "authored", "derived" | Derived = generated from the gate declaration (weaker evidence). Default: authored. |
| `suite.cases[].execute` | object | no |  | Executable form of the case, replayed by \`chock check --only evals --mode deterministic\`. Absent means the expectation is behavioural and belongs to agent mode. |
| `suite.cases[].execute.expect` | string | yes | One of: "block", "allow", "ask", "warn" | The verdict the replay must observe. ask: exit 3 of a guard, script or agent-event gate, or an ask gate refusing at commit or push. warn: exit 4 of a script or agent-event gate, or a warn gate at any event; at stop an ask gate warns. |
| `suite.cases[].execute.command` | string | no |  | Argv passed to the policy's guard script. Mutually exclusive with the gate fields. |
| `suite.cases[].execute.files` | object | no |  | Paths staged for the commit under test (or the pre-commit, pre-push and commit-msg script), as path -\> content. |
| `suite.cases[].execute.head_files` | object | no |  | Paths committed before staging, so a case can distinguish a dependency this commit ADDS from one already present. |
| `suite.cases[].execute.repo_files` | object | no |  | Untracked files the gate reads but the commit does not contain, e.g. an allowlist. |
| `suite.cases[].execute.branch` | string | no |  |  |
| `suite.cases[].execute.event` | string | no | One of: "commit", "push", "tool_use", "pre-tool-use", "stop", "pre-commit", "pre-push", "commit-msg" | Commit and push replay the policy's gate as the git hook would. tool_use (alias pre-tool-use) and stop replay it through the gate runner at that agent event. pre-commit, pre-push and commit-msg run the policy's own event script in the staged throwaway repo. |
| `suite.cases[].execute.writes` | object | no |  | Tool_use and stop only: each file as the agent would leave it, path -\> content; not staged. \`files\` is read the same way when \`writes\` is absent. head_files are committed first, so a waiver can be tested against HEAD. |
| `suite.cases[].execute.added` | object | no |  | Tool_use and stop only: the text an edit introduces, path -\> text, for a gate that judges an edit's own lines. |
| `suite.cases[].execute.message` | string | no |  | Commit-msg only: the commit message text, handed to the script as the message file. |
| `suite.cases[].execute.stdin` | Array of string | no |  | Pre-push script only: the stdin lines git feeds the hook, one \`\<local ref\> \<local sha\> \<remote ref\> \<remote sha\>\` each. |
| `suite.cases[].execute.push_refs` | Array of string | no |  |  |

## Run them

`chock eval` runs every policy's suite, or one policy's when given its id. `chock check --repo .` runs them too, after validation. A recorded run of both is on [Your own policy](https://chock.sh/docs/write/own-policy/index.md). Every option is on [chock eval](https://chock.sh/docs/cli/eval/index.md).


---

Built from engine [5a4b6287a46e](https://github.com/open-coder-ai/chock/commit/5a4b6287a46e07f9f856d9bbe10622bd1ff2e021) and catalog [1e60d211c1bb](https://github.com/open-coder-ai/chock-catalog/commit/1e60d211c1bb767152c6ab0167932da06ab3a10f).