chock eval

Syntax

This is the usage line argparse builds from this command's parser.

chock eval [-h] [--repo REPO] [--mode {deterministic,agent}] [--json] [--verbose]
           [policy_id]

Arguments and options

The table lists every argument the parser declares, except -h, --help, written as argparse prints it. Default is the value the parser uses when you leave the option out. It is empty for a flag or when the parser sets none.

Exit codes

• 2: argparse rejected the arguments, for example an option this command does not have. The site's tests run every command with an unknown option and check that it exits 2.
• Chock documents no other exit codes. Each example below shows the code its run returned.

Examples

Real runs, recorded by python -m sitegen.cliexamples with the engine at the pinned commit. The steps of data/cli-examples.yaml ran in order, offline, in one fresh git repository at /work/demo (made with git init -b main), with HOME at /home/demo. Those paths stand for the throwaway folders the runs used, and <time> stands for a timestamp. Output is stdout and stderr together.

Step 7 of 21, exit code 0.

$ chock eval

my-block-marker  [deterministic]
  PASS  tc-001                             authored  my-block-marker: this change adds a line with a DO-NOT-COMMIT marke…
  PASS  tc-002                             authored  gate exit 0
  PASS  tc-003                             authored  my-block-marker: this change adds a line with a DO-NOT-COMMIT marke…
  PASS  tc-004                             authored  gate exit 0
  score 1.00

5 policies: 4 pass, 44 skipped
44 case(s) have no executable form; they are agent-mode material (tier 3).

Help as printed

This is what --help prints, from the same parser. The usage line names the program as the engine's parser does.

usage: chock check --only evals [-h] [--repo REPO] [--mode {deterministic,agent}] [--json]
                                [--verbose]
                                [policy_id]

Run policy eval suites

positional arguments:
  policy_id             Policy to evaluate. Default: every policy in the repo.

options:
  -h, --help            show this help message and exit
  --repo REPO           Repository root (default: cwd)
  --mode {deterministic,agent}
  --json                Emit machine-readable results
  --verbose             Include skipped cases in the table