Catalog

Add a policy to the catalog

Add a new policy to chock-catalog step by step: folder, manifest, implementation, evals, the generators, and the CI jobs a pull request runs.

View as Markdown

A catalog policy is one folder: a manifest, its evals and, for a guard, its implementation. The catalog's tools generate the rest and its CI checks it. The steps below follow the catalog's CONTRIBUTING.md and tools at the pinned commit, and were dry-run against it.

From the catalog docs: CONTRIBUTING.md

The three rules

All are mechanical, and all are checked by a machine rather than by a reviewer's memory: rules 1 and 2 by CI on every PR, rule 3 by the nightly run against main and by the release.

1. A policy claims only what it can do. If it does not exit non-zero, it is advisory, and the tooling labels it that way whatever the manifest says. 25 of the 72 policies here are advisory, and the README says so in its third paragraph rather than its appendix. Do not raise a claim your mechanism cannot support.

2. Evals are the argument. A gate whose declared behaviour does not match its suite fails CI. Derived cases alone are not enough — attestation needs at least one authored case that could have failed.

3. A new or changed policy ships its adoption transcript. Run

python tools/gen_adoption_transcript.py --policy <your-policy-id>

(python tools/regen_all.py runs it for every policy you changed) and commit docs/<id>/adoption.md in the same PR. The tool adopts your policy into an empty repository — init, copy, recompile, install-hooks, validate, eval — and records what the tools printed. PR CI does not run it: a missing or stale transcript does not fail a PR. The nightly workflow runs python tools/gen_adoption_transcript.py --check against main and fails on a transcript that does not reproduce, and the release regenerates every transcript and checks it — so pasted or edited output is caught there, because the transcript's value is that it cannot be written by hand.

Add a policy, step by step

1. Clone the catalog and install its engine

The catalog's CI installs the engine at the commit named in its .framework-ref file. Install the same one, then the test tools:

git clone https://github.com/open-coder-ai/chock-catalog
cd chock-catalog
pip install "chock @ git+https://github.com/open-coder-ai/chock@$(cat .framework-ref)"
pip install --require-hashes -r requirements/test.txt

2. Pick a tree

A policy lives in one of the trees listed under Policy trees. The layout in CONTRIBUTING.md names base/<id>/. The catalog runs a subset of base/ on itself. The catalog publishes the compliance and agentic-security packs but does not install them in itself.

3. Scaffold the folder

chock new policy starts from a tested template of one kind: content_regex, guard, script, rule or skill. Inside the catalog, chock keeps the id as given. It writes the folder to .agents/policies/<id>/, the policies the catalog runs on itself, and refreshes that folder's index and .chock/registry.json. Move the new folder into its tree and put those three files back:

chock new policy <id> --kind content_regex --repo .
mv .agents/policies/<id> base/<id>
git restore .agents/policies/INDEX.md .agents/policies/INDEX-extended.md .chock/registry.json

The content_regex template writes manifest.yaml and evals/suite.yaml. CONTRIBUTING.md also offers the policy-init skill that chock init installs: ask your agent to run it. Its procedure writes to <repo_root>/generated/ unless told otherwise, so move its folder into the tree the same way.

4. Write the manifest

manifest.yaml holds the policy's identity and either a gate or rule text. The engine's manifest schema lists every field. The template's provenance names author: "me" and license: "proprietary". The catalog's own manifests, such as base/protect-main-branch/manifest.yaml, name license: "Apache-2.0" and chock-catalog as source_repo.

5. Add the implementation, if it has one

A guard consulted before a command runs is a script under implementations/. The catalog's CODEOWNERS file requests review for every change there. Its tests go in tests/, held to 100% line and branch coverage.

6. Write the evals

evals/suite.yaml holds the cases that decide whether the claim holds. chock check replays a case with an execute block against the real gate. The engine's evals guide describes a suite.

7. Add the prose entry

Add <id>: with solves and customise to docs/policy-prose.yaml. Without it, tools/gen_policy_docs.py stops with docs/policy-prose.yaml has no entry for '<id>'.

8. Add the README row

tools/check_readme.py fails until the README lists the policy as a row in its tier's table, for a gate under Enforced at commit.

9. Regenerate and check

python tools/regen_all.py regenerates every derived file and then runs every check CI runs. It ends == CLEAN or names the step that failed. --fast skips pytest with coverage and the staged adopter. What it runs is under What regen_all.py runs below.

python tools/regen_all.py --fast

10. Run the tests a pull request runs

tools/select_tests.py prints FULL, or the test files CI's targeted jobs run for your change, one per line. Run those files with pytest, or the whole suite for FULL:

python tools/select_tests.py origin/main
python -m pytest -n auto <the files it printed>

11. Commit with a sign-off and open a pull request

The dco job checks every commit for a Signed-off-by: line. git commit -s adds it.

Policy trees

These are the trees tools/trees.py names at the pinned commit, and how many policy folders each holds.

Tree	Policies
base	54
compliance	3
agentic-security	15

From the catalog docs: CONTRIBUTING.md

Layout

One folder per policy under base/<id>/:

base/<id>/
  manifest.yaml           identity, and either a gate or rule text
  implementations/*.sh    guard script, for policies consulted before a command runs
  evals/suite.yaml        the cases that decide whether the claim holds

docs/<id>/README.md is generated — edit manifest.yaml and docs/policy-prose.yaml, then run python tools/gen_policy_docs.py.

New policy

chock new policy writes the installed tree (.agents/policies/<id>/) and re-indexes it, not the catalog source, so move the folder to base/ and put the index files back:

chock new policy <id> --kind content_regex   # or guard, script, rule, skill: a tested template
mv .agents/policies/<id> base/<id>
git restore .agents/policies/INDEX.md .agents/policies/INDEX-extended.md .chock/registry.json

Then edit base/<id>/manifest.yaml and evals/suite.yaml, add a <id>: entry (solves, customise) to docs/policy-prose.yaml and a row for it in the README's table for its kind, and run python tools/regen_all.py until it ends == CLEAN.

A new policy's builder link pins a commit that holds the policy's files, generated ones included, so the link can only be right after they are committed. Run python tools/regen_all.py, commit base/<id>, then run python tools/gen_policy_docs.py again; until then python tools/builder_link.py lists the page.

What CI runs

Two things, kept apart on purpose:

What this repo runs — the subset of base/ that applies to this repository, installed in .agents/policies/ to govern the catalog itself (four do not apply here: chock-mise, java-security, no-a11y-regression and rtk-dangerous-actions-blocker): the repo protects itself the same way it asks adopters to. The compliance and agentic-security packs stay uninstalled — by their own doctrine they only earn their place where they apply. This repo is a Chock adopter, and the first commit after adoption was rejected by protect-main-branch.
What this repo ships — every published policy, staged into a throwaway repo the way an adopter installs them (where CI also re-derives the README's OWASP coverage claim).

Runs fewer than it ships: a catalog should publish more than it is bound by.

Pull requests change sources; docs and images lag until the release. A PR changes base/<id>/**, tests, tools and lib, and regenerates only what adopters read straight from main: the touched policy's plugin.json, SKILL.md, manifest and registry.yaml row. CI checks those. The README counts and rows, the docs/ pages, figures, brand card, coverage matrix and adoption transcripts are not checked on a PR and may be stale on main. The owner's release regenerates them in one PR. The nightly workflow runs their checks against main (python tools/regen_all.py --drift), lists the stale files in the job summary, and fails while any is stale. python tools/regen_all.py still refreshes everything locally.

What regen_all.py runs

The docstring of tools/regen_all.py at the pinned commit:

Regenerate every derived file in dependency order, then run every check CI runs, once.

    python tools/regen_all.py                # regenerate, then check everything CI checks
    python tools/regen_all.py --fast         # the same, without pytest --cov and the staged adopter
    python tools/regen_all.py --check-only   # write nothing; run the checks
    python tools/regen_all.py --drift        # write nothing; check only the files that lag until the release
    python tools/regen_all.py --base main    # the ref whose diff picks the transcripts to re-make

Order is load-bearing: lib/ copies -> plugin packages -> sync (the lockfile hashes packaged files) -> registry
and README counts -> docs, matrix, figures, brand card (read the registry) -> adoption
transcripts (adopt the packaged folder). Checks run cheapest first, in parallel; transcripts
are checked only once the plugin packages are known current.

These are the scripts its regenerate() runs, in the order they appear in it. It also runs chock plugin build for each tree and chock sync when the compiled output has drifted.

tools/gen_lib_copies.py
tools/gen_registry.py
tools/gen_policy_docs.py
tools/gen_coverage_matrix.py
tools/gen_java_security_contract.py
tools/gen_quickstart_sh.py
docs/assets/gen_brand_assets.py
tools/gen_adoption_transcript.py

CI jobs

This table lists every job of the catalog's .github/workflows/ci.yml at the pinned commit, with its condition and its named steps. A job with no condition runs on every run of the workflow. No job runs tools/gen_adoption_transcript.py. The nightly and release workflows do.

Job	Runs when	Steps
dco	github.event_name == 'pull_request' \|\| github.event_name == 'merge_group'	Check Signed-off-by on every PR commit
no-private-data	github.event_name == 'pull_request'	Scan commit messages for local or personal data
scope	always	Restore the file times main last recorded; Plan this run
windows-adopter	needs.scope.outputs.windows == 'true'	Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock; This repo is a valid adopter on Windows
generated	always	Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the lint tooling; Registry matches the policies on disk; Installed policies trail, never lead, what this catalog publishes; Workflow triggers cannot hand fork code our secrets; Compiled artifacts match their manifests; Published policies are current Agent Plugins packages; Lint (ruff, the framework's rule set); Formatting (ruff format)
adopter	always	Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock; This repo is a valid adopter; Declared effects match what the guards actually do; Reviewer evidence, if the PR carries any
tests-shard	needs.scope.outputs.full == 'true'	Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the test tooling; Tests, one shard (every policy, every rule)
durations	always() && github.event_name == 'push' && needs.scope.outputs.full == 'true'	Restore the file times main last recorded; Sum the junit times by test file and keep each file's longest test; Slow-test report; Save the file times for the next run; No single test takes longer than the cap
tests-policy	needs.scope.outputs.full == 'false' && needs.scope.outputs.has_matrix == 'true'	Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the test tooling; Tests of this policy, its implementations at 100% line and branch coverage
tests-targeted	needs.scope.outputs.full == 'false'	Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the test tooling; Changed tests no policy job owns, and repo standards
tests	always()	Every test job this run needed passed; Install the coverage tooling; Combine the shards, 100% line and branch coverage
staged-adopter	always	Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock; Stage the published policies and skills as an adopter would; Validate the staged policies and replay their evals; OWASP ASI coverage claim holds in the staged adopter
policies	always()	Every policies job passed

Built from engine 5a4b6287a46e and catalog 1e60d211c1bb.