# Add a policy to the catalog Add a new policy to chock-catalog step by step: folder, manifest, implementation, evals, the generators, and the CI jobs a pull request runs. A catalog policy is one folder: a manifest, its evals and, for a guard, its implementation. The catalog's tools generate the rest and its CI checks it. The steps below follow the catalog's CONTRIBUTING.md and tools at the pinned commit, and were dry-run against it. > Ask the user before running any command on this page: each one changes how their agent or repository behaves. From the catalog docs: [CONTRIBUTING.md](https://github.com/open-coder-ai/chock-catalog/blob/1e60d211c1bb767152c6ab0167932da06ab3a10f/CONTRIBUTING.md) ## The three rules All are mechanical, and all are checked by a machine rather than by a reviewer's memory: rules 1 and 2 by CI on every PR, rule 3 by the nightly run against `main` and by the release. **1. A policy claims only what it can do.** If it does not exit non-zero, it is advisory, and the tooling labels it that way whatever the manifest says. 25 of the 72 policies here are advisory, and the README says so in its third paragraph rather than its appendix. Do not raise a claim your mechanism cannot support. **2. Evals are the argument.** A gate whose declared behaviour does not match its suite fails CI. Derived cases alone are not enough — attestation needs at least one authored case that could have failed. **3. A new or changed policy ships its adoption transcript.** Run Ask the user before running the commands below: ```bash python tools/gen_adoption_transcript.py --policy ``` (`python tools/regen_all.py` runs it for every policy you changed) and commit `docs//adoption.md` in the same PR. The tool adopts your policy into an empty repository — init, copy, recompile, install-hooks, validate, eval — and records what the tools printed. PR CI does not run it: a missing or stale transcript does not fail a PR. The nightly workflow runs `python tools/gen_adoption_transcript.py --check` against `main` and fails on a transcript that does not reproduce, and the release regenerates every transcript and checks it — so pasted or edited output is caught there, because the transcript's value is that it *cannot* be written by hand. ## Add a policy, step by step ### 1. Clone the catalog and install its engine The catalog's CI installs the engine at the commit named in its `.framework-ref` file. Install the same one, then the test tools: Ask the user before running the commands below: ```bash git clone https://github.com/open-coder-ai/chock-catalog cd chock-catalog pip install "chock @ git+https://github.com/open-coder-ai/chock@$(cat .framework-ref)" pip install --require-hashes -r requirements/test.txt ``` ### 2. Pick a tree A policy lives in one of the trees listed under [Policy trees](#policy-trees). The layout in CONTRIBUTING.md names `base//`. The catalog runs a subset of `base/` on itself. The catalog publishes the `compliance` and `agentic-security` packs but does not install them in itself. ### 3. Scaffold the folder `chock new policy` starts from a tested template of one kind: `content_regex`, `guard`, `script`, `rule` or `skill`. Inside the catalog, chock keeps the id as given. It writes the folder to `.agents/policies//`, the policies the catalog runs on itself, and refreshes that folder's index and `.chock/registry.json`. Move the new folder into its tree and put those three files back: Ask the user before running the commands below: ```bash chock new policy --kind content_regex --repo . mv .agents/policies/ base/ git restore .agents/policies/INDEX.md .agents/policies/INDEX-extended.md .chock/registry.json ``` The `content_regex` template writes `manifest.yaml` and `evals/suite.yaml`. CONTRIBUTING.md also offers the `policy-init` skill that `chock init` installs: ask your agent to run it. Its procedure writes to `/generated/` unless told otherwise, so move its folder into the tree the same way. ### 4. Write the manifest `manifest.yaml` holds the policy's identity and either a gate or rule text. The engine's [manifest schema](https://github.com/open-coder-ai/chock/blob/5a4b6287a46e07f9f856d9bbe10622bd1ff2e021/src/chock/validation/schemas/manifest.schema.json) lists every field. The template's `provenance` names `author: "me"` and `license: "proprietary"`. The catalog's own manifests, such as `base/protect-main-branch/manifest.yaml`, name `license: "Apache-2.0"` and chock-catalog as `source_repo`. ### 5. Add the implementation, if it has one A guard consulted before a command runs is a script under `implementations/`. The catalog's CODEOWNERS file requests review for every change there. Its tests go in `tests/`, held to 100% line and branch coverage. ### 6. Write the evals `evals/suite.yaml` holds the cases that decide whether the claim holds. `chock check` replays a case with an `execute` block against the real gate. The engine's [evals guide](https://github.com/open-coder-ai/chock/blob/5a4b6287a46e07f9f856d9bbe10622bd1ff2e021/docs/evals.md) describes a suite. ### 7. Add the prose entry Add `:` with `solves` and `customise` to `docs/policy-prose.yaml`. Without it, `tools/gen_policy_docs.py` stops with `docs/policy-prose.yaml has no entry for ''`. ### 8. Add the README row `tools/check_readme.py` fails until the README lists the policy as a row in its tier's table, for a gate under **Enforced at commit**. ### 9. Regenerate and check `python tools/regen_all.py` regenerates every derived file and then runs every check CI runs. It ends `== CLEAN` or names the step that failed. `--fast` skips pytest with coverage and the staged adopter. What it runs is under [What regen_all.py runs](#what-regen-all-py-runs) below. Ask the user before running the commands below: ```bash python tools/regen_all.py --fast ``` ### 10. Run the tests a pull request runs `tools/select_tests.py` prints `FULL`, or the test files CI's targeted jobs run for your change, one per line. Run those files with pytest, or the whole suite for `FULL`: Ask the user before running the commands below: ```bash python tools/select_tests.py origin/main python -m pytest -n auto ``` ### 11. Commit with a sign-off and open a pull request The `dco` job checks every commit for a `Signed-off-by:` line. `git commit -s` adds it. ## Policy trees These are the trees `tools/trees.py` names at the pinned commit, and how many policy folders each holds. | Tree | Policies | | --- | --- | | base | 54 | | compliance | 3 | | agentic-security | 15 | From the catalog docs: [CONTRIBUTING.md](https://github.com/open-coder-ai/chock-catalog/blob/1e60d211c1bb767152c6ab0167932da06ab3a10f/CONTRIBUTING.md) ## Layout One folder per policy under `base//`: Ask the user before running the commands below: ``` base// manifest.yaml identity, and either a gate or rule text implementations/*.sh guard script, for policies consulted before a command runs evals/suite.yaml the cases that decide whether the claim holds ``` `docs//README.md` is generated — edit `manifest.yaml` and `docs/policy-prose.yaml`, then run `python tools/gen_policy_docs.py`. ### New policy `chock new policy` writes the installed tree (`.agents/policies//`) and re-indexes it, not the catalog source, so move the folder to `base/` and put the index files back: Ask the user before running the commands below: ```bash chock new policy --kind content_regex # or guard, script, rule, skill: a tested template mv .agents/policies/ base/ git restore .agents/policies/INDEX.md .agents/policies/INDEX-extended.md .chock/registry.json ``` Then edit `base//manifest.yaml` and `evals/suite.yaml`, add a `:` entry (`solves`, `customise`) to `docs/policy-prose.yaml` and a row for it in the README's table for its kind, and run `python tools/regen_all.py` until it ends `== CLEAN`. A new policy's builder link pins a commit that holds the policy's files, generated ones included, so the link can only be right after they are committed. Run `python tools/regen_all.py`, commit `base/`, then run `python tools/gen_policy_docs.py` again; until then `python tools/builder_link.py` lists the page. ## What CI runs Two things, kept apart on purpose: - **What this repo runs** — the subset of `base/` that applies to this repository, installed in `.agents/policies/` to govern the catalog itself (four do not apply here: `chock-mise`, `java-security`, `no-a11y-regression` and `rtk-dangerous-actions-blocker`): the repo protects itself the same way it asks adopters to. The compliance and agentic-security packs stay uninstalled — by their own doctrine they only earn their place where they apply. This repo is a Chock adopter, and the first commit after adoption was rejected by `protect-main-branch`. - **What this repo ships** — every published policy, staged into a throwaway repo the way an adopter installs them (where CI also re-derives the README's OWASP coverage claim). Runs fewer than it ships: a catalog should publish more than it is bound by. **Pull requests change sources; docs and images lag until the release.** A PR changes `base//**`, tests, tools and lib, and regenerates only what adopters read straight from main: the touched policy's `plugin.json`, `SKILL.md`, manifest and `registry.yaml` row. CI checks those. The README counts and rows, the `docs/` pages, figures, brand card, coverage matrix and adoption transcripts are not checked on a PR and may be stale on main. The owner's release regenerates them in one PR. The nightly workflow runs their checks against main (`python tools/regen_all.py --drift`), lists the stale files in the job summary, and fails while any is stale. `python tools/regen_all.py` still refreshes everything locally. ## What regen_all.py runs The docstring of `tools/regen_all.py` at the pinned commit: Ask the user before running the commands below: ```text Regenerate every derived file in dependency order, then run every check CI runs, once. python tools/regen_all.py # regenerate, then check everything CI checks python tools/regen_all.py --fast # the same, without pytest --cov and the staged adopter python tools/regen_all.py --check-only # write nothing; run the checks python tools/regen_all.py --drift # write nothing; check only the files that lag until the release python tools/regen_all.py --base main # the ref whose diff picks the transcripts to re-make Order is load-bearing: lib/ copies -> plugin packages -> sync (the lockfile hashes packaged files) -> registry and README counts -> docs, matrix, figures, brand card (read the registry) -> adoption transcripts (adopt the packaged folder). Checks run cheapest first, in parallel; transcripts are checked only once the plugin packages are known current. ``` These are the scripts its `regenerate()` runs, in the order they appear in it. It also runs `chock plugin build` for each tree and `chock sync` when the compiled output has drifted. - `tools/gen_lib_copies.py` - `tools/gen_registry.py` - `tools/gen_policy_docs.py` - `tools/gen_coverage_matrix.py` - `tools/gen_java_security_contract.py` - `tools/gen_quickstart_sh.py` - `docs/assets/gen_brand_assets.py` - `tools/gen_adoption_transcript.py` ## CI jobs This table lists every job of the catalog's `.github/workflows/ci.yml` at the pinned commit, with its condition and its named steps. A job with no condition runs on every run of the workflow. No job runs `tools/gen_adoption_transcript.py`. The nightly and release workflows do. | Job | Runs when | Steps | | --- | --- | --- | | dco | github.event_name == 'pull_request' \|\| github.event_name == 'merge_group' | Check Signed-off-by on every PR commit | | no-private-data | github.event_name == 'pull_request' | Scan commit messages for local or personal data | | scope | always | Restore the file times main last recorded; Plan this run | | windows-adopter | needs.scope.outputs.windows == 'true' | Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock; This repo is a valid adopter on Windows | | generated | always | Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the lint tooling; Registry matches the policies on disk; Installed policies trail, never lead, what this catalog publishes; Workflow triggers cannot hand fork code our secrets; Compiled artifacts match their manifests; Published policies are current Agent Plugins packages; Lint (ruff, the framework's rule set); Formatting (ruff format) | | adopter | always | Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock; This repo is a valid adopter; Declared effects match what the guards actually do; Reviewer evidence, if the PR carries any | | tests-shard | needs.scope.outputs.full == 'true' | Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the test tooling; Tests, one shard (every policy, every rule) | | durations | always() && github.event_name == 'push' && needs.scope.outputs.full == 'true' | Restore the file times main last recorded; Sum the junit times by test file and keep each file's longest test; Slow-test report; Save the file times for the next run; No single test takes longer than the cap | | tests-policy | needs.scope.outputs.full == 'false' && needs.scope.outputs.has_matrix == 'true' | Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the test tooling; Tests of this policy, its implementations at 100% line and branch coverage | | tests-targeted | needs.scope.outputs.full == 'false' | Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock and the test tooling; Changed tests no policy job owns, and repo standards | | tests | always() | Every test job this run needed passed; Install the coverage tooling; Combine the shards, 100% line and branch coverage | | staged-adopter | always | Read the framework pin; Check out the framework; The framework checkout is the pinned commit; Install Chock; Stage the published policies and skills as an adopter would; Validate the staged policies and replay their evals; OWASP ASI coverage claim holds in the staged adopter | | policies | always() | Every policies job passed | --- Built from engine [5a4b6287a46e](https://github.com/open-coder-ai/chock/commit/5a4b6287a46e07f9f856d9bbe10622bd1ff2e021) and catalog [1e60d211c1bb](https://github.com/open-coder-ai/chock-catalog/commit/1e60d211c1bb767152c6ab0167932da06ab3a10f).