52 lines
1.4 KiB
Markdown
52 lines
1.4 KiB
Markdown
# CTO Eval Suite
|
|
|
|
This directory holds the test-first promotion and regression suite for the CTO
|
|
WebUI coding agent PRD.
|
|
|
|
The suite is evidence-based: a run is not accepted from prose alone. Scoring
|
|
must inspect transcripts, diffs, logs, screenshots, approval events, capsule
|
|
artifacts, and report YAML.
|
|
|
|
Run the static PRD gate from the Hermes root:
|
|
|
|
```bash
|
|
pytest -q tests/e2e/test_j_cto_webui_prd.py
|
|
```
|
|
|
|
Score all current evidence reports from `cto/`:
|
|
|
|
```bash
|
|
for r in evals/reports/*.yaml; do python3 evals/runners/score.py "$r"; done
|
|
```
|
|
|
|
Run the deterministic local CTO/WebUI regression execution slice from `cto/`:
|
|
|
|
```bash
|
|
./evals/runners/run-webui-cto.sh
|
|
```
|
|
|
|
Run the executable promotion-suite readiness gate from `cto/`:
|
|
|
|
```bash
|
|
python3 evals/runners/run-promotion-suite.py
|
|
python3 evals/runners/score.py evals/reports/2026-05-25-promotion-suite-readiness.yaml
|
|
```
|
|
|
|
Run the isolated deterministic fixture execution gate from `cto/`:
|
|
|
|
```bash
|
|
python3 evals/runners/run-promotion-fixtures.py
|
|
python3 evals/runners/score.py evals/reports/2026-05-25-promotion-fixture-execution.yaml
|
|
```
|
|
|
|
Check Codex comparative readiness from `cto/`:
|
|
|
|
```bash
|
|
./evals/runners/run-codex-cli.sh
|
|
```
|
|
|
|
`fixtures/manifest.yaml` is the deterministic contract layer for the full PRD
|
|
promotion suite. It proves every required eval has a prompt, evidence
|
|
expectations, event expectations, and gates. It does not claim live promotion
|
|
success or Codex CLI parity.
|