hermes/cto
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Case model provider admission route

Browse Source
This commit is contained in:
Svrnty
2026-05-31 19:50:07 -04:00
parent eefa156734
commit bc2ea7ebcc
5 changed files with 229 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION-ISSUES.md
+56
View File
@@ -0,0 +1,56 @@
---
title: CTO Case Model Provider Admission Issues
status: draft
lifecycle_classification: sot
owner: jp
created: 2026-05-31
last_reviewed: 2026-05-31
core_promotion_status: not-promoted
route: cto
---
# CTO Case Model Provider Admission Issues
Local planning SOT only. Not a Core Protocol. Not active Core authority.
## CTO-WORK-019 - Case Model Provider Admission PRD
Status: validated.
Extract the existing `CTO-WORK-018` harness gate into a first-class model provider admission route. This is the SOT route for deciding which provider/model pair may power real Case Stage 2.
Acceptance:
- Records observed fallback provider `anthropic`.
- Records observed fallback model `claude-sonnet-4-6`.
- Requires explicit admitted provider and exact model ID before real Case starts.
- Requires `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` to match the admission record.
- Requires `backend/provider-model-not-admitted.txt` when admission is missing.
- Requires isolated `CASE_DATA_DIR/config.json` to contain admitted `models.default`.
- Requires negative gates for missing provider/model and unadmitted provider/model.
- Requires no secrets in task file, argv, report, trace, backend logs, SOT, or commits.
- Keeps Case as candidate execution backend, not CTO authority.
## CTO-WORK-020 - Admit Case Model Provider For Real Stage 2
Status: blocked.
Choose and admit the exact provider/model path for real Case Stage 2, then rerun Stage 2 through the Harness Evidence Interface.
Acceptance:
- Admission record names provider, exact model ID, credential source class, allowed network class, approval source, admission timestamp, review trigger, and evidence expectations.
- No provider/model is admitted by default.
- No secret is written to SOT, argv, task file, backend logs, report, trace, or commit.
- `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` match the admission record.
- Missing or unadmitted provider/model blocks before `case_process_started`.
- Report records `case_model_provider`, `case_model`, and `case_model_admission_status`.
- Real Case Stage 2 produces a pass report only if the admitted provider/model was used.
- Same-run fake baseline comparison remains required.
- No Target Repository path is inspected or copied.
Blocked by:
- Human provider approval if an external provider such as Anthropic is selected.
- A Case-compatible local provider route if external providers are not approved.
sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION-PRD.md
+90
View File
@@ -0,0 +1,90 @@
---
title: CTO Case Model Provider Admission PRD
status: draft
lifecycle_classification: sot
owner: jp
created: 2026-05-31
last_reviewed: 2026-05-31
core_promotion_status: not-promoted
route: cto
---
# CTO Case Model Provider Admission PRD
Local planning SOT only. Not a Core Protocol. Not active Core authority.
## Problem Statement
`CTO-WORK-018` validated a harness gate that blocks missing model configuration, but the CTO route still needs a first-class admission record for the model provider itself. Evidence showed WorkOS Case silently defaulted to provider `anthropic` and model `claude-sonnet-4-6` when the harness did not write a model registry. That path is an unadmitted external model path for CTO proof.
## Solution
Extract the model provider decision into a child-local admission route. The route requires an explicit admitted provider/model pair, redacted credential policy, isolated Case config proof, negative gates, and real Stage 2 retry conditions before any real Case run can claim progress.
## Scope
- Admit only one named Case model provider and exact model ID at a time.
- Require admission before `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` may be used for real Case.
- Preserve fail-closed behavior through `backend/provider-model-not-admitted.txt`.
- Require unadmitted provider/model blocks before `case_process_started`.
- Require the adapter to write admitted `models.default` into isolated `CASE_DATA_DIR/config.json`.
- Require provider evidence in `report.json`, backend logs, `trace.jsonl`, and artifact digests.
- Require secret-redaction evidence for task file, argv, report, trace, and backend logs.
- Keep Stage 2 mutation scope limited to copied artificial fixture only.
- Keep executable admission separate from model provider admission.
- Keep `ca run --task <task-file> --mode unattended` as the only real Case Stage 2 command shape.
- Preserve same-run fake baseline comparison.
## Non-Goals
- Do not approve Anthropic, Claude, local inference, or any other provider by default.
- Do not create a broad provider marketplace or registry abstraction.
- Do not store credentials in SOT, task files, argv, commits, reports, traces, or backend logs.
- Do not grant Case CTO authority.
- Do not authorize copied repo, sandbox repo, owned repo, default backend, WebUI product, or Core promotion behavior.
- Do not bypass the Harness Evidence Interface.
- Do not mutate Case source, Cortex Core, vendor source, or target repositories.
## Acceptance Criteria
- A model provider admission record names provider, exact model ID, credential source class, allowed network class, approval source, admission timestamp, and expiry or review trigger.
- Missing provider/model admission blocks before `case_process_started`.
- Unadmitted provider/model blocks before `case_process_started`.
- Missing credentials, unexpected fallback model, missing config write, or absent provider evidence blocks.
- Stage 2 report records `case_model_provider`, `case_model`, `case_model_admission_status`, `case_process_started`, `backend_exit_code`, `allowed_writes_passed`, `changed_files`, and `blockers`.
- Real Case Stage 2 cannot pass unless the report proves the admitted provider/model was used.
- Real Case Stage 2 remains blocked unless a pass report exists.
- Fake remains the default validation lane.
- Same-run fake baseline comparison remains required.
- No secrets appear in task file, argv, report, trace, backend logs, SOT, or commits.
## Validation
- `python3 tools/validate_cto_child.py` validates this child-local route.
- Hermes focused validation must include `python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json`.
- Required negative gates: missing provider/model blocks before `case_process_started`; unadmitted provider/model blocks before `case_process_started`; no secrets appear in task file, argv, report, trace, backend logs.
- Real provider validation must include `CTO_HARNESS_ALLOW_CASE=1 CTO_HARNESS_CASE_STAGE=2 CTO_HARNESS_CASE_BIN=<admitted-ca> CTO_HARNESS_CASE_MODEL_PROVIDER=<admitted-provider> CTO_HARNESS_CASE_MODEL=<admitted-model> harness/evals/run-case.sh r1-string-slugify --engine case --json`.
- Aggregate validation remains `harness/evals/health.sh --json` after focused gates pass.
## Risks And Dependencies
- Human approval may be required before any external provider is admitted.
- Local provider use may require a separate Case-compatible provider adapter or credentials path.
- Case defaults may change; model evidence must be read from actual run artifacts, not assumed from docs.
- Provider credentials may be unavailable in the current terminal.
- License status remains unresolved for broader execution modes.
## Success Definition
Real Case Stage 2 remains blocked until a named provider/model is admitted, then passes only when the Harness Evidence Interface proves the admitted provider/model executed the copied artificial fixture without forbidden writes, target inspection, fallback model use, or secret leakage.
## Current Evidence - 2026-05-31
- Existing gate: `CTO-WORK-018 - Case Model Provider Admission Gate`.
- Real Case defaulted to provider `anthropic` and model `claude-sonnet-4-6` without an explicit model registry.
- Runtime report path: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T234205Z-r1-string-slugify-1834617/report.json`.
- Hermes model gate commit: `4500082 Gate Case execution on admitted model`.
- Model gate variables: `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL`.
- Model gate marker: `backend/provider-model-not-admitted.txt`.
- Validator check: `model_provider_gate_blocks`.