129 lines
8.3 KiB
Markdown
129 lines
8.3 KiB
Markdown
---
|
|
title: CTO Case Model Provider Admission PRD
|
|
status: draft
|
|
lifecycle_classification: sot
|
|
owner: jp
|
|
created: 2026-05-31
|
|
last_reviewed: 2026-05-31
|
|
core_promotion_status: not-promoted
|
|
route: cto
|
|
---
|
|
|
|
# CTO Case Model Provider Admission PRD
|
|
|
|
Local planning SOT only. Not a Core Protocol. Not active Core authority.
|
|
|
|
## Problem Statement
|
|
|
|
`CTO-WORK-018` validated a harness gate that blocks missing model configuration, but the CTO route still needs a first-class admission record for the model provider itself. Evidence showed WorkOS Case silently defaulted to provider `anthropic` and model `claude-sonnet-4-6` when the harness did not write a model registry. That path is an unadmitted external model path for CTO proof.
|
|
|
|
## Solution
|
|
|
|
Extract the model provider decision into a child-local admission route. The route requires an explicit admitted provider/model pair, redacted credential policy, isolated Case config proof, negative gates, and real Stage 2 retry conditions before any real Case run can claim progress.
|
|
|
|
## Scope
|
|
|
|
- Admit only one named Case model provider and exact model ID at a time.
|
|
- Require admission before `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` may be used for real Case.
|
|
- Preserve fail-closed behavior through `backend/provider-model-not-admitted.txt`.
|
|
- Require unadmitted provider/model blocks before `case_process_started`.
|
|
- Require the adapter to write admitted `models.default` into isolated `CASE_DATA_DIR/config.json`.
|
|
- Require provider evidence in `report.json`, backend logs, `trace.jsonl`, and artifact digests.
|
|
- Require secret-redaction evidence for task file, argv, report, trace, and backend logs.
|
|
- Keep Stage 2 mutation scope limited to copied artificial fixture only.
|
|
- Keep executable admission separate from model provider admission.
|
|
- Keep `ca run --task <task-file> --mode unattended` as the only real Case Stage 2 command shape.
|
|
- Preserve same-run fake baseline comparison.
|
|
|
|
## Non-Goals
|
|
|
|
- Do not approve Anthropic, Claude, local inference, or any other provider by default.
|
|
- Do not create a broad provider marketplace or registry abstraction.
|
|
- Do not store credentials in SOT, task files, argv, commits, reports, traces, or backend logs.
|
|
- Do not grant Case CTO authority.
|
|
- Do not authorize copied repo, sandbox repo, owned repo, default backend, WebUI product, or Core promotion behavior.
|
|
- Do not bypass the Harness Evidence Interface.
|
|
- Do not mutate Case source, Cortex Core, vendor source, or target repositories.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- A model provider admission record names provider, exact model ID, credential source class, allowed network class, approval source, admission timestamp, and expiry or review trigger.
|
|
- Missing provider/model admission blocks before `case_process_started`.
|
|
- Unadmitted provider/model blocks before `case_process_started`.
|
|
- Missing credentials, unexpected fallback model, missing config write, or absent provider evidence blocks.
|
|
- Stage 2 report records `case_model_provider`, `case_model`, `case_model_admission_status`, `case_process_started`, `backend_exit_code`, `allowed_writes_passed`, `changed_files`, and `blockers`.
|
|
- Real Case Stage 2 cannot pass unless the report proves the admitted provider/model was used.
|
|
- Real Case Stage 2 remains blocked unless a pass report exists.
|
|
- Fake remains the default validation lane.
|
|
- Same-run fake baseline comparison remains required.
|
|
- No secrets appear in task file, argv, report, trace, backend logs, SOT, or commits.
|
|
|
|
## Validation
|
|
|
|
- `python3 tools/validate_cto_child.py` validates this child-local route.
|
|
- Hermes focused validation must include `python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json`.
|
|
- Required negative gates: missing provider/model blocks before `case_process_started`; unadmitted provider/model blocks before `case_process_started`; no secrets appear in task file, argv, report, trace, backend logs.
|
|
- Real provider validation must include `CTO_HARNESS_ALLOW_CASE=1 CTO_HARNESS_CASE_STAGE=2 CTO_HARNESS_CASE_BIN=<admitted-ca> CTO_HARNESS_CASE_MODEL_PROVIDER=<admitted-provider> CTO_HARNESS_CASE_MODEL=<admitted-model> harness/evals/run-case.sh r1-string-slugify --engine case --json`.
|
|
- Aggregate validation remains `harness/evals/health.sh --json` after focused gates pass.
|
|
|
|
## Risks And Dependencies
|
|
|
|
- Human approval may be required before any external provider is admitted.
|
|
- Local provider use may require a separate Case-compatible provider adapter or credentials path.
|
|
- Case defaults may change; model evidence must be read from actual run artifacts, not assumed from docs.
|
|
- Provider credentials may be unavailable in the current terminal.
|
|
- License status remains unresolved for broader execution modes.
|
|
|
|
## Success Definition
|
|
|
|
Real Case Stage 2 remains blocked until a named provider/model is admitted, then passes only when the Harness Evidence Interface proves the admitted provider/model executed the copied artificial fixture without forbidden writes, target inspection, fallback model use, or secret leakage.
|
|
|
|
## Current Evidence - 2026-05-31
|
|
|
|
- Existing gate: `CTO-WORK-018 - Case Model Provider Admission Gate`.
|
|
- Real Case defaulted to provider `anthropic` and model `claude-sonnet-4-6` without an explicit model registry.
|
|
- Runtime report path: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T234205Z-r1-string-slugify-1834617/report.json`.
|
|
- Hermes model gate commit: `4500082 Gate Case execution on admitted model`.
|
|
- Model gate variables: `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL`.
|
|
- Model gate marker: `backend/provider-model-not-admitted.txt`.
|
|
- Validator check: `model_provider_gate_blocks`.
|
|
|
|
## Hermes Implementation Evidence - 2026-05-31
|
|
|
|
- Hermes commit: `f39d8ab Require admitted Case model pair`.
|
|
- `f39d8ab` proves admission gating implementation only; it is not a real Case Stage 2 pass.
|
|
- Admission file variable: `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE`.
|
|
- Env provider/model is now a requested pair, not admission authority.
|
|
- The admission JSON is the authority for real Case Stage 2 model admission.
|
|
- The requested `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` must match the admitted JSON provider and model.
|
|
- Admission status values: `admitted`, `missing_admission`, `mismatch`, `invalid_admission`, `not_admitted`.
|
|
- Missing admission and mismatched admission block before `case_process_started`.
|
|
- Stage 2 reports include `case_model_provider`, `case_model`, and `case_model_admission_status` for pass and blocked paths.
|
|
- Secret scan covers `report.json`, `report.md`, `trace.jsonl`, backend logs, Case stdout/stderr, and generated Case config.
|
|
- Focused Hermes validator passed: `python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json`.
|
|
- Post-merge Hermes aggregate validator passed: `harness/evals/health.sh --json`.
|
|
- Focused validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235421Z-r1-string-slugify-1875638`.
|
|
- Aggregate validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235448Z-r1-string-slugify-1876884`.
|
|
- `CTO-WORK-020` remains blocked because no real provider/model has been approved and no real Case Stage 2 pass report exists.
|
|
|
|
## Decision Record Template For CTO-WORK-020
|
|
|
|
This template clarifies the decision required by `CTO-WORK-020`; it does not approve a provider.
|
|
|
|
- `decision_status`: `not_decided`, `external_provider_approved`, or `local_provider_required`.
|
|
- `provider_class`: `external_anthropic` or `local_case_compatible`.
|
|
- `provider`: exact provider string, or empty while blocked.
|
|
- `model`: exact model string, or empty while blocked.
|
|
- `approval_source`: JP approval reference or governed Core route reference.
|
|
- `credential_source_class`: credential class only; no secret value.
|
|
- `allowed_network_class`: allowed network class for this provider.
|
|
- `review_trigger`: expiry, date, or condition that forces review.
|
|
- `evidence_sources`: references to existing admission/build evidence, not copied runtime evidence.
|
|
- `effect`: `CTO-WORK-020 remains blocked until admitted provider/model and real Stage 2 pass report exist`.
|
|
|
|
Allowed pending states:
|
|
|
|
- `not_decided`: no provider/model may run.
|
|
- `local_provider_required`: no external provider may run; create a Case-compatible local provider route first.
|
|
- `external_provider_approved`: may proceed only when the approval source, credential source class, allowed network class, and admission JSON are recorded.
|