128 lines
6.7 KiB
Markdown
128 lines
6.7 KiB
Markdown
---
|
|
title: CTO Case Model Provider Admission Issues
|
|
status: draft
|
|
lifecycle_classification: sot
|
|
owner: jp
|
|
created: 2026-05-31
|
|
last_reviewed: 2026-05-31
|
|
core_promotion_status: not-promoted
|
|
route: cto
|
|
---
|
|
|
|
# CTO Case Model Provider Admission Issues
|
|
|
|
Local planning SOT only. Not a Core Protocol. Not active Core authority.
|
|
|
|
## CTO-WORK-019 - Case Model Provider Admission PRD
|
|
|
|
Status: validated.
|
|
|
|
Extract the existing `CTO-WORK-018` harness gate into a first-class model provider admission route. This is the SOT route for deciding which provider/model pair may power real Case Stage 2.
|
|
|
|
Acceptance:
|
|
|
|
- Records observed fallback provider `anthropic`.
|
|
- Records observed fallback model `claude-sonnet-4-6`.
|
|
- Requires explicit admitted provider and exact model ID before real Case starts.
|
|
- Requires `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` to match the admission record.
|
|
- Requires `backend/provider-model-not-admitted.txt` when admission is missing.
|
|
- Requires isolated `CASE_DATA_DIR/config.json` to contain admitted `models.default`.
|
|
- Requires negative gates for missing provider/model and unadmitted provider/model.
|
|
- Requires no secrets in task file, argv, report, trace, backend logs, SOT, or commits.
|
|
- Keeps Case as candidate execution backend, not CTO authority.
|
|
|
|
## CTO-WORK-020 - Admit Case Model Provider For Real Stage 2
|
|
|
|
Status: blocked.
|
|
|
|
Choose and admit the exact provider/model path for real Case Stage 2, then rerun Stage 2 through the Harness Evidence Interface.
|
|
|
|
Acceptance:
|
|
|
|
- Admission record names provider, exact model ID, credential source class, allowed network class, approval source, admission timestamp, review trigger, and evidence expectations.
|
|
- No provider/model is admitted by default.
|
|
- No secret is written to SOT, argv, task file, backend logs, report, trace, or commit.
|
|
- `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` match the admission record.
|
|
- Missing or unadmitted provider/model blocks before `case_process_started`.
|
|
- Report records `case_model_provider`, `case_model`, and `case_model_admission_status`.
|
|
- Real Case Stage 2 produces a pass report only if the admitted provider/model was used.
|
|
- Same-run fake baseline comparison remains required.
|
|
- No Target Repository path is inspected or copied.
|
|
|
|
Blocked by:
|
|
|
|
- Human provider approval if an external provider such as Anthropic is selected.
|
|
- A Case-compatible local provider route if external providers are not approved.
|
|
|
|
## Hermes Implementation Evidence - 2026-05-31
|
|
|
|
- Hermes commit: `f39d8ab Require admitted Case model pair`.
|
|
- `f39d8ab` proves admission gating implementation only; it is not a real Case Stage 2 pass.
|
|
- The Hermes adapter now requires `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE`.
|
|
- Env provider/model is only the requested pair; the admission JSON is the authority.
|
|
- Missing admission blocks before `case_process_started`.
|
|
- Mismatched admission blocks before `case_process_started`.
|
|
- Report evidence records `case_model_provider`, `case_model`, and `case_model_admission_status`.
|
|
- Status vocabulary includes `admitted`, `missing_admission`, `mismatch`, `invalid_admission`, and `not_admitted`.
|
|
- Secret scan covers `report.json`, `report.md`, `trace.jsonl`, backend logs, Case stdout/stderr, and generated Case config.
|
|
- Focused validator passed: `python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json`.
|
|
- Aggregate validator passed: `harness/evals/health.sh --json`.
|
|
- Focused validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235421Z-r1-string-slugify-1875638`.
|
|
- Aggregate validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235448Z-r1-string-slugify-1876884`.
|
|
- `CTO-WORK-020` remains blocked until a provider/model is explicitly approved and real Case Stage 2 produces a pass report.
|
|
|
|
## CTO-WORK-020 Decision Record Template
|
|
|
|
This template belongs to `CTO-WORK-020`; it is not a new provider approval.
|
|
|
|
Required fields:
|
|
|
|
- `decision_status`: `not_decided`, `external_provider_approved`, or `local_provider_required`.
|
|
- `provider_class`: `external_anthropic`, `external_openai_codex`, or `local_case_compatible`.
|
|
- `provider`: exact provider string, or empty while blocked.
|
|
- `model`: exact model string, or empty while blocked.
|
|
- `approval_source`: JP approval reference or governed Core route reference.
|
|
- `credential_source_class`: credential class only; no secret value.
|
|
- `allowed_network_class`: allowed network class for this provider.
|
|
- `review_trigger`: expiry, date, or condition that forces review.
|
|
- `evidence_sources`: references to existing admission/build evidence, not copied runtime evidence.
|
|
- `effect`: `CTO-WORK-020 remains blocked until admitted provider/model and real Stage 2 pass report exist`.
|
|
|
|
## CTO-WORK-027 - OpenAI Codex Model Admission JSON
|
|
|
|
Status: validated.
|
|
|
|
Record the exact non-secret admission JSON required by `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE` for the approved `openai-codex` / `gpt-5.5` primary provider path.
|
|
|
|
Acceptance:
|
|
|
|
- Admission file path is `.sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION.openai-codex-gpt-5.5.json`.
|
|
- Admission JSON has `status`: `admitted`.
|
|
- Admission JSON has `provider`: `openai-codex`.
|
|
- Admission JSON has `model`: `gpt-5.5`.
|
|
- Admission JSON has `credential_source_class`: `hermes-openai-codex-oauth-and-local-vllm-config`.
|
|
- Admission JSON has `allowed_network_class`: `codex-oauth-hosted-model-plus-local-vllm-fallback`.
|
|
- Admission JSON has `approval_source`: `JP chat approval on 2026-05-31`.
|
|
- Admission JSON has `admission_timestamp`.
|
|
- Admission JSON has `review_trigger`.
|
|
- Admission JSON contains no secret keys or secret values.
|
|
- Fallback to `vllm` / `qwen3.6-35b-a3b` remains explicit decision-record context and must be represented in runtime evidence before it may count as a Case provider/model path.
|
|
- `CTO-WORK-020` remains blocked until real Case Stage 2 produces a Harness Evidence Interface pass report using this admission file.
|
|
- Real Case Stage 2 command must set `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE` to this admission JSON path.
|
|
|
|
## Post-Admission Runtime Evidence - 2026-06-01
|
|
|
|
- Run artifact directory: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T013918Z-r1-string-slugify-2381028`.
|
|
- Report status: `fail`.
|
|
- Backend: `case`.
|
|
- Case process started: `true`.
|
|
- Case model provider: `openai-codex`.
|
|
- Case model: `gpt-5.5`.
|
|
- Case model admission status: `admitted`.
|
|
- The admitted provider/model path reached Case execution.
|
|
- Case failed before producing a diff.
|
|
- Case stderr recorded `AGENT_RESULT start delimiter not found`.
|
|
- Case retry classified the failure as `agent-protocol-error`.
|
|
- `CTO-WORK-020` remains blocked because no real Case Stage 2 pass report exists.
|
|
- Current downstream blocker is tracked by `CTO-WORK-028`.
|