181 lines
9.8 KiB
Markdown
181 lines
9.8 KiB
Markdown
---
|
|
title: CTO Case Model Provider Admission Issues
|
|
status: draft
|
|
lifecycle_classification: sot
|
|
owner: jp
|
|
created: 2026-05-31
|
|
last_reviewed: 2026-06-01
|
|
core_promotion_status: not-promoted
|
|
route: cto
|
|
---
|
|
|
|
# CTO Case Model Provider Admission Issues
|
|
|
|
Local planning SOT only. Not a Core Protocol. Not active Core authority.
|
|
|
|
## CTO-WORK-019 - Case Model Provider Admission PRD
|
|
|
|
Status: validated.
|
|
|
|
Extract the existing `CTO-WORK-018` harness gate into a first-class model provider admission route. This is the SOT route for deciding which provider/model pair may power real Case Stage 2.
|
|
|
|
Acceptance:
|
|
|
|
- Records observed fallback provider `anthropic`.
|
|
- Records observed fallback model `claude-sonnet-4-6`.
|
|
- Requires explicit admitted provider and exact model ID before real Case starts.
|
|
- Requires `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` to match the admission record.
|
|
- Requires `backend/provider-model-not-admitted.txt` when admission is missing.
|
|
- Requires isolated `CASE_DATA_DIR/config.json` to contain admitted `models.default`.
|
|
- Requires negative gates for missing provider/model and unadmitted provider/model.
|
|
- Requires no secrets in task file, argv, report, trace, backend logs, SOT, or commits.
|
|
- Keeps Case as candidate execution backend, not CTO authority.
|
|
|
|
## CTO-WORK-020 - Admit Case Model Provider For Real Stage 2
|
|
|
|
Status: blocked.
|
|
|
|
Choose and admit the exact provider/model path for real Case Stage 2, then rerun Stage 2 through the Harness Evidence Interface.
|
|
|
|
Acceptance:
|
|
|
|
- Admission record names provider, exact model ID, credential source class, allowed network class, approval source, admission timestamp, review trigger, and evidence expectations.
|
|
- No provider/model is admitted by default.
|
|
- No secret is written to SOT, argv, task file, backend logs, report, trace, or commit.
|
|
- `CTO_HARNESS_CASE_MODEL_PROVIDER` and `CTO_HARNESS_CASE_MODEL` match the admission record.
|
|
- Missing or unadmitted provider/model blocks before `case_process_started`.
|
|
- Report records `case_model_provider`, `case_model`, and `case_model_admission_status`.
|
|
- Real Case Stage 2 produces a pass report only if the admitted provider/model was used.
|
|
- Same-run fake baseline comparison remains required.
|
|
- No Target Repository path is inspected or copied.
|
|
|
|
Blocked by:
|
|
|
|
- Human provider approval if an external provider such as Anthropic is selected.
|
|
- A Case-compatible local provider route if external providers are not approved.
|
|
|
|
## Hermes Implementation Evidence - 2026-05-31
|
|
|
|
- Hermes commit: `f39d8ab Require admitted Case model pair`.
|
|
- `f39d8ab` proves admission gating implementation only; it is not a real Case Stage 2 pass.
|
|
- The Hermes adapter now requires `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE`.
|
|
- Env provider/model is only the requested pair; the admission JSON is the authority.
|
|
- Missing admission blocks before `case_process_started`.
|
|
- Mismatched admission blocks before `case_process_started`.
|
|
- Report evidence records `case_model_provider`, `case_model`, and `case_model_admission_status`.
|
|
- Status vocabulary includes `admitted`, `missing_admission`, `mismatch`, `invalid_admission`, and `not_admitted`.
|
|
- Secret scan covers `report.json`, `report.md`, `trace.jsonl`, backend logs, Case stdout/stderr, and generated Case config.
|
|
- Focused validator passed: `python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json`.
|
|
- Aggregate validator passed: `harness/evals/health.sh --json`.
|
|
- Focused validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235421Z-r1-string-slugify-1875638`.
|
|
- Aggregate validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235448Z-r1-string-slugify-1876884`.
|
|
- `CTO-WORK-020` remains blocked until a provider/model is explicitly approved and real Case Stage 2 produces a pass report.
|
|
|
|
## CTO-WORK-020 Decision Record Template
|
|
|
|
This template belongs to `CTO-WORK-020`; it is not a new provider approval.
|
|
|
|
Required fields:
|
|
|
|
- `decision_status`: `not_decided`, `external_provider_approved`, or `local_provider_required`.
|
|
- `provider_class`: `external_anthropic`, `external_openai_codex`, or `local_case_compatible`.
|
|
- `provider`: exact provider string, or empty while blocked.
|
|
- `model`: exact model string, or empty while blocked.
|
|
- `approval_source`: JP approval reference or governed Core route reference.
|
|
- `credential_source_class`: credential class only; no secret value.
|
|
- `allowed_network_class`: allowed network class for this provider.
|
|
- `review_trigger`: expiry, date, or condition that forces review.
|
|
- `evidence_sources`: references to existing admission/build evidence, not copied runtime evidence.
|
|
- `effect`: `CTO-WORK-020 remains blocked until admitted provider/model and real Stage 2 pass report exist`.
|
|
|
|
## CTO-WORK-027 - OpenAI Codex Model Admission JSON
|
|
|
|
Status: validated.
|
|
|
|
Record the exact non-secret admission JSON required by `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE` for the approved `openai-codex` / `gpt-5.5` primary provider path.
|
|
|
|
Acceptance:
|
|
|
|
- Admission file path is `.sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION.openai-codex-gpt-5.5.json`.
|
|
- Admission JSON has `status`: `admitted`.
|
|
- Admission JSON has `provider`: `openai-codex`.
|
|
- Admission JSON has `model`: `gpt-5.5`.
|
|
- Admission JSON has `credential_source_class`: `hermes-openai-codex-oauth-and-local-vllm-config`.
|
|
- Admission JSON has `allowed_network_class`: `codex-oauth-hosted-model-plus-local-vllm-fallback`.
|
|
- Admission JSON has `approval_source`: `JP chat approval on 2026-05-31`.
|
|
- Admission JSON has `admission_timestamp`.
|
|
- Admission JSON has `review_trigger`.
|
|
- Admission JSON contains no secret keys or secret values.
|
|
- Fallback to `vllm` / `qwen3.6-35b-a3b` remains explicit decision-record context and must be represented in runtime evidence before it may count as a Case provider/model path.
|
|
- `CTO-WORK-020` remains blocked until real Case Stage 2 produces a Harness Evidence Interface pass report using this admission file.
|
|
- Real Case Stage 2 command must set `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE` to this admission JSON path.
|
|
|
|
## Post-Admission Runtime Evidence - 2026-06-01
|
|
|
|
- Run artifact directory: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T013918Z-r1-string-slugify-2381028`.
|
|
- Report status: `fail`.
|
|
- Backend: `case`.
|
|
- Case process started: `true`.
|
|
- Case model provider: `openai-codex`.
|
|
- Case model: `gpt-5.5`.
|
|
- Case model admission status: `admitted`.
|
|
- The admitted provider/model path reached Case execution.
|
|
- Case failed before producing a diff.
|
|
- Case stderr recorded `AGENT_RESULT start delimiter not found`.
|
|
- Case retry classified the failure as `agent-protocol-error`.
|
|
- `CTO-WORK-020` remains blocked because no real Case Stage 2 pass report exists.
|
|
- Current downstream blocker is tracked by `CTO-WORK-028`.
|
|
|
|
## Hermes Auth Preflight Evidence - 2026-06-01
|
|
|
|
- Hermes commit: `5db23c7 Fail closed on Case Codex auth gap`.
|
|
- `5db23c7` proves fail-closed classification only; it is not a real Case Stage 2 pass.
|
|
- The Hermes adapter now blocks `openai-codex` before `case_process_started` unless `CTO_HARNESS_CASE_OPENAI_CODEX_AUTH_BRIDGE=1`.
|
|
- The block writes `backend/provider-auth-unavailable.txt`.
|
|
- The reason is specific: Case's pipeline SDK path constructs its Pi Agent runtime directly and does not pass Pi AuthStorage OAuth headers into `streamSimple`.
|
|
- Pi env API-key lookup does not map `openai-codex` to an environment API key.
|
|
- Focused validator passed: `python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json`.
|
|
- Aggregate validator passed: `harness/evals/health.sh --json`.
|
|
- Focused validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T014803Z-r1-string-slugify-2448172`.
|
|
- Aggregate validator artifact: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T014805Z-r1-string-slugify-2449212`.
|
|
- `CTO-WORK-020` remains blocked because no real Case Stage 2 pass report exists.
|
|
|
|
## CTO-WORK-029 - Qwen Local Case Model Admission JSON
|
|
|
|
Status: validated.
|
|
|
|
Record the exact non-secret admission JSON required by `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE` for the existing Pi local provider path `qwen-local` / `qwen3.6-35b-a3b`.
|
|
|
|
Acceptance:
|
|
|
|
- Admission file path is `.sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION.qwen-local-qwen3.6-35b-a3b.json`.
|
|
- Admission JSON has `status`: `admitted`.
|
|
- Admission JSON has `provider`: `qwen-local`.
|
|
- Admission JSON has `model`: `qwen3.6-35b-a3b`.
|
|
- Admission JSON has `credential_source_class`: `pi-models-json-local-provider-no-secret`.
|
|
- Admission JSON has `allowed_network_class`: `local-tailscale-vllm-spark1`.
|
|
- Admission JSON has `approval_source`: `JP chat approval on 2026-05-31`.
|
|
- Admission JSON has `admission_timestamp`.
|
|
- Admission JSON has `review_trigger`.
|
|
- Admission JSON contains no secret keys or secret values.
|
|
- This admission does not change Hermes primary model policy.
|
|
- This admission does not approve real-repo, copied-repo, sandbox-repo, owned-repo, default-candidate, or Core promotion stages.
|
|
- `CTO-WORK-020` remains blocked until real Case Stage 2 produces a Harness Evidence Interface pass report using this admission file.
|
|
- Real Case Stage 2 command must set `CTO_HARNESS_CASE_MODEL_ADMISSION_FILE` to this admission JSON path.
|
|
|
|
## Qwen Local Runtime Evidence - 2026-06-01
|
|
|
|
- Run artifact directory: `/home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T015208Z-r1-string-slugify-2478256`.
|
|
- Report status: `fail`.
|
|
- Backend: `case`.
|
|
- Case process started: `true`.
|
|
- Case model provider: `qwen-local`.
|
|
- Case model: `qwen3.6-35b-a3b`.
|
|
- Case model admission status: `admitted`.
|
|
- The admitted Qwen local provider/model path reached Case execution.
|
|
- Case failed before producing a diff.
|
|
- Case stderr recorded `AGENT_RESULT start delimiter not found`.
|
|
- Case retry classified the failure as `agent-protocol-error`.
|
|
- `CTO-WORK-020` remains blocked because no real Case Stage 2 pass report exists.
|
|
- Current downstream blocker remains `CTO-WORK-028`.
|