hermes/cto
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6af4199ebc
cto/sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION-PRD.md
Svrnty 3ff079e9f4 Add Case provider decision template
2026-05-31 20:01:04 -04:00

8.3 KiB
Raw Blame History

title status lifecycle_classification owner created last_reviewed core_promotion_status route
CTO Case Model Provider Admission PRD draft sot jp 2026-05-31 2026-05-31 not-promoted cto

CTO Case Model Provider Admission PRD

Local planning SOT only. Not a Core Protocol. Not active Core authority.

Problem Statement

CTO-WORK-018 validated a harness gate that blocks missing model configuration, but the CTO route still needs a first-class admission record for the model provider itself. Evidence showed WorkOS Case silently defaulted to provider anthropic and model claude-sonnet-4-6 when the harness did not write a model registry. That path is an unadmitted external model path for CTO proof.

Solution

Extract the model provider decision into a child-local admission route. The route requires an explicit admitted provider/model pair, redacted credential policy, isolated Case config proof, negative gates, and real Stage 2 retry conditions before any real Case run can claim progress.

Scope

  • Admit only one named Case model provider and exact model ID at a time.
  • Require admission before CTO_HARNESS_CASE_MODEL_PROVIDER and CTO_HARNESS_CASE_MODEL may be used for real Case.
  • Preserve fail-closed behavior through backend/provider-model-not-admitted.txt.
  • Require unadmitted provider/model blocks before case_process_started.
  • Require the adapter to write admitted models.default into isolated CASE_DATA_DIR/config.json.
  • Require provider evidence in report.json, backend logs, trace.jsonl, and artifact digests.
  • Require secret-redaction evidence for task file, argv, report, trace, and backend logs.
  • Keep Stage 2 mutation scope limited to copied artificial fixture only.
  • Keep executable admission separate from model provider admission.
  • Keep ca run --task <task-file> --mode unattended as the only real Case Stage 2 command shape.
  • Preserve same-run fake baseline comparison.

Non-Goals

  • Do not approve Anthropic, Claude, local inference, or any other provider by default.
  • Do not create a broad provider marketplace or registry abstraction.
  • Do not store credentials in SOT, task files, argv, commits, reports, traces, or backend logs.
  • Do not grant Case CTO authority.
  • Do not authorize copied repo, sandbox repo, owned repo, default backend, WebUI product, or Core promotion behavior.
  • Do not bypass the Harness Evidence Interface.
  • Do not mutate Case source, Cortex Core, vendor source, or target repositories.

Acceptance Criteria

  • A model provider admission record names provider, exact model ID, credential source class, allowed network class, approval source, admission timestamp, and expiry or review trigger.
  • Missing provider/model admission blocks before case_process_started.
  • Unadmitted provider/model blocks before case_process_started.
  • Missing credentials, unexpected fallback model, missing config write, or absent provider evidence blocks.
  • Stage 2 report records case_model_provider, case_model, case_model_admission_status, case_process_started, backend_exit_code, allowed_writes_passed, changed_files, and blockers.
  • Real Case Stage 2 cannot pass unless the report proves the admitted provider/model was used.
  • Real Case Stage 2 remains blocked unless a pass report exists.
  • Fake remains the default validation lane.
  • Same-run fake baseline comparison remains required.
  • No secrets appear in task file, argv, report, trace, backend logs, SOT, or commits.

Validation

  • python3 tools/validate_cto_child.py validates this child-local route.
  • Hermes focused validation must include python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json.
  • Required negative gates: missing provider/model blocks before case_process_started; unadmitted provider/model blocks before case_process_started; no secrets appear in task file, argv, report, trace, backend logs.
  • Real provider validation must include CTO_HARNESS_ALLOW_CASE=1 CTO_HARNESS_CASE_STAGE=2 CTO_HARNESS_CASE_BIN=<admitted-ca> CTO_HARNESS_CASE_MODEL_PROVIDER=<admitted-provider> CTO_HARNESS_CASE_MODEL=<admitted-model> harness/evals/run-case.sh r1-string-slugify --engine case --json.
  • Aggregate validation remains harness/evals/health.sh --json after focused gates pass.

Risks And Dependencies

  • Human approval may be required before any external provider is admitted.
  • Local provider use may require a separate Case-compatible provider adapter or credentials path.
  • Case defaults may change; model evidence must be read from actual run artifacts, not assumed from docs.
  • Provider credentials may be unavailable in the current terminal.
  • License status remains unresolved for broader execution modes.

Success Definition

Real Case Stage 2 remains blocked until a named provider/model is admitted, then passes only when the Harness Evidence Interface proves the admitted provider/model executed the copied artificial fixture without forbidden writes, target inspection, fallback model use, or secret leakage.

Current Evidence - 2026-05-31

  • Existing gate: CTO-WORK-018 - Case Model Provider Admission Gate.
  • Real Case defaulted to provider anthropic and model claude-sonnet-4-6 without an explicit model registry.
  • Runtime report path: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T234205Z-r1-string-slugify-1834617/report.json.
  • Hermes model gate commit: 4500082 Gate Case execution on admitted model.
  • Model gate variables: CTO_HARNESS_CASE_MODEL_PROVIDER and CTO_HARNESS_CASE_MODEL.
  • Model gate marker: backend/provider-model-not-admitted.txt.
  • Validator check: model_provider_gate_blocks.

Hermes Implementation Evidence - 2026-05-31

  • Hermes commit: f39d8ab Require admitted Case model pair.
  • f39d8ab proves admission gating implementation only; it is not a real Case Stage 2 pass.
  • Admission file variable: CTO_HARNESS_CASE_MODEL_ADMISSION_FILE.
  • Env provider/model is now a requested pair, not admission authority.
  • The admission JSON is the authority for real Case Stage 2 model admission.
  • The requested CTO_HARNESS_CASE_MODEL_PROVIDER and CTO_HARNESS_CASE_MODEL must match the admitted JSON provider and model.
  • Admission status values: admitted, missing_admission, mismatch, invalid_admission, not_admitted.
  • Missing admission and mismatched admission block before case_process_started.
  • Stage 2 reports include case_model_provider, case_model, and case_model_admission_status for pass and blocked paths.
  • Secret scan covers report.json, report.md, trace.jsonl, backend logs, Case stdout/stderr, and generated Case config.
  • Focused Hermes validator passed: python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json.
  • Post-merge Hermes aggregate validator passed: harness/evals/health.sh --json.
  • Focused validator artifact: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235421Z-r1-string-slugify-1875638.
  • Aggregate validator artifact: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260531T235448Z-r1-string-slugify-1876884.
  • CTO-WORK-020 remains blocked because no real provider/model has been approved and no real Case Stage 2 pass report exists.

Decision Record Template For CTO-WORK-020

This template clarifies the decision required by CTO-WORK-020; it does not approve a provider.

  • decision_status: not_decided, external_provider_approved, or local_provider_required.
  • provider_class: external_anthropic or local_case_compatible.
  • provider: exact provider string, or empty while blocked.
  • model: exact model string, or empty while blocked.
  • approval_source: JP approval reference or governed Core route reference.
  • credential_source_class: credential class only; no secret value.
  • allowed_network_class: allowed network class for this provider.
  • review_trigger: expiry, date, or condition that forces review.
  • evidence_sources: references to existing admission/build evidence, not copied runtime evidence.
  • effect: CTO-WORK-020 remains blocked until admitted provider/model and real Stage 2 pass report exist.

Allowed pending states:

  • not_decided: no provider/model may run.
  • local_provider_required: no external provider may run; create a Case-compatible local provider route first.
  • external_provider_approved: may proceed only when the approval source, credential source class, allowed network class, and admission JSON are recorded.