hermes/cto
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
99b5353e51
cto/.sot/03-PROTOCOLS/CTO-CASE-AGENT-PROTOCOL-BLOCKER.md
Svrnty 99b5353e51 Record Spark vLLM Case evidence
2026-05-31 22:26:39 -04:00

12 KiB
Raw Blame History

title status lifecycle_classification owner created last_reviewed core_promotion_status route
CTO Case Agent Protocol Blocker draft sot jp 2026-06-01 2026-06-01 not-promoted cto

CTO Case Agent Protocol Blocker

Local planning SOT only. Not a Core Protocol. Not active Core authority.

CTO-WORK-028 - Case Agent Result Protocol Blocker

Status: blocked.

Record the first admitted real Case Stage 2 run after OpenAI Codex model admission. The run proves that provider/model admission now reaches Case execution, but does not prove Stage 2. Case failed before producing a workspace diff because its implementer agent result did not satisfy the Case result-envelope contract.

The later admitted Qwen local run reproduced the same result-envelope failure after Case process start. This makes the active blocker the Case agent-result protocol seam, not model admission.

Acceptance:

  • Real Case Stage 2 remains blocked until Case produces a Harness Evidence Interface pass report.
  • The admitted provider/model pair evidence includes openai-codex / gpt-5.5 and qwen-local / qwen3.6-35b-a3b.
  • The admission files remain .sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION.openai-codex-gpt-5.5.json and .sot/03-PROTOCOLS/CTO-CASE-MODEL-PROVIDER-ADMISSION.qwen-local-qwen3.6-35b-a3b.json.
  • Evidence must show case_process_started: true before this blocker is accepted as the current blocker.
  • Evidence must show case_model_admission_status: admitted.
  • Evidence must show no target repository path was inspected or copied.
  • Evidence must show no workspace patch was produced.
  • Evidence must show tests did not pass.
  • The next implementation route must happen through the Hermes CTO harness seam, a Case-compatible provider adapter seam, or an external compatibility layer.
  • The next implementation route must not mutate Cortex Core, vendor Case source, or external developer repositories.
  • No real-repo, copied-repo, sandbox-repo, owned-repo, default-candidate, or Core promotion stage may use this failed run as pass evidence.

Evidence - 2026-06-01

  • Harness command class: real Case Stage 2 artificial fixture.
  • Run artifact directory: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T013918Z-r1-string-slugify-2381028.
  • Case binary path used by harness: /tmp/workos-case/dist/ca.
  • Case source pin for the built binary: 7959ac917cdeb0983b4aaa20bb9f42021747fed8.
  • Report status: fail.
  • Backend: case.
  • Backend exit code: 1.
  • Case process started: true.
  • Case model provider: openai-codex.
  • Case model: gpt-5.5.
  • Case model admission status: admitted.
  • Source admission status: not_admitted.
  • No target inspection proof: stage2-no-target-inspection.json.
  • Changed files: none.
  • Patch artifact: patch.diff.
  • Patch digest: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
  • Tests command: python3 -m pytest -q.
  • Tests passed: false.
  • Required events passed: false.
  • Report blocker: case engine failed with exit code 1.
  • Case stderr evidence: implementer failed with AGENT_RESULT start delimiter not found.
  • Case stderr evidence: retry classified the failure as agent-protocol-error.
  • Case stdout evidence: unattended mode auto-selected Abort.
  • Result: Stage 2 is still blocked.

Qwen Local Evidence - 2026-06-01

  • Harness command class: real Case Stage 2 artificial fixture.
  • Run artifact directory: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T015208Z-r1-string-slugify-2478256.
  • Case binary path used by harness: /tmp/workos-case/dist/ca.
  • Case source pin for the built binary: 7959ac917cdeb0983b4aaa20bb9f42021747fed8.
  • Report status: fail.
  • Backend: case.
  • Backend exit code: 1.
  • Case process started: true.
  • Case model provider: qwen-local.
  • Case model: qwen3.6-35b-a3b.
  • Case model admission status: admitted.
  • Source admission status: not_admitted.
  • No target inspection proof: stage2-no-target-inspection.json.
  • Changed files: none.
  • Patch artifact: patch.diff.
  • Patch digest: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
  • Tests command: python3 -m pytest -q.
  • Tests passed: false.
  • Required events passed: false.
  • Report blocker: case engine failed with exit code 1.
  • Case stderr evidence: implementer failed with AGENT_RESULT start delimiter not found.
  • Case stderr evidence: retry classified the failure as agent-protocol-error.
  • Case stdout evidence: unattended mode auto-selected Abort.
  • Result: Stage 2 is still blocked.

Current Interpretation

This is a protocol compatibility blocker, not a provider approval or model admission blocker.

Two admitted provider/model paths reached Case. Case then failed because the implementer agent did not return output framed by the Case AGENT_RESULT delimiter contract. The evidence does not prove whether the defect is Case provider configuration, provider adapter behavior, model output framing, or harness invocation shape.

Narrowed Interpretation - 2026-06-01

Hermes commit 5db23c7 Fail closed on Case Codex auth gap narrows the current OpenAI Codex path blocker.

The current known OpenAI Codex blocker is an auth-bridge gap:

  • Case's pipeline SDK path constructs its Pi Agent runtime directly.
  • That path does not pass Pi AuthStorage OAuth headers into streamSimple.
  • Pi env API-key lookup does not map openai-codex to an environment API key.
  • Hermes now blocks openai-codex before case_process_started unless an explicit non-vendor auth bridge is proven.

This does not prove that any local provider path passes Stage 2. It only prevents repeating the misleading malformed-output run for the openai-codex path.

Required Next Route

The next useful route is a small Case agent protocol compatibility investigation. It should answer only this question:

What minimal non-vendor seam makes admitted Case execution return the required
AGENT_RESULT envelope and produce a Stage 2 artificial fixture diff?

Allowed next actions:

  • Inspect Case provider adapter behavior read-only.
  • Inspect Hermes CTO Case invocation behavior.
  • Add fail-closed classification in Hermes CTO harness if needed.
  • Add a compatibility shim only outside vendor Case source.
  • Admit and test the existing Pi local provider id qwen-local only through the Harness Evidence Interface.
  • Re-run real Case Stage 2 only after a specific protocol compatibility change exists.

Forbidden next actions:

  • Do not edit /tmp/workos-case as the durable solution.
  • Do not mark Stage 2 validated from this run.
  • Do not promote Case to copied repo, sandbox repo, owned repo, or default candidate.
  • Do not write provider secrets to SOT, argv, task files, backend logs, reports, traces, or commits.

Hermes Classifier Evidence - 2026-06-01

  • Hermes commit: 48d487a Classify Case agent protocol failures.
  • Hermes commit: 798fb5a Harden Case protocol failure marker.
  • 48d487a adds a CTO Harness AGENT_RESULT protocol appendix to the Case task markdown.
  • 48d487a adds a fail-closed protocol marker path: backend/provider-agent-protocol.txt.
  • 798fb5a hardens report-time classification so missing AGENT_RESULT evidence is recorded even if backend marker creation is missed.
  • Focused validator passed: python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json.
  • Aggregate validator passed: harness/evals/health.sh --json.
  • Post-merge aggregate validator passed: harness/evals/health.sh --json.

Qwen Local Classified Runtime Evidence - 2026-06-01

  • Run artifact directory: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T020117Z-r1-string-slugify-2566310.
  • Report status: fail.
  • Backend: case.
  • Backend exit code: 1.
  • Case process started: true.
  • Case model provider: qwen-local.
  • Case model: qwen3.6-35b-a3b.
  • Case model admission status: admitted.
  • Failure reason: case agent result protocol failed.
  • Protocol marker: backend/provider-agent-protocol.txt.
  • Changed files: none.
  • Patch artifact: patch.diff.
  • Patch digest: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
  • Tests passed: false.
  • Required events passed: false.
  • Result: Stage 2 is still blocked.
  • Current next route remains a Case/Pi runtime protocol compatibility fix, not another admission record.

Isolated Pi Config Evidence - 2026-06-01

  • Hermes commit: 09b5851 Isolate Case Pi provider config.
  • The Hermes CTO Case harness now sets PI_CODING_AGENT_DIR under the run artifact directory before invoking Case.
  • The harness writes isolated Pi auth state under backend/case-data/pi-agent/auth.json.
  • Local Case providers now require explicit CTO_HARNESS_CASE_LOCAL_BASE_URL.
  • Missing local provider config writes backend/provider-local-config-unavailable.txt.
  • Missing local provider config blocks before case_process_started.
  • Focused validator passed: python3 harness/runner/validate-case-provider-adapter.py --harness-root harness --json.
  • Focused validator artifact: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T020732Z-r1-string-slugify-2609546.
  • Aggregate validator passed before merge: harness/evals/health.sh --json.
  • Aggregate validator artifact before merge: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T020741Z-r1-string-slugify-2611203.
  • Post-merge aggregate validator passed: harness/evals/health.sh --json.
  • Post-merge aggregate validator artifact: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T020801Z-r1-string-slugify-2613843.
  • Real Qwen local config-gate proof artifact: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T020847Z-r1-string-slugify-2619644.
  • Real Qwen local config-gate status: blocked.
  • Real Qwen local config-gate failure reason: provider unavailable.
  • Real Qwen local config-gate Case process started: false.
  • This removes the ambient ~/.pi/agent dependency from the harness path.
  • This does not prove Case can produce the required AGENT_RESULT envelope.
  • CTO-WORK-028 remains blocked until a configured local provider or another admitted provider returns a valid result envelope and produces a Stage 2 artificial fixture diff.

Spark Endpoint Config Dependency - 2026-06-01

  • CTO-WORK-030 must be resolved before another configured Qwen local run can retest the Case AGENT_RESULT protocol path.
  • Until CTO_HARNESS_CASE_LOCAL_BASE_URL is supplied, the harness blocks before Case starts.
  • The agent protocol blocker remains unproven for the isolated Spark endpoint path until Case reaches execution again and returns or fails the required result envelope.

Case Qwen Loop Entrypoint Evidence - 2026-06-01

  • Hermes commit: 6c453ee Add Case Qwen loop entrypoint.
  • The new harness/evals/case-qwen-loop.sh --json command is the next standard path for retesting the Case AGENT_RESULT protocol after Spark endpoint config is supplied.
  • This does not resolve the protocol blocker because Case has not reached execution through the configured isolated Spark endpoint path yet.

Spark vLLM Qwen Loop Evidence - 2026-06-01

  • Spark1 exposed a reachable OpenAI-compatible vLLM model route for qwen3.6-35b-a3b.
  • The endpoint value was supplied only at runtime and is not recorded in SOT.
  • Standard command class: harness/evals/case-qwen-loop.sh --json.
  • Run artifact directory: /home/svrnty/.hermes/profiles/cto-planb/harness-runs/20260601T022535Z-r1-string-slugify-2731603.
  • Report status: fail.
  • Backend: case.
  • Backend exit code: 1.
  • Case process started: true.
  • Case model provider: qwen-local.
  • Case model: qwen3.6-35b-a3b.
  • Case model admission status: admitted.
  • Source admission status: not_admitted.
  • Failure reason: case agent result protocol failed.
  • Protocol marker: backend/provider-agent-protocol.txt.
  • Changed files: none.
  • Patch artifact: patch.diff.
  • Patch digest: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
  • Tests passed: false.
  • Required events passed: false.
  • Result: Stage 2 is still blocked.
  • Current next route is a Case/Pi result-envelope compatibility fix outside vendor Case source.