76 lines
5.1 KiB
Markdown
76 lines
5.1 KiB
Markdown
---
|
|
name: cto-case-failure-fixture-matrix
|
|
tier: local
|
|
status: draft
|
|
owner: jp
|
|
source: sot/03-PROTOCOLS/CTO-CASE-CANDIDATE-BACKEND-PRD.md
|
|
created: 2026-05-31
|
|
last_reviewed: 2026-05-31
|
|
lifecycle_classification: planning
|
|
core_promotion_status: not-promoted
|
|
description: Child-local failure fixture matrix for later Case adapter fail-closed testing.
|
|
---
|
|
|
|
# CTO Case Failure Fixture Matrix
|
|
|
|
Local planning SOT only. Not a Core Protocol. Not active Core authority.
|
|
|
|
## Purpose
|
|
|
|
Define the required failure fixtures for later Case adapter testing before any executable Case integration exists.
|
|
|
|
This matrix turns failure-mode language into concrete expected evidence. It prevents the future adapter from proving only the happy path.
|
|
|
|
## Non-Authority Notice
|
|
|
|
This matrix does not authorize Runtime behavior, WebUI Product behavior, Case execution, real-repo mutation, merge, deploy, push, close, vendor-source mutation, external developer repository mutation, or Core promotion.
|
|
|
|
## Matrix Rules
|
|
|
|
- Each row maps to the CTO Harness Evidence Interface.
|
|
- Each row must produce `report.json`, `events.normalized.jsonl`, `patch.diff`, `test.log`, and `trace.jsonl` when implemented.
|
|
- Each row must produce `artifact_digests` and freshness proof when implemented.
|
|
- Each row must record a blocker reason in `report.json`.
|
|
- Each row must emit a normalized event when the failure class has one.
|
|
- Each row must fail closed with `status` set to `fail` or `blocked`.
|
|
- Each row must use nonzero exit behavior when the adapter contract marks the failure executable.
|
|
- This matrix is planning-only and does not run Case.
|
|
|
|
## Required Failure Fixtures
|
|
|
|
| Fixture | Trigger | Expected blocker reason | Normalized event | Report status | Exit behavior | Evidence mapping |
|
|
| --- | --- | --- | --- | --- | --- | --- |
|
|
| no-diff | Backend reports success but no source diff exists when a diff is required. | `no_diff` | `git.diff.checked` | `fail` | nonzero | `patch.diff` exists and is empty; `changed_files` is empty; `blockers` includes `no_diff`. |
|
|
| disallowed-file | Backend changes a path outside `allowed_paths`. | `disallowed_file_change` | `git.diff.checked` | `fail` | nonzero | `changed_files` lists offending path; `allowed_writes_passed` is `false`; `patch.diff` captures the change. |
|
|
| failed-tests | Verification command exits nonzero. | `verification_failed` | `verification.completed` | `fail` | nonzero | `test.log` captures failing command output; `verification` records command and nonzero exit. |
|
|
| missing-test-command | Task contract requires verification but no command is available. | `missing_test_command` | `verification.completed` | `blocked` | nonzero | `test.log` states no command was available; `verification` records missing command. |
|
|
| missing-event | Required normalized event is absent or out of order. | `missing_required_event` | `run.completed` | `fail` | nonzero | `events.normalized.jsonl` is present; validator identifies missing or out-of-order event. |
|
|
| reviewer-reject | Case reviewer rejects the result. | `reviewer_rejected` | `verification.completed` | `blocked` | nonzero | backend raw logs record reviewer rejection; normalized evidence records blocked result without approving mutation. |
|
|
| approval-denied | Human approval is required and denied. | `approval_denied` | `approval.denied` | `blocked` | nonzero | approval event records actor, scope, mutation mode, timestamp, and denial. |
|
|
| timeout | Backend or verification exceeds configured timeout. | `timeout` | `run.completed` | `blocked` | nonzero | `trace.jsonl` records timeout point; `report.json` records timeout blocker. |
|
|
| dirty-starting-tree | Target repository is dirty before backend work starts when clean start is required. | `dirty_starting_tree` | `task.contract.created` | `blocked` | nonzero | `trace.jsonl` records preflight failure; no mutation occurs. |
|
|
| dirty-ending-tree | Run leaves untracked or unintended dirty state after checks. | `dirty_ending_tree` | `run.completed` | `fail` | nonzero | `trace.jsonl` records ending state check; `report.json` names dirty paths. |
|
|
| artifact-write-failure | Required artifact cannot be written. | `artifact_write_failure` | `run.completed` | `fail` | nonzero | `report.json` may be partial only if write failure happens after report creation; trace records failed artifact path. |
|
|
| provider-unavailable | Backend provider, model, CLI, or dependency is unavailable. | `provider_unavailable` | `run.completed` | `blocked` | nonzero | backend raw logs record provider failure; no source mutation occurs. |
|
|
|
|
## Acceptance Rule
|
|
|
|
A future Case adapter cannot progress to staged proof until these fixture classes have expected evidence defined in the adapter validator.
|
|
|
|
The first executable implementation may start with a subset only if the staged proof gate explicitly marks the remaining rows as blocked and not accepted for default candidacy.
|
|
|
|
## Validator Expectations
|
|
|
|
Current validation is planning-only and checks this matrix exists with required rows.
|
|
|
|
Later adapter validation must execute or simulate each row and verify:
|
|
|
|
- blocker reason;
|
|
- normalized event;
|
|
- report status;
|
|
- exit behavior;
|
|
- required artifact presence;
|
|
- digest presence;
|
|
- freshness proof;
|
|
- no hidden source mutation.
|