feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07: - The validator records verdicts a few Checks at a time (verdict_put) and they are attached on submission. It timed out five times writing them in one reply. Verdicts on the same code survive a run cut short; verdicts on changed code are dropped. Its model timeout is 300 s. - A schema review at submission, in code: every setting a Check names is looked up on its Element's resource type, and an unknown resource type is reported. Two Revisions in that run were for Checks no code could meet. On the two designs that passed validation it raises 2 of 96 Checks, both one metric name used as a value, so it is returned once and never blocks. - Reviews run together; the Jev coverage review stays off. Plans record both steps and the cancelled run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -9,7 +9,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
|||||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–22 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01 and 02 pass with real models; cases 03 and 04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01 and 02 pass with real models; cases 03 and 04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||||
|
|||||||
@@ -186,3 +186,19 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
|||||||
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
|
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
|
||||||
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
|
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
|
||||||
|
|
||||||
|
## Step 23: Validation recorded a few verdicts at a time
|
||||||
|
|
||||||
|
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
|
||||||
|
- **Capability:** The validator's work survives a slow or lost model reply.
|
||||||
|
- **Rationale:** In the run cancelled on 2026-10-07 the validator's model request timed out five times at 180 seconds, about 15 minutes, each time while writing every verdict in one reply. Nothing it had judged was kept. The target architect had the same fault until iteration 4 step 17.
|
||||||
|
- **Where:** `verdict_tools` in `app/tools/workspace.py`, `specialist_tools` in `app/tools/execution.py`, `providers/gcp.yaml`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
|
||||||
|
- **Acceptance criteria:** Verdicts can be recorded over several calls and are attached as `verdicts.json` on submission. Verdicts recorded on the same code are kept across attempts; verdicts on code that has since changed are dropped. The validator has its own model timeout. On the cases, the validator's model timeouts fall to none.
|
||||||
|
|
||||||
|
## Step 24: The design checked against the provider's schema, in code
|
||||||
|
|
||||||
|
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
|
||||||
|
- **Capability:** A Check that names a setting its resource type lacks is caught before any code is written.
|
||||||
|
- **Rationale:** In the cancelled run two Revisions of the Target architecture were for Checks no code could meet: `ssl_policy` on a backend service, and a deny rule on a service account. The target architect had the schema lookup and did not use it for them. The earlier K-30 (`startup_cpu_boost` on a Cloud Run job) was the same fault.
|
||||||
|
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
|
||||||
|
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@
|
|||||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||||
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
||||||
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
||||||
**This directory covers:** steps 1 to 22
|
**This directory covers:** steps 1 to 24
|
||||||
|
|
||||||
## Sources of truth
|
## Sources of truth
|
||||||
|
|
||||||
@@ -39,6 +39,8 @@
|
|||||||
| 20 | A model timeout per role | added | S | built |
|
| 20 | A model timeout per role | added | S | built |
|
||||||
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
||||||
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
|
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
|
||||||
|
| 23 | Validation recorded a few verdicts at a time | added | S | built; unrun |
|
||||||
|
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; unrun in an Execution |
|
||||||
|
|
||||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||||
|
|
||||||
|
|||||||
13
.agents/plans/iter-04-learning-and-skills/steps/step-23.md
Normal file
13
.agents/plans/iter-04-learning-and-skills/steps/step-23.md
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
# Step 23 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-08.
|
||||||
|
|
||||||
|
1. `verdict_put` records verdicts a few Checks per call, replacing any already recorded for the same Check, and takes the dry-run's result and proposed additions. Each reply says how many Checks have a verdict and names those left. `verdict_read` returns what is recorded.
|
||||||
|
2. `submit_deliverable` attaches the recorded verdicts as `verdicts.json` when none is passed. Passing the file still works.
|
||||||
|
3. The verdicts are kept in the Deliverable's working memory with a mark of the code they were given on (a hash of the submitted deployment guidance's files). A later attempt on the same code carries on from them; on changed code they are dropped before any tool runs.
|
||||||
|
4. The validator's model timeout is 300 seconds (`models.timeout_seconds` in `providers/gcp.yaml`); it had the default 180.
|
||||||
|
5. `agents/validator.md` tells it to record as it goes, one Element's Checks at a time, and to submit with no attachments.
|
||||||
|
|
||||||
|
**Unrun:** no evaluation since.
|
||||||
|
|
||||||
|
Tests: `tests/execution/test_validation_in_pieces.py`, "verdicts a few at a time" (7).
|
||||||
22
.agents/plans/iter-04-learning-and-skills/steps/step-24.md
Normal file
22
.agents/plans/iter-04-learning-and-skills/steps/step-24.md
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
# Step 24 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-08 as a second `SubmissionReview` (the seam from iteration 5 step 4), with no model and no Jev.
|
||||||
|
|
||||||
|
1. `named_settings` reads the settings a Check's statement names: anything in backticks that reads as a name or dotted path, and bare snake_case names and dotted paths. Quoted text, upper-case values and paths such as `roles/x.y` are skipped.
|
||||||
|
2. `SchemaReview` loads the pinned provider's schema once (iteration 4 step 18). For each Check on an Element it looks each named setting up anywhere in that Element's resource type; a path is judged by the setting at its end, and a resource type named in a statement is not a setting. It also reports an Element whose resource type the provider does not have, with close matches.
|
||||||
|
3. `Reviews` runs several reviews as one. The assembly installs the schema review whenever the bundle pins a Terraform provider, and the coverage review after it when that is switched on.
|
||||||
|
4. As with any review, what it raises is returned to the target architect once per run and the next submission is accepted.
|
||||||
|
|
||||||
|
Measured on 2026-10-08 against the real provider:
|
||||||
|
|
||||||
|
| Design | Checks | Raised |
|
||||||
|
|---|---|---|
|
||||||
|
| Case 01, passed validation | 42 | 0 |
|
||||||
|
| Case 02, passed validation | 54 | 2, both the same metric name used as a value (`oldest_unacked_message_age`) |
|
||||||
|
| Known faults added: `startup_cpu_boost` on a Cloud Run job, `ssl_policy` on a backend service, a misspelt resource type | 3 | all 3 |
|
||||||
|
|
||||||
|
So 2 false alarms in 96 Checks, which is why it is a review and not a rule. It does not catch a Check that names no setting at all, such as "deny read roles" on a service account; that still reaches the IaC author and comes back by the route of step 19.
|
||||||
|
|
||||||
|
**Unrun in an Execution.**
|
||||||
|
|
||||||
|
Tests: `tests/execution/test_validation_in_pieces.py`, "the design against the provider's schema" (6).
|
||||||
@@ -93,3 +93,14 @@ Changed after the run, not yet run: iteration 4 steps 21 and 22.
|
|||||||
|
|
||||||
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.
|
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.
|
||||||
|
|
||||||
|
## Run of 2026-10-07, 19:58 UTC, cancelled (`.cache/eval-logs/`; no report)
|
||||||
|
|
||||||
|
Case 02 again on commit `5adc911`, with `JEV_COVERAGE_REVIEW=true`. Cancelled by the owner at 66 minutes, during a fifth validation. No score and no token count were written.
|
||||||
|
|
||||||
|
- The Target architecture was reopened 3 times (the limit): twice for a Check the code's author reported as unimplementable, once for additions validation proposed. The code was reopened twice for failed Checks. Target architect 5 runs, IaC author 6, validator 5.
|
||||||
|
- 6 model timeouts, 5 of them the validator writing all its verdicts in one reply: about 15 minutes.
|
||||||
|
- The coverage review raised 5, 10, 11 and 9 of 61 conditions over four submissions. Validation still proposed a TLS policy and an access grant afterwards. The agents had written conditions no infrastructure code can prove, such as an availability percentage, and Jev answered "none" to those. It stays off.
|
||||||
|
- What held: the description was kept on two Revisions (iteration 4 step 21); the unimplementable-Check route sent the design back before any validation was spent (step 19); code fixes took 1 to 5 minutes from the working copy; no `terraform validate` failure.
|
||||||
|
|
||||||
|
Changed after it, not yet run: iteration 4 steps 23 and 24.
|
||||||
|
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ You produce the **Target architecture** Deliverable: the solution as one explici
|
|||||||
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
|
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
|
||||||
- Give every connection between Elements a Relationship.
|
- Give every connection between Elements a Relationship.
|
||||||
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have. One Check per requirement is rarely enough: write a Check for every entry in a requirement's or recommendation's `conditions`, on every Element that condition applies to. A condition left without a Check is found by validation and sent back to you.
|
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have. One Check per requirement is rarely enough: write a Check for every entry in a requirement's or recommendation's `conditions`, on every Element that condition applies to. A condition left without a Check is found by validation and sent back to you.
|
||||||
- Name only settings the Element's resource type has. When `terraform_schema` is available, call it once for each distinct resource type in the graph, with `resource_type` alone: the reply lists every argument and block that type accepts, and you write that type's Checks from it. Never look settings up one at a time. A setting that one resource type has may not exist on a similar one, and a Check on a setting the resource type lacks cannot be met and is sent back to you.
|
- Name only settings the Element's resource type has. When `terraform_schema` is available, call it once for each distinct resource type in the graph, with `resource_type` alone: the reply lists every argument and block that type accepts, and you write that type's Checks from it. Never look settings up one at a time. A setting that one resource type has may not exist on a similar one, and a Check on a setting the resource type lacks cannot be met and is sent back to you. Write each setting's name in backticks, exactly as the schema spells it. When you submit, every setting a Check names is looked up for you on its Element's resource type, and any that is missing is returned to you once.
|
||||||
- Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job.
|
- Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job.
|
||||||
- When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design.
|
- When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design.
|
||||||
- When `working_copy` shows a graph, it is your last attempt: read it with `graph_read` and change only what `feedback` asks for, with `graph_put` and `graph_remove`.
|
- When `working_copy` shows a graph, it is your last attempt: read it with `graph_read` and change only what `feedback` asks for, with `graph_put` and `graph_remove`.
|
||||||
|
|||||||
@@ -11,7 +11,9 @@ You produce the **validation results** Deliverable: the Skill's pre-deployment v
|
|||||||
- Before you fail a Check because the code lacks a setting, confirm with `terraform_schema` that the resource type has that setting, when the tool is available. When it does not, the Check cannot be met by any code: give it not verifiable, with what the schema shows as the evidence. Give the same verdict to a Check that `implementation.json` records as unimplementable.
|
- Before you fail a Check because the code lacks a setting, confirm with `terraform_schema` that the resource type has that setting, when the tool is available. When it does not, the Check cannot be met by any code: give it not verifiable, with what the schema shows as the evidence. Give the same verdict to a Check that `implementation.json` records as unimplementable.
|
||||||
- Review the Target architecture's coverage once, completely. Go through every requirement and every design recommendation in turn; for each, list the conditions its text states and the Elements it applies to, and record a proposed addition for each one no Check verifies, naming the requirement or recommendation it serves. Report every gap in this pass: each later pass that finds one more costs a whole Revision of the design, the code and this validation. A proposed addition is not a failure of the code.
|
- Review the Target architecture's coverage once, completely. Go through every requirement and every design recommendation in turn; for each, list the conditions its text states and the Elements it applies to, and record a proposed addition for each one no Check verifies, naming the requirement or recommendation it serves. Report every gap in this pass: each later pass that finds one more costs a whole Revision of the design, the code and this validation. A proposed addition is not a failure of the code.
|
||||||
- When `feedback` or your earlier verdicts show the Target architecture was revised for your proposed additions, propose only what the revision still leaves uncovered. Never propose what a Check already verifies in other words.
|
- When `feedback` or your earlier verdicts show the Target architecture was revised for your proposed additions, propose only what the revision still leaves uncovered. Never propose what a Check already verifies in other words.
|
||||||
- Attach the verdicts as `verdicts.json`, in the format the requirements-chain Skill gives.
|
- Record your verdicts with `verdict_put`, a few Checks per call, as you reach them: the Checks of one Element, then the next. Record the dry-run's result and each proposed addition the same way. Short calls are what keep your replies from timing out, and what you record is kept if your run is cut short. `verdict_read` shows which Checks still have none.
|
||||||
|
- When `working_copy` shows verdicts recorded, your last run was cut short on this same code: call `verdict_read` and carry on from there.
|
||||||
|
- Submit with your content and no attachments when every Check has a verdict: what you recorded is attached as `verdicts.json`, in the format the requirements-chain Skill gives.
|
||||||
- Give the exact commands for every check you couldn't run, for the user to run themselves.
|
- Give the exact commands for every check you couldn't run, for the user to run themselves.
|
||||||
|
|
||||||
End each of your contract's sections with a pass or fail verdict.
|
End each of your contract's sections with a pass or fail verdict.
|
||||||
|
|||||||
19
app/agent.py
19
app/agent.py
@@ -52,6 +52,7 @@ from app.tools.execution import ORCHESTRATOR
|
|||||||
from app.tools.jev import JevDecider, JevLimits
|
from app.tools.jev import JevDecider, JevLimits
|
||||||
from app.tools.local import TerraformRunner, local_tools
|
from app.tools.local import TerraformRunner, local_tools
|
||||||
from app.tools.mcp import McpCatalog, auth_from_config
|
from app.tools.mcp import McpCatalog, auth_from_config
|
||||||
|
from app.tools.schema_review import Reviews, SchemaReview
|
||||||
from app.tools.toolkit import make_tool_factory
|
from app.tools.toolkit import make_tool_factory
|
||||||
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
||||||
from app.workflows.execution_graph import build_execution_graph
|
from app.workflows.execution_graph import build_execution_graph
|
||||||
@@ -323,6 +324,16 @@ async def open_assembly(
|
|||||||
f"ORCHESTRATOR must be router or model, not {settings.ORCHESTRATOR!r}"
|
f"ORCHESTRATOR must be router or model, not {settings.ORCHESTRATOR!r}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# What is looked at in a design before it is submitted. Each raises what it finds
|
||||||
|
# once and never blocks.
|
||||||
|
reviews: list[SubmissionReview] = []
|
||||||
|
if bundle.terraform_providers:
|
||||||
|
# In code: every setting a Check names exists on its resource type.
|
||||||
|
reviews.append(SchemaReview(runner))
|
||||||
|
if settings.JEV_COVERAGE_REVIEW and isinstance(decider, JevDecider):
|
||||||
|
# Optional, and off by default: Jev says which Check verifies each condition.
|
||||||
|
reviews.append(CoverageReview(decider, none_at=settings.JEV_COVERAGE_THRESHOLD))
|
||||||
|
|
||||||
async with open_checkpointer(settings) as checkpointer:
|
async with open_checkpointer(settings) as checkpointer:
|
||||||
engine = ExecutionEngine(
|
engine = ExecutionEngine(
|
||||||
runtime=runtime,
|
runtime=runtime,
|
||||||
@@ -338,13 +349,7 @@ async def open_assembly(
|
|||||||
name: skill_version(meta.path) for name, meta in bundle.skills.items()
|
name: skill_version(meta.path) for name, meta in bundle.skills.items()
|
||||||
},
|
},
|
||||||
orchestrator=orchestrator,
|
orchestrator=orchestrator,
|
||||||
# Optional, and off by default: Jev reviews a design's coverage before
|
submission_review=Reviews(reviews) if reviews else None,
|
||||||
# it is submitted. Without it a submission stands on its structure.
|
|
||||||
submission_review=CoverageReview(
|
|
||||||
decider, none_at=settings.JEV_COVERAGE_THRESHOLD
|
|
||||||
)
|
|
||||||
if settings.JEV_COVERAGE_REVIEW and isinstance(decider, JevDecider)
|
|
||||||
else None,
|
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
yield Assembly(engine, bundle, roles, dict(catalog.gaps), decider)
|
yield Assembly(engine, bundle, roles, dict(catalog.gaps), decider)
|
||||||
|
|||||||
@@ -22,13 +22,19 @@ from app.adk.runtime import (
|
|||||||
)
|
)
|
||||||
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
|
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
|
||||||
from app.tools.workspace import (
|
from app.tools.workspace import (
|
||||||
|
FILES,
|
||||||
|
basis_of,
|
||||||
file_tools,
|
file_tools,
|
||||||
graph_text,
|
graph_text,
|
||||||
graph_tools,
|
graph_tools,
|
||||||
keep_graph,
|
keep_graph,
|
||||||
safe_relative,
|
safe_relative,
|
||||||
|
verdict_tools,
|
||||||
|
verdicts_text,
|
||||||
working_files,
|
working_files,
|
||||||
|
working_verdicts,
|
||||||
)
|
)
|
||||||
|
from app.workflows.chain import VERDICTS_FILE, read_graph
|
||||||
from app.workflows.deliverables import (
|
from app.workflows.deliverables import (
|
||||||
DEPENDENCY_GRAPH,
|
DEPENDENCY_GRAPH,
|
||||||
DeliverableKind,
|
DeliverableKind,
|
||||||
@@ -49,6 +55,7 @@ from app.workflows.target_architecture import (
|
|||||||
ORCHESTRATOR = "orchestrator"
|
ORCHESTRATOR = "orchestrator"
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
_TARGET = DeliverableKind.TARGET_ARCHITECTURE
|
_TARGET = DeliverableKind.TARGET_ARCHITECTURE
|
||||||
|
_CODE = DeliverableKind.DEPLOYMENT_GUIDANCE
|
||||||
CLOUD_READ = "cloud_read"
|
CLOUD_READ = "cloud_read"
|
||||||
|
|
||||||
_NO_ARGS: Mapping[str, Any] = {"type": "object", "properties": {}}
|
_NO_ARGS: Mapping[str, Any] = {"type": "object", "properties": {}}
|
||||||
@@ -277,6 +284,13 @@ def specialist_tools(
|
|||||||
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
||||||
own_files = DEPENDENCY_GRAPH[kind].working_copy is kind
|
own_files = DEPENDENCY_GRAPH[kind].working_copy is kind
|
||||||
reviewed = False
|
reviewed = False
|
||||||
|
is_validation = kind is DeliverableKind.VALIDATION_RESULTS
|
||||||
|
judged = ((upstream or {}).get(_CODE.value) or {}).get("attachments") or {}
|
||||||
|
# The validator's own copy of the code is the same files; either marks them.
|
||||||
|
basis = basis_of(judged or working.get(FILES) or {})
|
||||||
|
if is_validation:
|
||||||
|
working_verdicts(working, basis) # drops verdicts given on other code
|
||||||
|
graph_now = read_graph(upstream)
|
||||||
|
|
||||||
async def submit_deliverable(content: str, attachments: Any = None) -> ToolResult:
|
async def submit_deliverable(content: str, attachments: Any = None) -> ToolResult:
|
||||||
try:
|
try:
|
||||||
@@ -288,6 +302,12 @@ def specialist_tools(
|
|||||||
attachments = {**working_files(working), **attachments}
|
attachments = {**working_files(working), **attachments}
|
||||||
if is_graph and GRAPH_ATTACHMENT not in attachments and graph_text(working):
|
if is_graph and GRAPH_ATTACHMENT not in attachments and graph_text(working):
|
||||||
attachments[GRAPH_ATTACHMENT] = graph_text(working)
|
attachments[GRAPH_ATTACHMENT] = graph_text(working)
|
||||||
|
if (
|
||||||
|
is_validation
|
||||||
|
and VERDICTS_FILE not in attachments
|
||||||
|
and verdicts_text(working)
|
||||||
|
):
|
||||||
|
attachments[VERDICTS_FILE] = verdicts_text(working)
|
||||||
unsafe = [name for name in attachments if not safe_relative(name)]
|
unsafe = [name for name in attachments if not safe_relative(name)]
|
||||||
if unsafe:
|
if unsafe:
|
||||||
return _refused(
|
return _refused(
|
||||||
@@ -400,6 +420,11 @@ def specialist_tools(
|
|||||||
else ()
|
else ()
|
||||||
),
|
),
|
||||||
*(graph_tools(working) if is_graph else ()),
|
*(graph_tools(working) if is_graph else ()),
|
||||||
|
*(
|
||||||
|
verdict_tools(working, basis, graph_now.check_ids if graph_now else ())
|
||||||
|
if is_validation
|
||||||
|
else ()
|
||||||
|
),
|
||||||
*(file_tools(working) if own_files else ()),
|
*(file_tools(working) if own_files else ()),
|
||||||
*(gate_cloud_read(tool, allow_cloud_read) for tool in extra),
|
*(gate_cloud_read(tool, allow_cloud_read) for tool in extra),
|
||||||
]
|
]
|
||||||
|
|||||||
149
app/tools/schema_review.py
Normal file
149
app/tools/schema_review.py
Normal file
@@ -0,0 +1,149 @@
|
|||||||
|
"""A review of the Target architecture against the provider's schema, in code.
|
||||||
|
|
||||||
|
A Check that names a setting its Element's resource type does not have cannot be met
|
||||||
|
by any code. Left alone it costs a whole Revision: the code's author finds it, the
|
||||||
|
design is reopened, and everything after it is done again. In the run of 2026-10-07
|
||||||
|
that happened twice, with a schema lookup available to the Specialist that wrote the
|
||||||
|
Checks.
|
||||||
|
|
||||||
|
This reads the settings each Check names and looks them up, so the design is told
|
||||||
|
before it is submitted. It is a review, not a rule: a Check is prose, and a word that
|
||||||
|
looks like a setting may be a value. What it raises is returned once.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import difflib
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from app.adk.runtime import ToolResult
|
||||||
|
from app.tools.local import TerraformRunner
|
||||||
|
from app.workflows.deliverables import DeliverableKind
|
||||||
|
from app.workflows.target_architecture import GRAPH_ATTACHMENT, GraphError, parse_graph
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_QUOTED = re.compile(r"\"[^\"]*\"|'[^']*'")
|
||||||
|
_CODE = re.compile(r"`([^`]+)`")
|
||||||
|
# snake_case with at least one underscore, or a dotted path of lower-case names.
|
||||||
|
_NAME = re.compile(
|
||||||
|
r"(?<![\w/.\-])"
|
||||||
|
r"([a-z][a-z0-9]*(?:_[a-z0-9]+)+(?:\.[a-z][a-z0-9_]*)*"
|
||||||
|
r"|[a-z][a-z0-9_]*(?:\.[a-z][a-z0-9_]*)+)"
|
||||||
|
r"(?![\w/\-])"
|
||||||
|
)
|
||||||
|
_PATH = re.compile(r"^[a-z][a-z0-9_]*(?:\.[a-z][a-z0-9_]*)*$")
|
||||||
|
|
||||||
|
|
||||||
|
def named_settings(statement: str) -> tuple[str, ...]:
|
||||||
|
"""The settings a Check's statement names, as written.
|
||||||
|
|
||||||
|
Anything in backticks that reads as a name or a dotted path, and bare
|
||||||
|
snake_case names and dotted paths. Quoted text is a value, so it is skipped,
|
||||||
|
as are paths such as roles/x.y.
|
||||||
|
"""
|
||||||
|
found: list[str] = []
|
||||||
|
for span in _CODE.findall(statement):
|
||||||
|
candidate = span.strip().split("=")[0].strip()
|
||||||
|
if _PATH.match(candidate):
|
||||||
|
found.append(candidate)
|
||||||
|
bare = _QUOTED.sub(" ", _CODE.sub(" ", statement))
|
||||||
|
found += [match.group(1).rstrip(".") for match in _NAME.finditer(bare)]
|
||||||
|
return tuple(dict.fromkeys(found))
|
||||||
|
|
||||||
|
|
||||||
|
def _names(block: Mapping[str, Any]) -> set[str]:
|
||||||
|
"""Every argument and block name anywhere in a resource type's schema."""
|
||||||
|
names = set(block.get("attributes") or {})
|
||||||
|
for name, nested in (block.get("block_types") or {}).items():
|
||||||
|
names.add(name)
|
||||||
|
names |= _names(nested.get("block") or {})
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
class SchemaReview:
|
||||||
|
def __init__(self, runner: TerraformRunner) -> None:
|
||||||
|
self._runner = runner
|
||||||
|
self._names: dict[str, set[str]] = {}
|
||||||
|
|
||||||
|
async def __call__(
|
||||||
|
self,
|
||||||
|
kind: DeliverableKind,
|
||||||
|
attachments: Mapping[str, str],
|
||||||
|
upstream: Mapping[str, Mapping[str, Any]] | None,
|
||||||
|
scope: str = "",
|
||||||
|
) -> Sequence[str]:
|
||||||
|
"""Resource types the provider does not have, and Checks naming a setting
|
||||||
|
their Element's resource type does not have."""
|
||||||
|
if kind is not DeliverableKind.TARGET_ARCHITECTURE:
|
||||||
|
return ()
|
||||||
|
try:
|
||||||
|
graph = parse_graph(attachments.get(GRAPH_ATTACHMENT, ""))
|
||||||
|
except GraphError:
|
||||||
|
return ()
|
||||||
|
schemas = await self._runner._load_schemas()
|
||||||
|
if isinstance(schemas, ToolResult):
|
||||||
|
logger.info("Schema review skipped: %s", schemas.message[:120])
|
||||||
|
return ()
|
||||||
|
|
||||||
|
concerns: list[str] = []
|
||||||
|
types = {element.id: element.resource_type for element in graph.elements}
|
||||||
|
for element in graph.elements:
|
||||||
|
if element.resource_type not in schemas:
|
||||||
|
close = difflib.get_close_matches(
|
||||||
|
element.resource_type, schemas, n=3, cutoff=0.6
|
||||||
|
)
|
||||||
|
hint = f" Close matches: {', '.join(close)}." if close else ""
|
||||||
|
concerns.append(
|
||||||
|
f"Element {element.id}: {element.resource_type} is not a "
|
||||||
|
f"resource type of the provider.{hint}"
|
||||||
|
)
|
||||||
|
for check in graph.checks:
|
||||||
|
resource_type = types.get(check.applies_to)
|
||||||
|
if resource_type not in schemas:
|
||||||
|
continue # a Check on a Relationship, or a type already raised
|
||||||
|
known = self._names.setdefault(
|
||||||
|
resource_type, _names(schemas[resource_type][1])
|
||||||
|
)
|
||||||
|
missing = [
|
||||||
|
setting
|
||||||
|
for setting in named_settings(check.statement)
|
||||||
|
# A resource type named in the statement is not a setting, and a
|
||||||
|
# path is judged by the setting at its end.
|
||||||
|
if setting not in schemas and setting.split(".")[-1] not in known
|
||||||
|
]
|
||||||
|
if missing:
|
||||||
|
concerns.append(
|
||||||
|
f"Check {check.id}: {resource_type} has no setting named "
|
||||||
|
f"{', '.join(missing)}, so no code can meet it there. Put the "
|
||||||
|
"Check on the Element whose resource type has that setting, "
|
||||||
|
"or reword it. Leave it if that word is a value, not a setting"
|
||||||
|
)
|
||||||
|
if concerns:
|
||||||
|
logger.info("[%s] schema review raised %d item(s)", scope, len(concerns))
|
||||||
|
return tuple(concerns)
|
||||||
|
|
||||||
|
|
||||||
|
class Reviews:
|
||||||
|
"""Several reviews as one: everything any of them raises."""
|
||||||
|
|
||||||
|
def __init__(self, reviews: Sequence[Any]) -> None:
|
||||||
|
self._reviews = tuple(reviews)
|
||||||
|
|
||||||
|
async def __call__(
|
||||||
|
self,
|
||||||
|
kind: Any,
|
||||||
|
attachments: Mapping[str, str],
|
||||||
|
upstream: Mapping[str, Mapping[str, Any]] | None,
|
||||||
|
scope: str = "",
|
||||||
|
) -> Sequence[str]:
|
||||||
|
raised: list[str] = []
|
||||||
|
for review in self._reviews:
|
||||||
|
try:
|
||||||
|
raised += await review(kind, attachments, upstream, scope)
|
||||||
|
except Exception: # one review failing must not lose the others
|
||||||
|
logger.exception("A submission review failed; continuing")
|
||||||
|
return tuple(raised)
|
||||||
@@ -8,12 +8,15 @@ last attempt instead of from nothing.
|
|||||||
|
|
||||||
It lives in the Deliverable's working memory:
|
It lives in the Deliverable's working memory:
|
||||||
|
|
||||||
working["files"] {path: content}
|
working["files"] {path: content}
|
||||||
working["graph"] the Target architecture, as target-architecture.json holds it
|
working["graph"] the Target architecture, as target-architecture.json holds it
|
||||||
|
working["verdicts"] the validation's verdicts, as verdicts.json holds them, with
|
||||||
|
a mark of the code they were given on
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
from collections.abc import Mapping, MutableMapping, Sequence
|
from collections.abc import Mapping, MutableMapping, Sequence
|
||||||
from pathlib import PurePosixPath
|
from pathlib import PurePosixPath
|
||||||
@@ -269,6 +272,150 @@ def graph_tools(working: MutableMapping[str, Any]) -> list[ToolSpec]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# --- the verdicts --------------------------------------------------------------------
|
||||||
|
|
||||||
|
VERDICTS = "verdicts"
|
||||||
|
|
||||||
|
|
||||||
|
def basis_of(files: Mapping[str, str]) -> str:
|
||||||
|
"""A mark of the files a validation judged, to tell when they have changed."""
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
for name in sorted(files):
|
||||||
|
digest.update(name.encode())
|
||||||
|
digest.update(b"\0")
|
||||||
|
digest.update(files[name].encode())
|
||||||
|
digest.update(b"\0")
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def working_verdicts(working: MutableMapping[str, Any], basis: str) -> dict[str, Any]:
|
||||||
|
"""The working verdicts for the code marked `basis`.
|
||||||
|
|
||||||
|
Verdicts given on other code are dropped: they are about work that has changed.
|
||||||
|
Verdicts on the same code are kept, so a run cut short carries on.
|
||||||
|
"""
|
||||||
|
verdicts = working.get(VERDICTS)
|
||||||
|
if not isinstance(verdicts, dict) or verdicts.get("basis") != basis:
|
||||||
|
verdicts = working[VERDICTS] = {"basis": basis}
|
||||||
|
if not isinstance(verdicts.get("checks"), list):
|
||||||
|
verdicts["checks"] = []
|
||||||
|
if not isinstance(verdicts.get("proposed_additions"), list):
|
||||||
|
verdicts["proposed_additions"] = []
|
||||||
|
return verdicts
|
||||||
|
|
||||||
|
|
||||||
|
def verdicts_text(working: Mapping[str, Any]) -> str:
|
||||||
|
"""The working verdicts as verdicts.json, or "" when none have been recorded."""
|
||||||
|
verdicts = working.get(VERDICTS)
|
||||||
|
if not isinstance(verdicts, Mapping) or not (
|
||||||
|
verdicts.get("checks") or verdicts.get("dry_run")
|
||||||
|
):
|
||||||
|
return ""
|
||||||
|
return json.dumps(
|
||||||
|
{
|
||||||
|
"dry_run": verdicts.get("dry_run") or {},
|
||||||
|
"checks": verdicts.get("checks") or [],
|
||||||
|
"proposed_additions": verdicts.get("proposed_additions") or [],
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def verdict_tools(
|
||||||
|
working: MutableMapping[str, Any], basis: str, check_ids: Sequence[str] = ()
|
||||||
|
) -> list[ToolSpec]:
|
||||||
|
"""Tools that record a validation a few verdicts at a time."""
|
||||||
|
expected = tuple(check_ids)
|
||||||
|
|
||||||
|
def remaining(verdicts: Mapping[str, Any]) -> str:
|
||||||
|
given = {item.get("check") for item in verdicts["checks"]}
|
||||||
|
left = [identifier for identifier in expected if identifier not in given]
|
||||||
|
if not expected:
|
||||||
|
return f"{len(given)} Checks have a verdict."
|
||||||
|
if not left:
|
||||||
|
return f"All {len(expected)} Checks have a verdict."
|
||||||
|
shown = ", ".join(left[:20]) + (" ..." if len(left) > 20 else "")
|
||||||
|
return f"{len(given)} of {len(expected)} Checks have a verdict. Left: {shown}."
|
||||||
|
|
||||||
|
async def verdict_put(
|
||||||
|
checks: Sequence[Any] = (),
|
||||||
|
dry_run: Any = None,
|
||||||
|
proposed_additions: Sequence[Any] = (),
|
||||||
|
) -> ToolResult:
|
||||||
|
verdicts = working_verdicts(working, basis)
|
||||||
|
bad = [
|
||||||
|
f"checks[{index}]"
|
||||||
|
for index, item in enumerate(checks or ())
|
||||||
|
if not isinstance(item, Mapping)
|
||||||
|
or not isinstance(item.get("check"), str)
|
||||||
|
or not item["check"].strip()
|
||||||
|
]
|
||||||
|
if bad:
|
||||||
|
return ToolResult(
|
||||||
|
f"Refused: each verdict needs a `check` id: {', '.join(bad)}"
|
||||||
|
)
|
||||||
|
for item in checks or ():
|
||||||
|
for index, existing in enumerate(verdicts["checks"]):
|
||||||
|
if existing.get("check") == item["check"]:
|
||||||
|
verdicts["checks"][index] = dict(item)
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
verdicts["checks"].append(dict(item))
|
||||||
|
if isinstance(dry_run, Mapping) and dry_run:
|
||||||
|
verdicts["dry_run"] = dict(dry_run)
|
||||||
|
for item in proposed_additions or ():
|
||||||
|
if (
|
||||||
|
isinstance(item, Mapping)
|
||||||
|
and dict(item) not in verdicts["proposed_additions"]
|
||||||
|
):
|
||||||
|
verdicts["proposed_additions"].append(dict(item))
|
||||||
|
return ToolResult(f"Recorded. {remaining(verdicts)}")
|
||||||
|
|
||||||
|
async def verdict_read() -> ToolResult:
|
||||||
|
verdicts = working_verdicts(working, basis)
|
||||||
|
return ToolResult(
|
||||||
|
(verdicts_text(working) or "No verdicts recorded yet.")
|
||||||
|
+ f"\n{remaining(verdicts)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
text = {"type": "string"}
|
||||||
|
verdict = _schema(
|
||||||
|
{"check": text, "verdict": text, "evidence": text},
|
||||||
|
["check", "verdict", "evidence"],
|
||||||
|
)
|
||||||
|
return [
|
||||||
|
ToolSpec(
|
||||||
|
"verdict_put",
|
||||||
|
"Record verdicts, a few Checks per call, replacing any already "
|
||||||
|
"recorded for the same Check. Also takes the dry-run's result and "
|
||||||
|
"proposed additions. What you record is attached as verdicts.json "
|
||||||
|
"when you submit, and is kept if your run is cut short.",
|
||||||
|
_schema(
|
||||||
|
{
|
||||||
|
"checks": {"type": "array", "items": verdict},
|
||||||
|
"dry_run": _schema({"verdict": text, "evidence": text}, []),
|
||||||
|
"proposed_additions": {
|
||||||
|
"type": "array",
|
||||||
|
"items": _schema(
|
||||||
|
{"statement": text, "applies_to": text, "reason": text},
|
||||||
|
["statement", "applies_to", "reason"],
|
||||||
|
),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
[],
|
||||||
|
),
|
||||||
|
verdict_put,
|
||||||
|
),
|
||||||
|
ToolSpec(
|
||||||
|
"verdict_read",
|
||||||
|
"Read the verdicts recorded so far and which Checks have none.",
|
||||||
|
{"type": "object", "properties": {}},
|
||||||
|
verdict_read,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def _counts(graph: Mapping[str, Sequence[Any]]) -> str:
|
def _counts(graph: Mapping[str, Sequence[Any]]) -> str:
|
||||||
return (
|
return (
|
||||||
f"The graph has {len(graph['elements'])} Elements, "
|
f"The graph has {len(graph['elements'])} Elements, "
|
||||||
@@ -286,4 +433,8 @@ def summary(working: Mapping[str, Any]) -> dict[str, Any] | None:
|
|||||||
graph = working.get(GRAPH)
|
graph = working.get(GRAPH)
|
||||||
if isinstance(graph, Mapping) and any(graph.get(name) for name in _GRAPH_LISTS):
|
if isinstance(graph, Mapping) and any(graph.get(name) for name in _GRAPH_LISTS):
|
||||||
told["graph"] = {name: len(graph.get(name) or ()) for name in _GRAPH_LISTS}
|
told["graph"] = {name: len(graph.get(name) or ()) for name in _GRAPH_LISTS}
|
||||||
|
verdicts = working.get(VERDICTS)
|
||||||
|
if isinstance(verdicts, Mapping) and verdicts.get("checks"):
|
||||||
|
# Only useful when the code is unchanged; the tools drop them otherwise.
|
||||||
|
told["verdicts_recorded"] = len(verdicts["checks"])
|
||||||
return told or None
|
return told or None
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ models:
|
|||||||
timeout_seconds:
|
timeout_seconds:
|
||||||
target-architect: 420
|
target-architect: 420
|
||||||
iac-author: 300
|
iac-author: 300
|
||||||
|
validator: 300 # timed out five times at 180 s in a run of 2026-10-07
|
||||||
packager: 300
|
packager: 300
|
||||||
|
|
||||||
# Skills each role may load. Role files stay provider-neutral; the bundle names Skills.
|
# Skills each role may load. Role files stay provider-neutral; the bundle names Skills.
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ The files are your working copy: the files you wrote with `write_file`, or, when
|
|||||||
|
|
||||||
## Reporting verdicts
|
## Reporting verdicts
|
||||||
|
|
||||||
Give a verdict on every Check in the Target architecture and attach them as `verdicts.json`, in the format the requirements-chain Skill gives. The dry-run's result goes in its `dry_run` entry: `pass` when `terraform_validate` passes, `fail` with the error when it does not, and `not_run` when the tool could not run.
|
Give a verdict on every Check in the Target architecture. Record them with `verdict_put`, a few Checks per call; they are attached as `verdicts.json`, in the format the requirements-chain Skill gives, when you submit. The dry-run's result goes in its `dry_run` entry: `pass` when `terraform_validate` passes, `fail` with the error when it does not, and `not_run` when the tool could not run.
|
||||||
|
|
||||||
A Check fails when the code could meet it and does not. When `terraform_schema` shows the resource type has no such setting, no code could meet it: the verdict is `not_verifiable`, with the schema's answer as evidence.
|
A Check fails when the code could meet it and does not. When `terraform_schema` shows the resource type has no such setting, no code could meet it: the verdict is `not_verifiable`, with the schema's answer as evidence.
|
||||||
|
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ def test_roles_with_long_replies_are_given_longer_for_a_model_request(bundle):
|
|||||||
roles = load_roles(AGENTS_DIR, bundle)
|
roles = load_roles(AGENTS_DIR, bundle)
|
||||||
|
|
||||||
assert roles["target-architect"].model_timeout_seconds == 420
|
assert roles["target-architect"].model_timeout_seconds == 420
|
||||||
|
assert roles["validator"].model_timeout_seconds == 300
|
||||||
assert roles["decomposer"].model_timeout_seconds is None
|
assert roles["decomposer"].model_timeout_seconds is None
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
279
tests/execution/test_validation_in_pieces.py
Normal file
279
tests/execution/test_validation_in_pieces.py
Normal file
@@ -0,0 +1,279 @@
|
|||||||
|
"""Two fixes from the run cancelled on 2026-10-07.
|
||||||
|
|
||||||
|
The validator wrote every verdict in one reply and timed out five times; it now
|
||||||
|
records them a few at a time. And two Checks named settings their resource types do
|
||||||
|
not have, each costing a Revision; the design is now checked against the provider's
|
||||||
|
schema, in code, before it is submitted.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
from app.adk.fake_runtime import SAMPLE_DATA, sample_submission
|
||||||
|
from app.adk.runtime import Submitted, ToolResult
|
||||||
|
from app.tools.execution import specialist_tools
|
||||||
|
from app.tools.local import TerraformRunner
|
||||||
|
from app.tools.schema_review import Reviews, SchemaReview, named_settings
|
||||||
|
from app.workflows.deliverables import DeliverableKind
|
||||||
|
|
||||||
|
K = DeliverableKind
|
||||||
|
CODE = {"terraform/main.tf": "# v1\n", "implementation.json": "[]"}
|
||||||
|
VERDICTS = json.loads(SAMPLE_DATA["verdicts.json"])
|
||||||
|
CONTENT = sample_submission(K.VALIDATION_RESULTS).args["content"]
|
||||||
|
|
||||||
|
|
||||||
|
def upstream(code=CODE) -> dict:
|
||||||
|
return {
|
||||||
|
"target_architecture": {
|
||||||
|
"attachments": {
|
||||||
|
"target-architecture.json": SAMPLE_DATA["target-architecture.json"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"deployment_guidance": {"attachments": dict(code)},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def validator_tools(working: dict, code=CODE) -> dict:
|
||||||
|
return {
|
||||||
|
tool.name: tool
|
||||||
|
for tool in specialist_tools(
|
||||||
|
K.VALIDATION_RESULTS,
|
||||||
|
(),
|
||||||
|
allow_cloud_read=False,
|
||||||
|
upstream=upstream(code),
|
||||||
|
working=working,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# --- verdicts a few at a time ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_verdicts_are_recorded_a_few_at_a_time_and_attached_on_submission():
|
||||||
|
working: dict = {}
|
||||||
|
tools = validator_tools(working)
|
||||||
|
|
||||||
|
first = await tools["verdict_put"].handler(
|
||||||
|
dry_run=VERDICTS["dry_run"], checks=VERDICTS["checks"][:1]
|
||||||
|
)
|
||||||
|
second = await tools["verdict_put"].handler(checks=VERDICTS["checks"][1:])
|
||||||
|
result = await tools["submit_deliverable"].handler(content=CONTENT)
|
||||||
|
|
||||||
|
assert first.message == "Recorded. 1 of 2 Checks have a verdict. Left: K-2."
|
||||||
|
assert second.message == "Recorded. All 2 Checks have a verdict."
|
||||||
|
assert isinstance(result.outcome, Submitted)
|
||||||
|
attached = json.loads(result.outcome.attachments["verdicts.json"])
|
||||||
|
assert [item["check"] for item in attached["checks"]] == ["K-1", "K-2"]
|
||||||
|
assert attached["dry_run"]["verdict"] == "pass"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_submission_with_a_check_left_out_names_it():
|
||||||
|
tools = validator_tools({})
|
||||||
|
await tools["verdict_put"].handler(
|
||||||
|
dry_run=VERDICTS["dry_run"], checks=VERDICTS["checks"][:1]
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await tools["submit_deliverable"].handler(content=CONTENT)
|
||||||
|
|
||||||
|
assert result.outcome is None
|
||||||
|
assert "Check K-2 is not accounted for" in result.message
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_verdict_given_again_replaces_the_first_and_additions_are_kept():
|
||||||
|
working: dict = {}
|
||||||
|
tools = validator_tools(working)
|
||||||
|
addition = {
|
||||||
|
"statement": "A timeout is set.",
|
||||||
|
"applies_to": "E-1",
|
||||||
|
"reason": "NFR-1",
|
||||||
|
}
|
||||||
|
await tools["verdict_put"].handler(checks=VERDICTS["checks"])
|
||||||
|
|
||||||
|
await tools["verdict_put"].handler(
|
||||||
|
checks=[{"check": "K-1", "verdict": "fail", "evidence": "min is 1"}],
|
||||||
|
proposed_additions=[addition, addition],
|
||||||
|
)
|
||||||
|
read = await tools["verdict_read"].handler()
|
||||||
|
|
||||||
|
recorded = json.loads(read.message.rsplit("\n", 1)[0])
|
||||||
|
assert [(c["check"], c["verdict"]) for c in recorded["checks"]] == [
|
||||||
|
("K-1", "fail"),
|
||||||
|
("K-2", "pass"),
|
||||||
|
]
|
||||||
|
assert recorded["proposed_additions"] == [addition]
|
||||||
|
assert read.message.endswith("All 2 Checks have a verdict.")
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_run_cut_short_carries_on_from_what_it_recorded():
|
||||||
|
working: dict = {}
|
||||||
|
await validator_tools(working)["verdict_put"].handler(checks=VERDICTS["checks"][:1])
|
||||||
|
|
||||||
|
# The same code, a later attempt: the working memory came back with the Execution.
|
||||||
|
read = await validator_tools(working)["verdict_read"].handler()
|
||||||
|
|
||||||
|
assert "1 of 2 Checks have a verdict. Left: K-2." in read.message
|
||||||
|
|
||||||
|
|
||||||
|
async def test_verdicts_on_code_that_has_changed_are_dropped():
|
||||||
|
working: dict = {}
|
||||||
|
await validator_tools(working)["verdict_put"].handler(checks=VERDICTS["checks"])
|
||||||
|
|
||||||
|
revised = {**CODE, "terraform/main.tf": "# v2\n"}
|
||||||
|
read = await validator_tools(working, revised)["verdict_read"].handler()
|
||||||
|
|
||||||
|
assert read.message.startswith("No verdicts recorded yet.")
|
||||||
|
|
||||||
|
|
||||||
|
async def test_verdicts_passed_as_an_attachment_still_work():
|
||||||
|
call = sample_submission(K.VALIDATION_RESULTS)
|
||||||
|
|
||||||
|
result = await validator_tools({})["submit_deliverable"].handler(**call.args)
|
||||||
|
|
||||||
|
assert isinstance(result.outcome, Submitted)
|
||||||
|
|
||||||
|
|
||||||
|
def test_only_the_validator_records_verdicts():
|
||||||
|
assert "verdict_put" in validator_tools({})
|
||||||
|
assert "verdict_put" not in {
|
||||||
|
tool.name
|
||||||
|
for tool in specialist_tools(K.DEPLOYMENT_GUIDANCE, (), allow_cloud_read=False)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# --- the design against the provider's schema --------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_settings_a_check_names_are_read_from_its_statement():
|
||||||
|
assert named_settings("`kms_key_name` is set on the topic") == ("kms_key_name",)
|
||||||
|
assert named_settings("template.scaling.min_instance_count is at least 2") == (
|
||||||
|
"template.scaling.min_instance_count",
|
||||||
|
)
|
||||||
|
assert named_settings("startup_cpu_boost is true") == ("startup_cpu_boost",)
|
||||||
|
# Values, roles and plain prose name no setting.
|
||||||
|
assert named_settings("ingress is INGRESS_TRAFFIC_ALL") == ()
|
||||||
|
assert named_settings("roles/run.invoker is granted to allUsers") == ()
|
||||||
|
assert named_settings('the filter contains "oldest_unacked_message_age"') == ()
|
||||||
|
assert named_settings("The service is private.") == ()
|
||||||
|
|
||||||
|
|
||||||
|
SCHEMA = {
|
||||||
|
"cloud_service": (
|
||||||
|
"example/cloud 1.2.3",
|
||||||
|
{
|
||||||
|
"attributes": {"name": {}, "ingress": {}},
|
||||||
|
"block_types": {
|
||||||
|
"template": {
|
||||||
|
"block": {
|
||||||
|
"attributes": {"service_account": {}},
|
||||||
|
"block_types": {
|
||||||
|
"scaling": {
|
||||||
|
"block": {"attributes": {"min_instance_count": {}}}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
),
|
||||||
|
"cloud_job": ("example/cloud 1.2.3", {"attributes": {"name": {}}}),
|
||||||
|
"cloud_service_iam_member": ("example/cloud 1.2.3", {"attributes": {"role": {}}}),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class Runner(TerraformRunner):
|
||||||
|
def __init__(self, schemas=SCHEMA) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self._given = schemas
|
||||||
|
|
||||||
|
async def _load_schemas(self):
|
||||||
|
return self._given
|
||||||
|
|
||||||
|
|
||||||
|
def graph(checks, resource_types=("cloud_service", "cloud_job")) -> dict[str, str]:
|
||||||
|
elements = [
|
||||||
|
{
|
||||||
|
"id": f"E-{n}",
|
||||||
|
"name": f"Thing {n}",
|
||||||
|
"product": "Example",
|
||||||
|
"resource_type": kind,
|
||||||
|
}
|
||||||
|
for n, kind in enumerate(resource_types, start=1)
|
||||||
|
]
|
||||||
|
data = {
|
||||||
|
"elements": elements,
|
||||||
|
"relationships": [{"id": "R-1", "from": "E-1", "to": "E-2", "kind": "calls"}],
|
||||||
|
"checks": [
|
||||||
|
{"id": f"K-{n}", "applies_to": on, "statement": text, "source": "NFR-1"}
|
||||||
|
for n, (on, text) in enumerate(checks, start=1)
|
||||||
|
],
|
||||||
|
}
|
||||||
|
return {"target-architecture.json": json.dumps(data)}
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_check_naming_a_setting_its_resource_type_lacks_is_raised():
|
||||||
|
design = graph(
|
||||||
|
[
|
||||||
|
("E-1", "template.scaling.min_instance_count is at least 2"),
|
||||||
|
("E-2", "`min_instance_count` is at least 2"),
|
||||||
|
("E-2", "the name is set"),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
raised = await SchemaReview(Runner())(K.TARGET_ARCHITECTURE, design, None)
|
||||||
|
|
||||||
|
assert len(raised) == 1
|
||||||
|
assert raised[0].startswith(
|
||||||
|
"Check K-2: cloud_job has no setting named min_instance_count"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_setting_found_anywhere_in_the_resource_type_is_accepted():
|
||||||
|
design = graph(
|
||||||
|
[
|
||||||
|
("E-1", "service_account is the dedicated identity"),
|
||||||
|
("E-1", "ingress is INTERNAL"),
|
||||||
|
("E-1", "a cloud_service_iam_member grants the invoker role"),
|
||||||
|
("R-1", "any_made_up_setting is checked on a Relationship"),
|
||||||
|
("E-2", "the name is set"),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
assert await SchemaReview(Runner())(K.TARGET_ARCHITECTURE, design, None) == ()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_resource_type_the_provider_lacks_is_raised_with_close_matches():
|
||||||
|
design = graph(
|
||||||
|
[("E-1", "the name is set"), ("E-2", "the name is set")],
|
||||||
|
resource_types=("cloud_servce", "cloud_job"),
|
||||||
|
)
|
||||||
|
|
||||||
|
raised = await SchemaReview(Runner())(K.TARGET_ARCHITECTURE, design, None)
|
||||||
|
|
||||||
|
assert len(raised) == 1
|
||||||
|
assert raised[0].startswith(
|
||||||
|
"Element E-1: cloud_servce is not a resource type of the provider."
|
||||||
|
)
|
||||||
|
assert "Close matches: cloud_service" in raised[0]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_no_schema_and_other_deliverables_raise_nothing():
|
||||||
|
design = graph([("E-1", "made_up_thing is set"), ("E-2", "the name is set")])
|
||||||
|
unavailable = Runner(ToolResult("Schema lookup is unavailable."))
|
||||||
|
|
||||||
|
assert await SchemaReview(unavailable)(K.TARGET_ARCHITECTURE, design, None) == ()
|
||||||
|
assert await SchemaReview(Runner())(K.DEPLOYMENT_GUIDANCE, design, None) == ()
|
||||||
|
assert await SchemaReview(Runner())(K.TARGET_ARCHITECTURE, {}, None) == ()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_several_reviews_raise_together_and_one_failing_loses_nothing():
|
||||||
|
async def one(kind, attachments, upstream, scope=""):
|
||||||
|
return ["first"]
|
||||||
|
|
||||||
|
async def broken(kind, attachments, upstream, scope=""):
|
||||||
|
raise RuntimeError("down")
|
||||||
|
|
||||||
|
async def two(kind, attachments, upstream, scope=""):
|
||||||
|
return ("second",)
|
||||||
|
|
||||||
|
raised = await Reviews([one, broken, two])(K.TARGET_ARCHITECTURE, {}, None)
|
||||||
|
|
||||||
|
assert raised == ("first", "second")
|
||||||
Reference in New Issue
Block a user