feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target architecture for gaps validation found one pass at a time): - A Deliverable stands while what it was written from is unchanged. The architecture description is written from the Elements and Relationships, so a Revision that only adds or rewords Checks no longer runs the describer. - Non-functional requirements and recommendations list their conditions, one entry per separate thing that must be true; submission requires them. - A coverage review at submission (JEV_COVERAGE_REVIEW, off by default): for each condition, Jev chooses which Check verifies it, or none. Asked as one broad yes-or-no per requirement it could not tell a complete design from one with known gaps; asked as a choice per condition it raised all four known gaps of case 02 and nothing on the complete graph. An opt-in live test (LIVE_JEV=1) repeats that. - The validator is told to report every gap in its first pass. Plans record the case 02 run and both forms of the review with figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -9,9 +9,9 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–20 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–22 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; case 01 passes with real models; cases 02–04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01 and 02 pass with real models; cases 03 and 04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||
|
||||
@@ -169,3 +169,20 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
||||
- **Rationale:** One `MODEL_TIMEOUT_SECONDS` (180) applied to every role. Raising it for all of them makes every stuck request slower to recover.
|
||||
- **Where:** `models.timeout_seconds` in `providers/gcp.yaml`, `ModelConfig` in `app/providers.py`, `RoleSpec.model_timeout_seconds`, `AdkAgentRuntime._content_config`.
|
||||
- **Acceptance criteria:** A role named in the bundle gets its own timeout; every other role gets the default.
|
||||
|
||||
## Step 21: A Deliverable stands while what it was written from is unchanged
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
|
||||
- **Capability:** A Revision upstream does not rewrite a Deliverable it leaves true.
|
||||
- **Rationale:** In case 02 the Target architecture was revised twice to add Checks. The architecture description, written from the Elements and Relationships, was rewritten both times: 24 requests and 942 thousand tokens, and each rewrite held the Phase open for about two minutes after the code was ready.
|
||||
- **Where:** `DeliverableSpec.written_from` in `app/workflows/deliverables.py`, `graph_structure` in `app/workflows/chain.py`, `specialist_task` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** When the Target architecture is revised with the same Elements and Relationships, the describer is not run and the description is submitted again as it stood. When either changes, or the description itself has feedback, it is written again.
|
||||
|
||||
## Step 22: Validation reports every gap in one pass
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-07 after case 02
|
||||
- **Capability:** One validation finds the gaps in the Target architecture, not one validation per gap.
|
||||
- **Rationale:** In case 02 the first validation proposed 1 addition and the second proposed 3 more, none caused by the first Revision. Each cost a Revision of the design, the code and the validation: 15 minutes and about 2.4 million tokens in all.
|
||||
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
|
||||
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
||||
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
||||
**This directory covers:** steps 1 to 20
|
||||
**This directory covers:** steps 1 to 22
|
||||
|
||||
## Sources of truth
|
||||
|
||||
@@ -37,6 +37,8 @@
|
||||
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
|
||||
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
||||
| 20 | A model timeout per role | added | S | built |
|
||||
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
||||
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
|
||||
14
.agents/plans/iter-04-learning-and-skills/steps/step-21.md
Normal file
14
.agents/plans/iter-04-learning-and-skills/steps/step-21.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 21 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07.
|
||||
|
||||
1. `DeliverableSpec.written_from` returns one comparable value for what a Deliverable is written from. The architecture description's is `graph_structure`: the graph's Elements and Relationships, without its Checks.
|
||||
2. When a Specialist submits, `specialist_task` keeps the submission in the Deliverable's working memory with that value and the feedback it was written under.
|
||||
3. When the Deliverable is started again and the value and the feedback are the same, no Specialist is run: the kept submission is returned as the attempt's outcome. The Ledger records a submission as for any attempt.
|
||||
4. A note that a dependency was revised is not counted as feedback on the Deliverable. Anything else is, and has it written again.
|
||||
|
||||
Only the architecture description declares `written_from`. The code answers the Checks, so it is not kept; its working copy already makes its Revision cheap (27 and 88 seconds in case 02).
|
||||
|
||||
**Unrun:** no evaluation since. The saving expected on a case like 02 is the describer's second and third runs.
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py`, "a description stands while the shape of the design does" (4).
|
||||
11
.agents/plans/iter-04-learning-and-skills/steps/step-22.md
Normal file
11
.agents/plans/iter-04-learning-and-skills/steps/step-22.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# Step 22 — Implementation Detail
|
||||
|
||||
Changed on 2026-10-07. Instructions only; no code.
|
||||
|
||||
- `agents/validator.md`: review coverage once and completely, going through every requirement and recommendation in turn, and report every gap in that pass. After a Revision made for its additions, propose only what is still uncovered, and never what a Check already verifies in other words.
|
||||
- `agents/target-architect.md`: one Check per requirement is rarely enough; write a Check for every condition a requirement states and every Element it applies to.
|
||||
- `skills/requirements-chain/SKILL.md`: a "Coverage" rule for the graph, and "find every gap in one pass" for the verdicts.
|
||||
|
||||
Also changed: `agents/requirements-analyst.md` and `agents/design-advisor.md` now have every non-functional requirement and recommendation list its `conditions`, which the submission check requires. A review of those conditions by Jev is built and off by default: see `../../iter-05-jev-decisions/steps/step-4.md`.
|
||||
|
||||
**Unrun:** an instruction's effect is only known from an evaluation. Case 02 is the comparison: 2 Revisions of the Target architecture, with 1 and then 3 proposed additions.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 5, Jev decisions
|
||||
**Intent:** Use Jev, a hosted decision model, for the agent's closed questions (yes or no, pick one, rate on a scale) so that large-model calls are kept for writing and reasoning. The aims are fewer tokens, faster turns, and checks the engine cannot make with string matching.
|
||||
**Status:** steps 2, 11 and 12 built (2026-10-07); the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**Status:** steps 2, 11 and 12 built (2026-10-07) and used live in case 02, where Jev judged two gaps; step 4 built in part and switched off; the rest planned. Step 1's answers are still open: the API key, rate limits and data retention. Build `../iter-06-target-architecture-graph/` first: it removes the need for step 5 here and changes what step 4 checks.
|
||||
**This directory covers:** steps 1 to 19
|
||||
|
||||
## What Jev is
|
||||
@@ -78,7 +78,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | a coverage review of the Target architecture built: one choice per listed condition; passes a live test on case 02's graph; off until a full case has run with it (`steps/step-4.md`); other checks planned |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
| 6 | Model tier per delegation | J5 | planned |
|
||||
| 7 | Context selection | J6 | planned |
|
||||
|
||||
50
.agents/plans/iter-05-jev-decisions/steps/step-4.md
Normal file
50
.agents/plans/iter-05-jev-decisions/steps/step-4.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Step 4 — Implementation Detail
|
||||
|
||||
Built on 2026-10-07 for one check only, and left switched off until a full case has run with it. The first form failed a live test; the second passed one. Both are recorded here.
|
||||
|
||||
What was built:
|
||||
|
||||
1. A seam: `SubmissionReview` in `app/adk/runtime.py`, passed through `ExecutionEngine`, `build_execution_graph` and `make_execution_nodes` to `specialist_tools`. After a submission passes its structural contract, the review may raise items; they are returned to the Specialist as a refusal once per run, and the next submission is accepted whatever the review would say. A review that fails or decides nothing never stops a submission.
|
||||
2. `CoverageReview` in `app/tools/coverage.py`, for the Target architecture. Its first form: one noul question per non-functional requirement and per design recommendation (not those marked not applicable), giving its text and the Checks that name it as source, with the Elements and Relationships as the state.
|
||||
3. `JEV_COVERAGE_REVIEW`, default `false`. The assembly installs the review only when it is `true` and a Jev key is set.
|
||||
|
||||
## Why the first form failed
|
||||
|
||||
Case 02 (`eval/results/20261007T190702Z/`) needed two Revisions because validation proposed additions for NFR-8, then NFR-10, DR-10 and NFR-1. The review was run against live Jev on that case's final graph, which validation passed with nothing proposed, and on a copy with the Checks those Revisions added removed:
|
||||
|
||||
| Question asked | On the complete graph | On the graph with 4 known gaps |
|
||||
|---|---|---|
|
||||
| "Does it state a condition none of these Checks verifies?" at 0.75 | raised 7 of 28 | raised 6; caught 1 of the 4 |
|
||||
| "Do these Checks verify everything it asks?" (probability of yes) | mean 0.33 over the other 24 | the 4 fell (0.10 to 0.04, 0.34 to 0.11, 0.16 to 0.14, 0.28 to 0.14); the other 24 stayed at 0.33 |
|
||||
| Score: none, part, all | mean 1.24 of 2 over the other 24 | the 4 fell by 0.06 to 0.55; the others unchanged |
|
||||
|
||||
Removing Checks moves the answer the right way and leaves unrelated items where they were, so Jev does read coverage. But it scores most of a design that passed validation as incomplete, so no threshold separates the two. Each call took 0.4 seconds and about 4,000 tokens.
|
||||
|
||||
## The form that works: one choice per condition
|
||||
|
||||
The same day, two of Jev's patterns were applied: a task-completion check against an explicit checklist, and a choice with "none" among the options in place of a yes or no.
|
||||
|
||||
1. Requirements and recommendations list their `conditions`: one entry for each separate thing that must be true. `check_requirements` and `check_recommendations` refuse a non-functional requirement or a recommendation without them. `conditions_of` reads an older item as one condition, its text.
|
||||
2. `CoverageReview` asks, per condition, "which one Check verifies this condition?" with that source's Check ids and `none` as the options. A condition answered `none` with a probability of at least `JEV_COVERAGE_THRESHOLD` (0.6) is raised, as `NFR-1: no Check verifies "<condition>" (its Checks: ...)`.
|
||||
|
||||
Measured on the same two graphs, with nine conditions written by hand (four the known gaps, five covered):
|
||||
|
||||
| | Complete graph | Graph with the 4 gaps |
|
||||
|---|---|---|
|
||||
| The 4 gap conditions | a Check chosen each time | `none` each time, probability 0.72 to 1.00 |
|
||||
| The 5 covered conditions | a Check chosen; `none` at most 0.40 | a Check chosen; `none` at most 0.45 |
|
||||
| Raised at 0.6 | 0 | 4 of 4, and nothing else |
|
||||
|
||||
Each call: about 0.25 seconds and 2,000 tokens. `tests/execution/test_coverage_live.py` repeats this through `CoverageReview` itself and is opt-in (`LIVE_JEV=1`); it passed on 2026-10-07.
|
||||
|
||||
Still off (`JEV_COVERAGE_REVIEW=false`), because three things are unproven:
|
||||
|
||||
- The conditions in that test were written by a person. In an Execution the requirements analyst and the design advisor write them, and the review is only as good as that list.
|
||||
- Nine conditions from one case is a small sample; one covered condition scored `none` at 0.45 against a threshold of 0.6.
|
||||
- A condition that applies to several Elements and has a Check on one of them is answered with that Check. NFR-8 was caught here only because the condition named the Element.
|
||||
|
||||
Turn it on for a full case before relying on it.
|
||||
|
||||
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py`, with a scripted decider.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 6, the Target architecture graph
|
||||
**Intent:** Establish the Target architecture as one explicit graph of Elements, Relationships and Checks before any infrastructure code is written, and derive and check every later Deliverable from it, so that the outcome is understood before code exists and validation has a fixed bar.
|
||||
**Status:** steps 1 to 10 built and covered by the deterministic suite; case 01 completed at 100% with no Revision on 2026-10-07 (`steps/step-11.md`); cases 02 to 04 have not run on this build
|
||||
**Status:** steps 1 to 10 built and covered by the deterministic suite; cases 01 and 02 completed at 100% on 2026-10-07 (`steps/step-11.md`), case 02 after two Revisions of the Target architecture; cases 03 and 04 have not run on this build
|
||||
**This directory covers:** steps 1 to 11
|
||||
|
||||
## Order of work
|
||||
@@ -56,7 +56,7 @@ flowchart TD
|
||||
| 8 | Deployment guidance implements the graph | M | complete |
|
||||
| 9 | Validation gives a verdict per Check | M | complete |
|
||||
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
||||
| 11 | Evaluation: prove the loop converges | M | case 01 passes (100%, no Revision); cases 02 to 04 and the thresholds outstanding |
|
||||
| 11 | Evaluation: prove the loop converges | M | cases 01 and 02 pass at 100%; cases 03 and 04 and the thresholds outstanding |
|
||||
|
||||
Size: S under a day, M a few days.
|
||||
|
||||
|
||||
@@ -79,3 +79,17 @@ What went wrong: the target architect exhausted its 30 tool calls on its first a
|
||||
|
||||
Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.
|
||||
|
||||
## Run of 2026-10-07, 19:07 UTC (`eval/results/20261007T190702Z/`)
|
||||
|
||||
Case 02, on commit `f1f6f1b`. Completed, 100% (11 of 11 checks), no warnings: 1,781 seconds and 4.78 million tokens; 904 seconds to the first validation.
|
||||
|
||||
- The graph ended with 20 Elements, 23 Relationships and 54 Checks. The code passed every Check in all three validations; no Check failed, was not verifiable or unimplementable.
|
||||
- The Target architecture was revised twice, both for proposed additions: 1 after the first validation, 3 after the second, 0 after the third. The set of gaps did not shrink between the first two passes, so the loop converged without the criterion "never grows" holding.
|
||||
- The router asked Jev whether each gap was material: 0.92 and 0.88, both above the 0.75 threshold. This is the first use of Jev inside an Execution.
|
||||
- The target architect made 17 schema lookups in its first attempt (39 in case 01) and did not exhaust its budget.
|
||||
- By role: IaC author 56 requests and 1.32 million tokens; target architect 57 and 1.20 million; describer 24 and 942 thousand, rewriting the description on each Revision.
|
||||
|
||||
Changed after the run, not yet run: iteration 4 steps 21 and 22.
|
||||
|
||||
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.
|
||||
|
||||
|
||||
@@ -58,6 +58,11 @@ JEV_BUDGET_TOKENS=1000000
|
||||
JEV_ACT_THRESHOLD=0.75
|
||||
JEV_CLOSE_THRESHOLD=0.15
|
||||
JEV_CONFIDENCE_THRESHOLD=0.6
|
||||
# Review a design's coverage with Jev before it is submitted: for each condition a
|
||||
# requirement lists, which Check verifies it. Off until a full case has run with it.
|
||||
JEV_COVERAGE_REVIEW=false
|
||||
# How likely "no Check verifies this" must be before the condition is raised.
|
||||
JEV_COVERAGE_THRESHOLD=0.6
|
||||
|
||||
# PostgreSQL (Execution records, and checkpoints when CHECKPOINTER=postgres)
|
||||
POSTGRES_HOST=localhost
|
||||
|
||||
@@ -10,6 +10,6 @@ You produce the **design recommendations** Deliverable: the Skill's task for rec
|
||||
- When `feedback` is not empty you are revising: pass `pillars` naming only the pillars the feedback touches. The guidance you were given before is returned for the others.
|
||||
- Reconcile the pillars' guidance into recommendations for each component. Where pillars conflict, name the trade-off and recommend a resolution tied to the requirements.
|
||||
- When an advisor's guidance is missing, derive that pillar's recommendations from the best-practices references in the solution-architecture Skill.
|
||||
- Give every recommendation an id and attach them as `recommendations.json`, in the format the requirements-chain Skill gives. Write one configurable decision per recommendation, so each can become a Check in the Target architecture.
|
||||
- Give every recommendation an id and attach them as `recommendations.json`, in the format the requirements-chain Skill gives. Write one configurable decision per recommendation, so each can become a Check in the Target architecture. Give each its `conditions`: one entry for each separate thing it asks to be configured, and for each component it must hold on when it names several.
|
||||
|
||||
Your contract's sections are one per pillar.
|
||||
|
||||
@@ -13,5 +13,6 @@ You produce the **requirements** Deliverable: the Skill's requirements-discovery
|
||||
- When the user runs workloads in a cloud environment today, use your cloud discovery tools to describe the current state from what actually exists, and record anything you couldn't read as a discovery gap.
|
||||
- Name no products, services or architectures. This Deliverable describes the problem, not the solution.
|
||||
- Give every requirement an id and attach them all as `requirements.json`, in the format the requirements-chain Skill gives. Use the same ids in your content.
|
||||
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
|
||||
|
||||
Add `Open questions` and `Assumptions` headings when they apply.
|
||||
|
||||
@@ -11,7 +11,7 @@ You produce the **Target architecture** Deliverable: the solution as one explici
|
||||
- Build the graph with `graph_put`, a few items per call: the Elements first, then the Relationships, then the Checks one Element at a time. Short calls are what keep your replies from timing out.
|
||||
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
|
||||
- Give every connection between Elements a Relationship.
|
||||
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have.
|
||||
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have. One Check per requirement is rarely enough: write a Check for every entry in a requirement's or recommendation's `conditions`, on every Element that condition applies to. A condition left without a Check is found by validation and sent back to you.
|
||||
- Name only settings the Element's resource type has. When `terraform_schema` is available, call it once for each distinct resource type in the graph, with `resource_type` alone: the reply lists every argument and block that type accepts, and you write that type's Checks from it. Never look settings up one at a time. A setting that one resource type has may not exist on a similar one, and a Check on a setting the resource type lacks cannot be met and is sent back to you.
|
||||
- Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job.
|
||||
- When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design.
|
||||
|
||||
@@ -9,7 +9,8 @@ You produce the **validation results** Deliverable: the Skill's pre-deployment v
|
||||
- Run the deployment dry-run with your tools: validate the Terraform, and preview it with a plan when you have permission to read live cloud resources. The tools already hold the submitted files, so pass them no files.
|
||||
- Give a verdict on every Check in the Target architecture, from the infrastructure code: pass, fail or not verifiable, each with the evidence you read. The Checks are the whole bar: judge the code against them and nothing else.
|
||||
- Before you fail a Check because the code lacks a setting, confirm with `terraform_schema` that the resource type has that setting, when the tool is available. When it does not, the Check cannot be met by any code: give it not verifiable, with what the schema shows as the evidence. Give the same verdict to a Check that `implementation.json` records as unimplementable.
|
||||
- When you see something the Target architecture lacks, record it as a proposed addition, naming the requirement or recommendation it serves. It is not a failure of the code.
|
||||
- Review the Target architecture's coverage once, completely. Go through every requirement and every design recommendation in turn; for each, list the conditions its text states and the Elements it applies to, and record a proposed addition for each one no Check verifies, naming the requirement or recommendation it serves. Report every gap in this pass: each later pass that finds one more costs a whole Revision of the design, the code and this validation. A proposed addition is not a failure of the code.
|
||||
- When `feedback` or your earlier verdicts show the Target architecture was revised for your proposed additions, propose only what the revision still leaves uncovered. Never propose what a Check already verifies in other words.
|
||||
- Attach the verdicts as `verdicts.json`, in the format the requirements-chain Skill gives.
|
||||
- Give the exact commands for every check you couldn't run, for the user to run themselves.
|
||||
|
||||
|
||||
@@ -40,7 +40,12 @@ SAMPLE_DATA: dict[str, str] = {
|
||||
"requirements.json": json.dumps(
|
||||
[
|
||||
{"id": "FR-1", "kind": "functional", "text": "Accept events."},
|
||||
{"id": "NFR-1", "kind": "non_functional", "text": "99.9% availability."},
|
||||
{
|
||||
"id": "NFR-1",
|
||||
"kind": "non_functional",
|
||||
"text": "99.9% availability.",
|
||||
"conditions": ["The service stays available when one instance fails."],
|
||||
},
|
||||
]
|
||||
),
|
||||
"components.json": json.dumps(
|
||||
@@ -54,6 +59,7 @@ SAMPLE_DATA: dict[str, str] = {
|
||||
"pillar": "security",
|
||||
"text": "Use a dedicated identity.",
|
||||
"components": ["C-1"],
|
||||
"conditions": ["The service runs as an identity nothing else uses."],
|
||||
}
|
||||
]
|
||||
),
|
||||
|
||||
@@ -255,6 +255,23 @@ class NoDecider:
|
||||
return None
|
||||
|
||||
|
||||
class SubmissionReview(Protocol):
|
||||
"""A review of a submission's meaning, beyond what its structure shows.
|
||||
|
||||
Returns what it would have the Specialist look at again, one line each; empty
|
||||
when it has nothing to raise or could not decide. `kind` is the Deliverable's
|
||||
kind, `upstream` the Deliverables it builds on.
|
||||
"""
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
kind: Any,
|
||||
attachments: Mapping[str, str],
|
||||
upstream: Mapping[str, Mapping[str, Any]] | None,
|
||||
scope: str = "",
|
||||
) -> Sequence[str]: ...
|
||||
|
||||
|
||||
class AgentRuntime(Protocol):
|
||||
async def run(
|
||||
self, role: RoleSpec, task: AgentTask, tools: Sequence[ToolSpec]
|
||||
|
||||
11
app/agent.py
11
app/agent.py
@@ -29,6 +29,7 @@ from app.adk.runtime import (
|
||||
NoDecider,
|
||||
RoleSpec,
|
||||
RunLimiter,
|
||||
SubmissionReview,
|
||||
ToolFactory,
|
||||
)
|
||||
from app.checkpointing import open_checkpointer
|
||||
@@ -46,6 +47,7 @@ from app.recorder import DatabaseRecorder, ExecutionRecorder
|
||||
from app.skills.agent_skills import skill_version
|
||||
from app.states.execution_state import check_customer_case
|
||||
from app.states.ledger import Ledger
|
||||
from app.tools.coverage import CoverageReview
|
||||
from app.tools.execution import ORCHESTRATOR
|
||||
from app.tools.jev import JevDecider, JevLimits
|
||||
from app.tools.local import TerraformRunner, local_tools
|
||||
@@ -84,6 +86,7 @@ class ExecutionEngine:
|
||||
limiter: RunLimiter | None = None,
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
submission_review: SubmissionReview | None = None,
|
||||
) -> None:
|
||||
required = {
|
||||
ORCHESTRATOR,
|
||||
@@ -104,6 +107,7 @@ class ExecutionEngine:
|
||||
limiter=limiter,
|
||||
skill_versions=skill_versions,
|
||||
orchestrator=orchestrator,
|
||||
submission_review=submission_review,
|
||||
).compile(checkpointer=checkpointer)
|
||||
|
||||
async def start(
|
||||
@@ -334,6 +338,13 @@ async def open_assembly(
|
||||
name: skill_version(meta.path) for name, meta in bundle.skills.items()
|
||||
},
|
||||
orchestrator=orchestrator,
|
||||
# Optional, and off by default: Jev reviews a design's coverage before
|
||||
# it is submitted. Without it a submission stands on its structure.
|
||||
submission_review=CoverageReview(
|
||||
decider, none_at=settings.JEV_COVERAGE_THRESHOLD
|
||||
)
|
||||
if settings.JEV_COVERAGE_REVIEW and isinstance(decider, JevDecider)
|
||||
else None,
|
||||
)
|
||||
try:
|
||||
yield Assembly(engine, bundle, roles, dict(catalog.gaps), decider)
|
||||
|
||||
@@ -82,6 +82,11 @@ class Settings(BaseSettings):
|
||||
JEV_ACT_THRESHOLD: float = 0.75
|
||||
JEV_CLOSE_THRESHOLD: float = 0.15
|
||||
JEV_CONFIDENCE_THRESHOLD: float = 0.6
|
||||
# Review a design's coverage at submission: for each condition a requirement
|
||||
# lists, which Check verifies it. Off until a full case has run with it.
|
||||
JEV_COVERAGE_REVIEW: bool = False
|
||||
# How likely "no Check verifies this" must be before the condition is raised.
|
||||
JEV_COVERAGE_THRESHOLD: float = 0.6
|
||||
|
||||
# Paths
|
||||
BASE_DIR: Path = Path(__file__).resolve().parent.parent
|
||||
|
||||
@@ -28,6 +28,7 @@ from app.adk.runtime import (
|
||||
RoleSpec,
|
||||
RunLimiter,
|
||||
Stopped,
|
||||
SubmissionReview,
|
||||
Submitted,
|
||||
ToolFactory,
|
||||
ToolSpec,
|
||||
@@ -61,6 +62,9 @@ from app.workflows.deliverables import (
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Working-memory key: the last submission of a Deliverable with what it was written from.
|
||||
STANDING = "standing"
|
||||
|
||||
NO_USER_REPLY = (
|
||||
"No user is available. State your assumption, record it in the Deliverable under "
|
||||
"'Assumptions', and proceed."
|
||||
@@ -98,6 +102,7 @@ def make_execution_nodes(
|
||||
limiter: RunLimiter | None = None,
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
submission_review: SubmissionReview | None = None,
|
||||
) -> ExecutionNodes:
|
||||
tool_factory = tool_factory or _no_tools
|
||||
# How an Orchestrator turn is decided. The default drives the tools from the
|
||||
@@ -357,6 +362,48 @@ def make_execution_nodes(
|
||||
working[workspace.FILES] = dict(
|
||||
upstream.get(source.value, {}).get("attachments", {})
|
||||
)
|
||||
spec = DEPENDENCY_GRAPH[kind]
|
||||
# Feedback on this Deliverable itself. A note that a dependency was revised
|
||||
# is not a reason to write it again; the basis says whether that matters.
|
||||
feedback = [
|
||||
item
|
||||
for item in task_input["context"].get("feedback") or []
|
||||
if not str(item).startswith("Dependency ")
|
||||
]
|
||||
basis = spec.written_from(upstream) if spec.written_from else None
|
||||
standing = working.get(STANDING)
|
||||
if (
|
||||
basis
|
||||
and isinstance(standing, Mapping)
|
||||
and standing.get("basis") == basis
|
||||
and standing.get("feedback") == feedback
|
||||
):
|
||||
# What it was written from has not changed, so the last submission
|
||||
# still stands: no Specialist is run.
|
||||
logger.info(
|
||||
"[%s] %s kept %s (attempt %d): written from an unchanged basis",
|
||||
task_input["execution_id"],
|
||||
role.name,
|
||||
kind,
|
||||
task_input["attempt"],
|
||||
)
|
||||
kept = Submitted(standing["content"], dict(standing["attachments"]))
|
||||
return {
|
||||
"outcomes": [
|
||||
{
|
||||
**_outcome_record(role.name, kind, task_input["attempt"], kept),
|
||||
"refusals": [],
|
||||
}
|
||||
],
|
||||
"working": {kind.value: working},
|
||||
}
|
||||
|
||||
async def review(attachments: Mapping[str, str]) -> Sequence[str]:
|
||||
assert submission_review is not None
|
||||
return await submission_review(
|
||||
kind, attachments, upstream, task_input["execution_id"]
|
||||
)
|
||||
|
||||
refusals: list[dict[str, Any]] = []
|
||||
tools = _noting_refusals(
|
||||
specialist_tools(
|
||||
@@ -366,6 +413,7 @@ def make_execution_nodes(
|
||||
upstream=upstream,
|
||||
working=working,
|
||||
library=task_input.get("library"),
|
||||
review=review if submission_review else None,
|
||||
),
|
||||
refusals,
|
||||
role.name,
|
||||
@@ -402,6 +450,13 @@ def make_execution_nodes(
|
||||
time.monotonic() - started,
|
||||
_summarise_outcome(outcome),
|
||||
)
|
||||
if basis and isinstance(outcome, Submitted):
|
||||
working[STANDING] = {
|
||||
"basis": basis,
|
||||
"feedback": feedback,
|
||||
"content": outcome.content,
|
||||
"attachments": dict(outcome.attachments),
|
||||
}
|
||||
return {
|
||||
"outcomes": [
|
||||
{
|
||||
|
||||
170
app/tools/coverage.py
Normal file
170
app/tools/coverage.py
Normal file
@@ -0,0 +1,170 @@
|
||||
"""A review of the Target architecture's coverage, before it is submitted.
|
||||
|
||||
The structural check knows that every non-functional requirement and every design
|
||||
recommendation is the source of a Check. It cannot know whether those Checks verify
|
||||
everything the text asks for: a requirement with three conditions and a Check for one
|
||||
passes it. The validator finds the rest later, a few at a time, and each finding
|
||||
costs a Revision of the design, the code and the validation.
|
||||
|
||||
Requirements and recommendations list their conditions one by one. For each
|
||||
condition this asks the decider (ADR-0007) which of the source's Checks verifies it,
|
||||
with "none" as an answer. A condition answered "none" is raised.
|
||||
|
||||
Two things make the question answerable, found by measurement on 2026-10-07 (see
|
||||
.agents/plans/iter-05-jev-decisions/steps/step-4.md). It is about one explicit
|
||||
condition, not a whole requirement; and it is a choice among the Checks, not a yes or
|
||||
no. Asked the broad way, a complete design and an incomplete one scored alike.
|
||||
|
||||
It never blocks: with no decision, the submission stands on its structure.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import Decider, Question
|
||||
from app.workflows.chain import (
|
||||
RECOMMENDATIONS_FILE,
|
||||
REQUIREMENTS_FILE,
|
||||
Upstream,
|
||||
conditions_of,
|
||||
read_list,
|
||||
)
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
from app.workflows.target_architecture import (
|
||||
GRAPH_ATTACHMENT,
|
||||
Check,
|
||||
GraphError,
|
||||
TargetArchitecture,
|
||||
parse_graph,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
NONE = "none"
|
||||
|
||||
|
||||
def _described(graph: TargetArchitecture) -> str:
|
||||
lines = ["Elements of the Target architecture:"]
|
||||
lines += [
|
||||
f"- {e.id} {e.name} ({e.product}, {e.resource_type})"
|
||||
+ (f", realises component {e.component}" if e.component else "")
|
||||
for e in graph.elements
|
||||
]
|
||||
lines.append("Relationships:")
|
||||
lines += [f"- {r.id}: {r.source} {r.kind} {r.target}" for r in graph.relationships]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _constraints(upstream: Upstream, graph: TargetArchitecture) -> list[dict[str, Any]]:
|
||||
"""What must be covered: every non-functional requirement, and every design
|
||||
recommendation that applies."""
|
||||
excused = {item.recommendation for item in graph.not_applicable}
|
||||
return [
|
||||
item
|
||||
for item in read_list(upstream, "requirements", REQUIREMENTS_FILE)
|
||||
if item.get("kind") == "non_functional"
|
||||
] + [
|
||||
item
|
||||
for item in read_list(upstream, "design_recommendations", RECOMMENDATIONS_FILE)
|
||||
if item.get("id") not in excused
|
||||
]
|
||||
|
||||
|
||||
def coverage_questions(
|
||||
graph: TargetArchitecture, constraints: Sequence[Mapping[str, Any]]
|
||||
) -> dict[str, tuple[str, str, Question]]:
|
||||
"""One question per condition: (source id, the condition, the question).
|
||||
|
||||
A source with no Check is the structural check's to report, so it is not asked
|
||||
about here.
|
||||
"""
|
||||
names = {element.id: element.name for element in graph.elements}
|
||||
names.update(
|
||||
{
|
||||
r.id: f"{names[r.source]} {r.kind} {names[r.target]}"
|
||||
for r in graph.relationships
|
||||
}
|
||||
)
|
||||
by_source: dict[str, list[Check]] = {}
|
||||
for check in graph.checks:
|
||||
by_source.setdefault(check.source, []).append(check)
|
||||
|
||||
questions: dict[str, tuple[str, str, Question]] = {}
|
||||
for item in constraints:
|
||||
source = item.get("id")
|
||||
checks = by_source.get(source, []) if isinstance(source, str) else []
|
||||
if not checks:
|
||||
continue
|
||||
listed = "\n".join(
|
||||
f"- {check.id} on {names[check.applies_to]}: {check.statement}"
|
||||
for check in checks
|
||||
)
|
||||
for number, condition in enumerate(conditions_of(item), start=1):
|
||||
questions[f"{source}#{number}"] = (
|
||||
str(source),
|
||||
condition,
|
||||
Question(
|
||||
"choice",
|
||||
f'Condition: "{condition}".\nChecks:\n{listed}\n'
|
||||
"Which one Check verifies this condition? Answer none when "
|
||||
"no Check does.",
|
||||
(*(check.id for check in checks), NONE),
|
||||
),
|
||||
)
|
||||
return questions
|
||||
|
||||
|
||||
class CoverageReview:
|
||||
def __init__(self, decider: Decider, *, none_at: float = 0.6) -> None:
|
||||
self._decider = decider
|
||||
# How likely "none" must be before a condition is raised. Below it the
|
||||
# answer is too uncertain to send work back on.
|
||||
self._none_at = none_at
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
kind: DeliverableKind,
|
||||
attachments: Mapping[str, str],
|
||||
upstream: Upstream | None,
|
||||
scope: str = "",
|
||||
) -> Sequence[str]:
|
||||
"""The conditions no Check appears to verify, each as a line for the
|
||||
Specialist. Empty when coverage looks complete or was not judged."""
|
||||
if kind is not DeliverableKind.TARGET_ARCHITECTURE or not upstream:
|
||||
return ()
|
||||
try:
|
||||
graph = parse_graph(attachments.get(GRAPH_ATTACHMENT, ""))
|
||||
except GraphError:
|
||||
return ()
|
||||
asked = coverage_questions(graph, _constraints(upstream, graph))
|
||||
if not asked:
|
||||
return ()
|
||||
answers = await self._decider.decide(
|
||||
_described(graph),
|
||||
{key: question for key, (_, _, question) in asked.items()},
|
||||
scope=scope,
|
||||
)
|
||||
if not answers:
|
||||
return ()
|
||||
concerns = []
|
||||
for key, (source, condition, question) in asked.items():
|
||||
answer = answers.get(key)
|
||||
if answer is None or answer.value != NONE:
|
||||
continue
|
||||
if answer.probabilities.get(NONE, answer.confidence) < self._none_at:
|
||||
continue
|
||||
checks = ", ".join(question.options[:-1])
|
||||
concerns.append(
|
||||
f'{source}: no Check verifies "{condition}" (its Checks: {checks})'
|
||||
)
|
||||
if concerns:
|
||||
logger.info(
|
||||
"[%s] coverage review raised %d of %d conditions",
|
||||
scope,
|
||||
len(concerns),
|
||||
len(asked),
|
||||
)
|
||||
return tuple(concerns)
|
||||
@@ -7,7 +7,8 @@ outcome.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, MutableMapping, Sequence
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable, Mapping, MutableMapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import (
|
||||
@@ -46,6 +47,7 @@ from app.workflows.target_architecture import (
|
||||
)
|
||||
|
||||
ORCHESTRATOR = "orchestrator"
|
||||
logger = logging.getLogger(__name__)
|
||||
_TARGET = DeliverableKind.TARGET_ARCHITECTURE
|
||||
CLOUD_READ = "cloud_read"
|
||||
|
||||
@@ -259,6 +261,7 @@ def specialist_tools(
|
||||
upstream: Mapping[str, Mapping[str, Any]] | None = None,
|
||||
working: MutableMapping[str, Any] | None = None,
|
||||
library: Mapping[str, Mapping[str, Any]] | None = None,
|
||||
review: Callable[[Mapping[str, str]], Awaitable[Sequence[str]]] | None = None,
|
||||
) -> list[ToolSpec]:
|
||||
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
|
||||
the cloud-read gate.
|
||||
@@ -267,10 +270,13 @@ def specialist_tools(
|
||||
along the Requirements chain are checked. `working` is the Deliverable's working
|
||||
memory, where its working copy is kept. `library` is the earlier Deliverables
|
||||
given to the Specialist as data files alone, to read whole when it needs one.
|
||||
`review` looks at the meaning of a structurally complete submission; what it
|
||||
raises is returned to the Specialist once per run, and never a second time.
|
||||
"""
|
||||
working = {} if working is None else working
|
||||
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
||||
own_files = DEPENDENCY_GRAPH[kind].working_copy is kind
|
||||
reviewed = False
|
||||
|
||||
async def submit_deliverable(content: str, attachments: Any = None) -> ToolResult:
|
||||
try:
|
||||
@@ -295,11 +301,28 @@ def specialist_tools(
|
||||
)
|
||||
if own_files:
|
||||
working_files(working).update(attachments)
|
||||
if is_graph:
|
||||
keep_graph(working, attachments[GRAPH_ATTACHMENT])
|
||||
nonlocal reviewed
|
||||
if review and not reviewed:
|
||||
reviewed = True
|
||||
try:
|
||||
concerns = await review(attachments)
|
||||
except Exception: # a review must never stop a submission
|
||||
logger.exception("The submission review failed; accepting")
|
||||
concerns = ()
|
||||
if concerns:
|
||||
return ToolResult(
|
||||
"Not submitted. A review found: "
|
||||
+ "; ".join(concerns)
|
||||
+ ". Look at each one: add what is missing, or leave it when "
|
||||
"it is already covered. This review runs once, so your next "
|
||||
"submission is accepted."
|
||||
)
|
||||
if is_graph:
|
||||
# The tables and the diagram are views of the graph, so the engine
|
||||
# writes them: they cannot disagree with it.
|
||||
graph = parse_graph(attachments[GRAPH_ATTACHMENT])
|
||||
keep_graph(working, attachments[GRAPH_ATTACHMENT])
|
||||
content = content.rstrip() + "\n\n" + render_tables(graph)
|
||||
attachments[DIAGRAM_ATTACHMENT] = render_mermaid(graph)
|
||||
return ToolResult(f"Submitted {kind}.", Submitted(content, attachments))
|
||||
|
||||
@@ -67,6 +67,29 @@ def read_graph(upstream: Upstream | None) -> TargetArchitecture | None:
|
||||
return None
|
||||
|
||||
|
||||
def graph_structure(upstream: Upstream | None) -> str | None:
|
||||
"""The Target architecture's Elements and Relationships, as one comparable value.
|
||||
|
||||
What a Deliverable written from the shape of the design, and not from its
|
||||
Checks, was written from. None when there is no readable graph.
|
||||
"""
|
||||
graph = read_graph(upstream)
|
||||
if graph is None:
|
||||
return None
|
||||
return json.dumps(
|
||||
{
|
||||
"elements": [
|
||||
[e.id, e.name, e.product, e.resource_type, e.component]
|
||||
for e in graph.elements
|
||||
],
|
||||
"relationships": [
|
||||
[r.id, r.source, r.target, r.kind] for r in graph.relationships
|
||||
],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
|
||||
|
||||
def _objects(
|
||||
text: str, name: str, strings: Sequence[str], lists: Sequence[str] = ()
|
||||
) -> tuple[list[dict[str, Any]], list[str]]:
|
||||
@@ -113,6 +136,40 @@ def _repeated(
|
||||
return problems
|
||||
|
||||
|
||||
def conditions_of(item: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
"""The separate things a requirement or recommendation asks for.
|
||||
|
||||
Each is one statement that a Check can verify on its own. An item written before
|
||||
conditions were listed is read as one condition: its whole text.
|
||||
"""
|
||||
listed = item.get("conditions")
|
||||
if isinstance(listed, list):
|
||||
conditions = tuple(
|
||||
c.strip() for c in listed if isinstance(c, str) and c.strip()
|
||||
)
|
||||
if conditions:
|
||||
return conditions
|
||||
text = item.get("text")
|
||||
return (text.strip(),) if isinstance(text, str) and text.strip() else ()
|
||||
|
||||
|
||||
def _lacking_conditions(items: Sequence[Mapping[str, Any]], name: str) -> list[str]:
|
||||
"""A constraint must say, one by one, what would have to be true for it to be met."""
|
||||
problems = []
|
||||
for item in items:
|
||||
listed = item.get("conditions")
|
||||
if (
|
||||
not isinstance(listed, list)
|
||||
or not listed
|
||||
or not all(isinstance(c, str) and c.strip() for c in listed)
|
||||
):
|
||||
problems.append(
|
||||
f"{name}: {item['id']} needs `conditions`: a list with one entry "
|
||||
"for each separate thing it asks for"
|
||||
)
|
||||
return problems
|
||||
|
||||
|
||||
def check_requirements(
|
||||
attachments: Mapping[str, str], upstream: Upstream | None
|
||||
) -> list[str]:
|
||||
@@ -126,6 +183,9 @@ def check_requirements(
|
||||
]
|
||||
if not items and not problems:
|
||||
problems.append(f"{name}: lists no requirements")
|
||||
problems += _lacking_conditions(
|
||||
[item for item in items if item["kind"] == "non_functional"], name
|
||||
)
|
||||
return problems
|
||||
|
||||
|
||||
@@ -189,6 +249,7 @@ def check_recommendations(
|
||||
problems += _repeated(items, name)
|
||||
if not items and not problems:
|
||||
problems.append(f"{name}: lists no recommendations")
|
||||
problems += _lacking_conditions(items, name)
|
||||
components = read_list(upstream, "technical_decomposition", COMPONENTS_FILE)
|
||||
if components:
|
||||
known = {item["id"] for item in components}
|
||||
|
||||
@@ -28,6 +28,7 @@ from app.workflows.chain import (
|
||||
check_requirements,
|
||||
check_target_architecture,
|
||||
check_verdicts,
|
||||
graph_structure,
|
||||
)
|
||||
from app.workflows.target_architecture import GRAPH_ATTACHMENT
|
||||
|
||||
@@ -72,6 +73,10 @@ class DeliverableSpec:
|
||||
# files are its working copy, kept across attempts and attached on submission.
|
||||
# Another kind: a fresh copy of that Deliverable's files on every attempt.
|
||||
working_copy: DeliverableKind | None = None
|
||||
# What the Deliverable is written from, as one comparable value. When a Revision
|
||||
# upstream leaves that value as it was, the last submission still stands and the
|
||||
# Specialist is not run again.
|
||||
written_from: Callable[[Upstream | None], str | None] | None = None
|
||||
|
||||
def contract(self) -> dict[str, list[str]]:
|
||||
"""What a submission must contain, as plain data for the Specialist's context."""
|
||||
@@ -146,6 +151,9 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
|
||||
"describer",
|
||||
depends_on=(_K.TARGET_ARCHITECTURE,),
|
||||
sections=("Components", "Flow"),
|
||||
# It describes the Elements and how they connect. A Revision that only
|
||||
# adds or rewords Checks leaves it true.
|
||||
written_from=graph_structure,
|
||||
),
|
||||
DeliverableSpec(
|
||||
_K.DEPLOYMENT_GUIDANCE,
|
||||
|
||||
@@ -18,7 +18,13 @@ from typing import Any, cast
|
||||
|
||||
from langgraph.graph import END, START, StateGraph
|
||||
|
||||
from app.adk.runtime import AgentRuntime, RoleSpec, RunLimiter, ToolFactory
|
||||
from app.adk.runtime import (
|
||||
AgentRuntime,
|
||||
RoleSpec,
|
||||
RunLimiter,
|
||||
SubmissionReview,
|
||||
ToolFactory,
|
||||
)
|
||||
from app.nodes.execution_nodes import make_execution_nodes
|
||||
from app.nodes.router import OrchestratorPolicy
|
||||
from app.recorder import ExecutionRecorder
|
||||
@@ -36,6 +42,7 @@ def build_execution_graph(
|
||||
limiter: RunLimiter | None = None,
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
submission_review: SubmissionReview | None = None,
|
||||
) -> StateGraph:
|
||||
nodes = make_execution_nodes(
|
||||
runtime=runtime,
|
||||
@@ -47,6 +54,7 @@ def build_execution_graph(
|
||||
limiter=limiter,
|
||||
skill_versions=skill_versions,
|
||||
orchestrator=orchestrator,
|
||||
submission_review=submission_review,
|
||||
)
|
||||
|
||||
graph = StateGraph(ExecutionState)
|
||||
|
||||
@@ -25,10 +25,13 @@ A list. Number functional requirements `FR-1`, `FR-2` and non-functional ones `N
|
||||
```json
|
||||
[
|
||||
{"id": "FR-1", "kind": "functional", "text": "Accept signed webhooks over HTTPS."},
|
||||
{"id": "NFR-1", "kind": "non_functional", "text": "Raw events are retained for 30 days."}
|
||||
{"id": "NFR-1", "kind": "non_functional", "text": "Raw events are encrypted at rest and retained for 30 days.",
|
||||
"conditions": ["Raw events are encrypted at rest.", "Raw events are kept for 30 days and then deleted."]}
|
||||
]
|
||||
```
|
||||
|
||||
Every non-functional requirement lists its `conditions`: one entry for each separate thing that must be true for it to be met. Write each so it can be checked on its own, and so that together they say everything the text says. The Target architecture needs a Check for each one.
|
||||
|
||||
`kind` is `functional` or `non_functional`. Write one requirement per entry, so each can be checked on its own.
|
||||
|
||||
### `components.json` (technical decomposition)
|
||||
@@ -52,9 +55,12 @@ One entry per component, naming the product selected for it.
|
||||
A list. Each recommendation names its pillar and the components it applies to; use an empty list for one that applies to the whole workload.
|
||||
|
||||
```json
|
||||
[{"id": "DR-1", "pillar": "security", "text": "Encrypt the topic with a customer-managed key.", "components": ["C-2"]}]
|
||||
[{"id": "DR-1", "pillar": "security", "text": "Encrypt the topic with a customer-managed key.", "components": ["C-2"],
|
||||
"conditions": ["The topic is encrypted with a customer-managed key.", "The messaging service's own identity may use that key."]}]
|
||||
```
|
||||
|
||||
Every recommendation lists its `conditions`, as a non-functional requirement does. When it applies to several components, give a condition for each.
|
||||
|
||||
Write one configurable decision per recommendation. "Harden the service" cannot become a Check; "disable public ingress on the worker service" can.
|
||||
|
||||
### `target-architecture.json` (Target architecture)
|
||||
@@ -81,6 +87,7 @@ Write one configurable decision per recommendation. "Harden the service" cannot
|
||||
- **Elements.** One per resource to be created. `component` is the component it realises; leave it out for a supporting Element such as an identity, a key or a network. `product` for a component's Element is exactly the product selected for that component.
|
||||
- **Relationships.** `from` and `to` are Element ids. `kind` says what the connection is: calls, publishes to, reads from, runs as, is granted access to.
|
||||
- **Checks.** `applies_to` is an Element or Relationship id. `source` is the requirement or recommendation id the Check comes from. Every Element has at least one Check.
|
||||
- **Coverage.** Every entry in the `conditions` of a non-functional requirement or a recommendation needs a Check that verifies it, with that requirement or recommendation as its `source`. A condition that applies to several Elements needs a Check on each: "processing scales to zero when idle" applies to every processing Element, not one of them.
|
||||
- **Completeness.** Every component has an Element. Every non-functional requirement and every recommendation is the `source` of at least one Check, unless the recommendation is listed under `not_applicable` with a reason. Every selected product appears on its component's Element.
|
||||
|
||||
#### Writing a Check
|
||||
@@ -128,4 +135,5 @@ Implement every Element, Relationship and Check, and nothing the Target architec
|
||||
|
||||
- Give every Check exactly one verdict: `pass`, `fail` or `not_verifiable`, with the evidence you read in the code.
|
||||
- `dry_run.verdict` is `pass`, `fail`, or `not_run` when the tool could not run.
|
||||
- Find every gap in one pass: go through each requirement and recommendation in turn and compare what its text asks with the Checks that name it as `source`.
|
||||
- Judge the code against the Checks and nothing else. Something the Target architecture lacks is a `proposed_additions` entry, with the requirement or recommendation id it serves in `reason`. It is never a failed Check.
|
||||
|
||||
131
tests/execution/test_coverage_live.py
Normal file
131
tests/execution/test_coverage_live.py
Normal file
@@ -0,0 +1,131 @@
|
||||
"""The coverage review against the live decision service. Opt-in: it costs calls.
|
||||
|
||||
LIVE_JEV=1 .venv/bin/python -m pytest tests/execution/test_coverage_live.py
|
||||
|
||||
The graph is the Target architecture evaluation case 02 ended with on 2026-10-07,
|
||||
which validation passed with nothing proposed. Its first two validations proposed
|
||||
four additions; the Checks added for them are removed here to rebuild those gaps.
|
||||
The review must raise each gap, and nothing on the complete graph.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.agent import open_decider
|
||||
from app.config import Settings
|
||||
from app.tools.coverage import CoverageReview
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not os.environ.get("LIVE_JEV"), reason="set LIVE_JEV=1 to call the live service"
|
||||
)
|
||||
|
||||
GRAPH = Path(__file__).parents[1] / "fixtures/coverage/case-02-target-architecture.json"
|
||||
|
||||
# (source, condition, whether the Check for it is removed below)
|
||||
CONDITIONS = [
|
||||
(
|
||||
"NFR-1",
|
||||
"Ingestion endpoints enforce TLS 1.2 or higher for data in transit",
|
||||
True,
|
||||
),
|
||||
("NFR-8", "The Validation Worker's compute scales to zero when idle", True),
|
||||
(
|
||||
"NFR-10",
|
||||
(
|
||||
"The dashboard shows queue depth, processing latency and validation "
|
||||
"failure rate"
|
||||
),
|
||||
True,
|
||||
),
|
||||
("DR-10", "An alert watches the Validation Worker's 5xx error rate", True),
|
||||
("NFR-1", "Stored data is encrypted at rest", False),
|
||||
("NFR-8", "The ingestion service's compute scales to zero when idle", False),
|
||||
("NFR-9", "Quarantined records in dead-letter storage expire after 30 days", False),
|
||||
(
|
||||
"DR-7",
|
||||
"The push subscription has a dead-letter topic with max delivery attempts of 5",
|
||||
False,
|
||||
),
|
||||
("DR-10", "An alert watches the age of the oldest unacknowledged message", False),
|
||||
]
|
||||
GAPS = {condition for _, condition, removed in CONDITIONS if removed}
|
||||
|
||||
|
||||
def upstream() -> dict:
|
||||
by_source: dict[str, list[str]] = {}
|
||||
for source, condition, _ in CONDITIONS:
|
||||
by_source.setdefault(source, []).append(condition)
|
||||
requirements = [
|
||||
{"id": s, "kind": "non_functional", "text": s, "conditions": c}
|
||||
for s, c in by_source.items()
|
||||
if s.startswith("NFR")
|
||||
]
|
||||
recommendations = [
|
||||
{"id": s, "pillar": "reliability", "text": s, "components": [], "conditions": c}
|
||||
for s, c in by_source.items()
|
||||
if s.startswith("DR")
|
||||
]
|
||||
return {
|
||||
"requirements": {
|
||||
"attachments": {"requirements.json": json.dumps(requirements)}
|
||||
},
|
||||
"design_recommendations": {
|
||||
"attachments": {"recommendations.json": json.dumps(recommendations)}
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def with_the_gaps(graph: dict) -> dict:
|
||||
"""The graph without what the two Revisions added."""
|
||||
cut = json.loads(json.dumps(graph))
|
||||
added = {"E-18", "E-19", "E-20"} # the TLS policy and what it attaches to
|
||||
cut["elements"] = [e for e in cut["elements"] if e["id"] not in added]
|
||||
dangling = {
|
||||
r["id"] for r in cut["relationships"] if r["from"] in added or r["to"] in added
|
||||
}
|
||||
cut["relationships"] = [r for r in cut["relationships"] if r["id"] not in dangling]
|
||||
cut["checks"] = [
|
||||
c
|
||||
for c in cut["checks"]
|
||||
if c["applies_to"] not in added | dangling
|
||||
and c["id"] not in {"K-50", "K-51", "K-52"}
|
||||
]
|
||||
for check in cut["checks"]:
|
||||
if check["id"] == "K-25":
|
||||
check["statement"] = "dashboard_json is set"
|
||||
return cut
|
||||
|
||||
|
||||
async def raised(graph: dict) -> set[str]:
|
||||
settings = Settings()
|
||||
decider = open_decider(settings)
|
||||
try:
|
||||
concerns = await CoverageReview(
|
||||
decider, none_at=settings.JEV_COVERAGE_THRESHOLD
|
||||
)(
|
||||
DeliverableKind.TARGET_ARCHITECTURE,
|
||||
{"target-architecture.json": json.dumps(graph)},
|
||||
upstream(),
|
||||
"coverage-live",
|
||||
)
|
||||
finally:
|
||||
aclose = getattr(decider, "aclose", None)
|
||||
if aclose:
|
||||
await aclose()
|
||||
return {
|
||||
condition
|
||||
for _, condition, _ in CONDITIONS
|
||||
if any(f'"{condition}"' in line for line in concerns)
|
||||
}
|
||||
|
||||
|
||||
async def test_nothing_is_raised_on_the_design_validation_passed():
|
||||
assert await raised(json.loads(GRAPH.read_text())) == set()
|
||||
|
||||
|
||||
async def test_every_known_gap_is_raised_and_nothing_else():
|
||||
assert await raised(with_the_gaps(json.loads(GRAPH.read_text()))) == GAPS
|
||||
401
tests/execution/test_coverage_review.py
Normal file
401
tests/execution/test_coverage_review.py
Normal file
@@ -0,0 +1,401 @@
|
||||
"""Two savings on a Revision of the design.
|
||||
|
||||
A coverage review at submission finds what validation would otherwise find later,
|
||||
at the cost of a Revision. And a Deliverable written from the shape of the design is
|
||||
not written again when a Revision only changes its Checks.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
from app.adk.fake_runtime import (
|
||||
SAMPLE_DATA,
|
||||
Call,
|
||||
FakeAgentRuntime,
|
||||
FakeDecider,
|
||||
sample_submission,
|
||||
)
|
||||
from app.adk.runtime import Answer, NoDecider, RoleSpec, Submitted
|
||||
from app.agent import ExecutionEngine
|
||||
from app.nodes.router import Router, Thresholds
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.tools.coverage import CoverageReview
|
||||
from app.tools.execution import specialist_tools
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||
|
||||
K = DeliverableKind
|
||||
GRAPH = {"target-architecture.json": SAMPLE_DATA["target-architecture.json"]}
|
||||
UPSTREAM = {
|
||||
"requirements": {
|
||||
"attachments": {"requirements.json": SAMPLE_DATA["requirements.json"]}
|
||||
},
|
||||
"technical_decomposition": {
|
||||
"attachments": {"components.json": SAMPLE_DATA["components.json"]}
|
||||
},
|
||||
"product_selection": {
|
||||
"attachments": {"products.json": SAMPLE_DATA["products.json"]}
|
||||
},
|
||||
"design_recommendations": {
|
||||
"attachments": {"recommendations.json": SAMPLE_DATA["recommendations.json"]}
|
||||
},
|
||||
}
|
||||
CONTENT = "## Design rationale\nReasons.\n"
|
||||
|
||||
|
||||
# --- the review ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def choice(option: str, probability: float) -> Answer:
|
||||
return Answer(option, probability, {option: probability})
|
||||
|
||||
|
||||
async def test_the_review_asks_which_check_verifies_each_condition():
|
||||
decider = FakeDecider({"NFR-1#1": choice("K-1", 0.9), "DR-1#1": choice("K-2", 0.9)})
|
||||
|
||||
concerns = await CoverageReview(decider)(
|
||||
K.TARGET_ARCHITECTURE, GRAPH, UPSTREAM, "exec-1"
|
||||
)
|
||||
|
||||
state, questions, scope = decider.asked[0]
|
||||
assert concerns == ()
|
||||
assert scope == "exec-1"
|
||||
assert "E-1 Ingress service (Example Run, example_run_service)" in state
|
||||
# Constraints only: a functional requirement is carried by its component.
|
||||
assert set(questions) == {"NFR-1#1", "DR-1#1"}
|
||||
question = questions["NFR-1#1"]
|
||||
assert question.kind == "choice" and question.options == ("K-1", "none")
|
||||
assert 'Condition: "The service stays available when one instance fails."' in (
|
||||
question.instructions
|
||||
)
|
||||
assert "- K-1 on Ingress service: The service runs at least two instances." in (
|
||||
question.instructions
|
||||
)
|
||||
|
||||
|
||||
async def test_a_condition_no_check_verifies_is_raised_with_the_checks_it_has():
|
||||
decider = FakeDecider({"NFR-1#1": choice("none", 0.9), "DR-1#1": choice("K-2", 1)})
|
||||
|
||||
concerns = await CoverageReview(decider)(K.TARGET_ARCHITECTURE, GRAPH, UPSTREAM)
|
||||
|
||||
condition = "The service stays available when one instance fails."
|
||||
assert concerns == (f'NFR-1: no Check verifies "{condition}" (its Checks: K-1)',)
|
||||
|
||||
|
||||
async def test_an_uncertain_none_is_not_raised():
|
||||
decider = FakeDecider({"NFR-1#1": choice("none", 0.45), "DR-1#1": choice("K-2", 1)})
|
||||
|
||||
review = CoverageReview(decider, none_at=0.6)
|
||||
|
||||
assert await review(K.TARGET_ARCHITECTURE, GRAPH, UPSTREAM) == ()
|
||||
|
||||
|
||||
async def test_each_condition_of_a_requirement_is_asked_about_separately():
|
||||
requirements = json.loads(SAMPLE_DATA["requirements.json"])
|
||||
requirements[1]["conditions"] = [
|
||||
"Survives one instance failing.",
|
||||
"Survives a zone failing.",
|
||||
]
|
||||
upstream = {
|
||||
**UPSTREAM,
|
||||
"requirements": {
|
||||
"attachments": {"requirements.json": json.dumps(requirements)}
|
||||
},
|
||||
}
|
||||
decider = FakeDecider(
|
||||
{
|
||||
"NFR-1#1": choice("K-1", 0.9),
|
||||
"NFR-1#2": choice("none", 0.8),
|
||||
"DR-1#1": choice("K-2", 1),
|
||||
}
|
||||
)
|
||||
|
||||
concerns = await CoverageReview(decider)(K.TARGET_ARCHITECTURE, GRAPH, upstream)
|
||||
|
||||
assert len(concerns) == 1 and "Survives a zone failing." in concerns[0]
|
||||
|
||||
|
||||
async def test_an_item_without_conditions_is_read_as_one_condition_its_text():
|
||||
requirements = json.loads(SAMPLE_DATA["requirements.json"])
|
||||
del requirements[1]["conditions"]
|
||||
upstream = {
|
||||
**UPSTREAM,
|
||||
"requirements": {
|
||||
"attachments": {"requirements.json": json.dumps(requirements)}
|
||||
},
|
||||
}
|
||||
decider = FakeDecider({"NFR-1#1": choice("K-1", 1), "DR-1#1": choice("K-2", 1)})
|
||||
|
||||
await CoverageReview(decider)(K.TARGET_ARCHITECTURE, GRAPH, upstream)
|
||||
|
||||
assert 'Condition: "99.9% availability."' in (
|
||||
decider.asked[0][1]["NFR-1#1"].instructions
|
||||
)
|
||||
|
||||
|
||||
async def test_no_decision_and_other_deliverables_raise_nothing():
|
||||
review = CoverageReview(NoDecider())
|
||||
decider = FakeDecider({})
|
||||
|
||||
assert await review(K.TARGET_ARCHITECTURE, GRAPH, UPSTREAM) == ()
|
||||
assert await CoverageReview(decider)(K.REQUIREMENTS, {}, UPSTREAM) == ()
|
||||
assert await CoverageReview(decider)(K.TARGET_ARCHITECTURE, {}, UPSTREAM) == ()
|
||||
assert decider.asked == []
|
||||
|
||||
|
||||
async def test_a_recommendation_marked_not_applicable_is_not_reviewed():
|
||||
graph = json.loads(SAMPLE_DATA["target-architecture.json"])
|
||||
graph["checks"][1]["source"] = "NFR-1"
|
||||
graph["not_applicable"] = [{"recommendation": "DR-1", "reason": "No identity."}]
|
||||
decider = FakeDecider({"NFR-1#1": choice("K-1", 1)})
|
||||
|
||||
await CoverageReview(decider)(
|
||||
K.TARGET_ARCHITECTURE,
|
||||
{"target-architecture.json": json.dumps(graph)},
|
||||
UPSTREAM,
|
||||
)
|
||||
|
||||
assert set(decider.asked[0][1]) == {"NFR-1#1"}
|
||||
|
||||
|
||||
# --- at submission ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def submit_with(review, working=None):
|
||||
(submit,) = (
|
||||
tool
|
||||
for tool in specialist_tools(
|
||||
K.TARGET_ARCHITECTURE,
|
||||
(),
|
||||
allow_cloud_read=False,
|
||||
upstream=UPSTREAM,
|
||||
working={} if working is None else working,
|
||||
review=review,
|
||||
)
|
||||
if tool.name == "submit_deliverable"
|
||||
)
|
||||
return submit.handler
|
||||
|
||||
|
||||
async def test_what_the_review_raises_is_returned_once_and_then_accepted():
|
||||
calls = []
|
||||
|
||||
async def review(attachments):
|
||||
calls.append(attachments)
|
||||
return ['NFR-1: no Check verifies "Two instances." (its Checks: K-1)']
|
||||
|
||||
working: dict = {}
|
||||
submit = submit_with(review, working)
|
||||
|
||||
first = await submit(content=CONTENT, attachments=dict(GRAPH))
|
||||
second = await submit(content=CONTENT)
|
||||
|
||||
assert first.outcome is None
|
||||
assert first.message.startswith("Not submitted. A review found")
|
||||
assert "NFR-1: no Check verifies" in first.message
|
||||
assert "runs once" in first.message
|
||||
# The graph it sent is kept, so it amends that and does not send it again.
|
||||
assert isinstance(second.outcome, Submitted)
|
||||
assert len(calls) == 1
|
||||
|
||||
|
||||
async def test_a_structural_fault_is_reported_before_any_review():
|
||||
async def review(attachments):
|
||||
raise AssertionError("not reached")
|
||||
|
||||
result = await submit_with(review)(content=CONTENT, attachments={})
|
||||
|
||||
assert "attachment: target-architecture.json" in result.message
|
||||
|
||||
|
||||
async def test_a_review_that_fails_or_raises_nothing_never_stops_a_submission():
|
||||
async def broken(attachments):
|
||||
raise RuntimeError("service down")
|
||||
|
||||
async def quiet(attachments):
|
||||
return ()
|
||||
|
||||
for review in (broken, quiet):
|
||||
result = await submit_with(review)(content=CONTENT, attachments=dict(GRAPH))
|
||||
assert isinstance(result.outcome, Submitted)
|
||||
|
||||
|
||||
# --- a description stands while the shape of the design does ---------------------------
|
||||
|
||||
ROLES = {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||
**{
|
||||
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
|
||||
for spec in DEPENDENCY_GRAPH.values()
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def graph_with(**changes) -> Call:
|
||||
graph = json.loads(SAMPLE_DATA["target-architecture.json"])
|
||||
for key, change in changes.items():
|
||||
graph[key] = change(graph[key])
|
||||
return Call(
|
||||
"submit_deliverable",
|
||||
{
|
||||
"content": CONTENT,
|
||||
"attachments": {"target-architecture.json": json.dumps(graph)},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def addition() -> Call:
|
||||
"""A validation that passes every Check and proposes one more."""
|
||||
call = sample_submission(K.VALIDATION_RESULTS)
|
||||
verdicts = json.loads(SAMPLE_DATA["verdicts.json"])
|
||||
verdicts["proposed_additions"] = [
|
||||
{"statement": "A timeout is set.", "applies_to": "E-1", "reason": "NFR-1"}
|
||||
]
|
||||
return Call(
|
||||
call.tool,
|
||||
{**call.args, "attachments": {"verdicts.json": json.dumps(verdicts)}},
|
||||
)
|
||||
|
||||
|
||||
async def run(tmp_path, second_graph: Call, **options):
|
||||
turns = {
|
||||
spec.specialist: [[sample_submission(kind)]] * 2
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
turns["target-architect"] = [
|
||||
[sample_submission(K.TARGET_ARCHITECTURE)],
|
||||
[second_graph],
|
||||
]
|
||||
turns["validator"] = [[addition()], [sample_submission(K.VALIDATION_RESULTS)]]
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=ROLES,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
**options,
|
||||
)
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True)
|
||||
assert result.status == "completed"
|
||||
return result, [run.role for run in runtime.runs]
|
||||
|
||||
|
||||
async def test_the_description_is_kept_when_a_revision_only_changes_checks(tmp_path):
|
||||
reworded = graph_with(
|
||||
checks=lambda checks: [
|
||||
{**checks[0], "statement": "min_instance_count is at least 2"},
|
||||
checks[1],
|
||||
]
|
||||
)
|
||||
|
||||
result, roles_run = await run(tmp_path, reworded)
|
||||
|
||||
assert roles_run.count("target-architect") == 2
|
||||
assert roles_run.count("iac-author") == 2 # the code answers the Checks
|
||||
assert roles_run.count("describer") == 1
|
||||
# The Ledger still records a second submission: the Deliverable was not skipped.
|
||||
submitted = [
|
||||
event
|
||||
for event in result.state["ledger"]["events"]
|
||||
if event["type"] == "submitted"
|
||||
and event["subject"] == "architecture_description"
|
||||
]
|
||||
assert len(submitted) == 2
|
||||
|
||||
|
||||
async def test_the_description_is_written_again_when_the_shape_changes(tmp_path):
|
||||
renamed = graph_with(
|
||||
elements=lambda elements: [{**elements[0], "name": "Edge service"}, elements[1]]
|
||||
)
|
||||
|
||||
_, roles_run = await run(tmp_path, renamed)
|
||||
|
||||
assert roles_run.count("describer") == 2
|
||||
|
||||
|
||||
async def test_the_engine_reviews_a_design_before_it_is_submitted(tmp_path):
|
||||
raised = []
|
||||
|
||||
async def review(kind, attachments, upstream, scope=""):
|
||||
if kind is K.TARGET_ARCHITECTURE and not raised:
|
||||
raised.append(scope)
|
||||
return ['NFR-1 "Two instances." (its Checks: K-1)']
|
||||
return ()
|
||||
|
||||
turns = {
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
turns["target-architect"] = [
|
||||
[sample_submission(K.TARGET_ARCHITECTURE)] * 2 # refused once, then accepted
|
||||
]
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
engine = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=ROLES,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
submission_review=review,
|
||||
)
|
||||
|
||||
result = await engine.start("Ingest webhooks", pre_approved=True, execution_id="e1")
|
||||
|
||||
assert result.status == "completed"
|
||||
assert raised == ["e1"]
|
||||
architect = next(run for run in runtime.runs if run.role == "target-architect")
|
||||
assert architect.results[0][1].startswith("Not submitted. A review found")
|
||||
# What a review raises is kept as a lesson candidate, like any refusal.
|
||||
assert any(
|
||||
"A review found" in json.dumps(outcome.get("refusals"))
|
||||
for outcome in result.state["outcomes"]
|
||||
)
|
||||
|
||||
|
||||
async def test_feedback_on_the_description_itself_has_it_written_again(tmp_path):
|
||||
"""Kept only while nobody has asked for the description to change."""
|
||||
from app.nodes.execution_nodes import STANDING, make_execution_nodes
|
||||
|
||||
nodes = make_execution_nodes(
|
||||
runtime=FakeAgentRuntime(
|
||||
{"describer": [[sample_submission(K.ARCHITECTURE_DESCRIPTION, "new")]]}
|
||||
),
|
||||
roles=ROLES,
|
||||
recorder=InMemoryRecorder(),
|
||||
output_root=tmp_path,
|
||||
)
|
||||
upstream = {"target_architecture": {"attachments": dict(GRAPH)}}
|
||||
from app.workflows.chain import graph_structure
|
||||
|
||||
standing = {
|
||||
"basis": graph_structure(upstream),
|
||||
"feedback": [],
|
||||
"content": "old",
|
||||
"attachments": {},
|
||||
}
|
||||
task = {
|
||||
"execution_id": "e1",
|
||||
"role": "describer",
|
||||
"kind": "architecture_description",
|
||||
"attempt": 2,
|
||||
"allow_cloud_read": False,
|
||||
"working": {STANDING: standing},
|
||||
"context": {
|
||||
"dependencies": upstream,
|
||||
"feedback": ["Dependency x was revised: y"],
|
||||
},
|
||||
}
|
||||
|
||||
kept = await nodes.specialist_task(task)
|
||||
asked = await nodes.specialist_task(
|
||||
{**task, "context": {**task["context"], "feedback": ["Make the flow clearer."]}}
|
||||
)
|
||||
|
||||
assert kept["outcomes"][0]["content"] == "old"
|
||||
assert "new" in asked["outcomes"][0]["content"]
|
||||
assert asked["working"]["architecture_description"][STANDING]["feedback"] == [
|
||||
"Make the flow clearer."
|
||||
]
|
||||
@@ -665,3 +665,51 @@ async def test_the_orchestrator_is_told_what_validation_decided(tmp_path):
|
||||
run for run in runtime.runs if run.role == specialist(K.DEPLOYMENT_GUIDANCE)
|
||||
][1]
|
||||
assert revised.task.context["feedback"] == ["K-1 failed: min instances is 1"]
|
||||
|
||||
|
||||
# --- constraints list their conditions --------------------------------------------------
|
||||
|
||||
|
||||
def test_a_non_functional_requirement_must_list_its_conditions():
|
||||
requirements = [
|
||||
{"id": "FR-1", "kind": "functional", "text": "Accept events."},
|
||||
{"id": "NFR-1", "kind": "non_functional", "text": "Encrypted and retained."},
|
||||
{
|
||||
"id": "NFR-2",
|
||||
"kind": "non_functional",
|
||||
"text": "Available.",
|
||||
"conditions": [],
|
||||
},
|
||||
]
|
||||
args = sample_submission(K.REQUIREMENTS).args
|
||||
|
||||
missing = check_submission(
|
||||
K.REQUIREMENTS,
|
||||
args["content"],
|
||||
{"requirements.json": json.dumps(requirements)},
|
||||
)
|
||||
|
||||
needs = (
|
||||
"needs `conditions`: a list with one entry for each separate thing it asks for"
|
||||
)
|
||||
assert missing == (
|
||||
f"requirements.json: NFR-1 {needs}",
|
||||
f"requirements.json: NFR-2 {needs}",
|
||||
)
|
||||
|
||||
|
||||
def test_a_recommendation_must_list_its_conditions():
|
||||
recommendations = json.loads(SAMPLE_DATA["recommendations.json"])
|
||||
del recommendations[0]["conditions"]
|
||||
args = sample_submission(K.DESIGN_RECOMMENDATIONS).args
|
||||
|
||||
missing = check_submission(
|
||||
K.DESIGN_RECOMMENDATIONS,
|
||||
args["content"],
|
||||
{"recommendations.json": json.dumps(recommendations)},
|
||||
)
|
||||
|
||||
needs = (
|
||||
"needs `conditions`: a list with one entry for each separate thing it asks for"
|
||||
)
|
||||
assert missing == (f"recommendations.json: DR-1 {needs}",)
|
||||
|
||||
604
tests/fixtures/coverage/case-02-target-architecture.json
vendored
Normal file
604
tests/fixtures/coverage/case-02-target-architecture.json
vendored
Normal file
@@ -0,0 +1,604 @@
|
||||
{
|
||||
"elements": [
|
||||
{
|
||||
"name": "Ingestion Endpoint Service",
|
||||
"product": "Cloud Run",
|
||||
"resource_type": "google_cloud_run_v2_service",
|
||||
"id": "E-1",
|
||||
"component": "C-1"
|
||||
},
|
||||
{
|
||||
"resource_type": "google_pubsub_topic",
|
||||
"component": "C-2",
|
||||
"product": "Cloud Pub/Sub",
|
||||
"id": "E-2",
|
||||
"name": "Asynchronous Ingestion Buffer Topic"
|
||||
},
|
||||
{
|
||||
"id": "E-3",
|
||||
"component": "C-3",
|
||||
"product": "Cloud Run",
|
||||
"resource_type": "google_cloud_run_v2_service",
|
||||
"name": "Validation Worker"
|
||||
},
|
||||
{
|
||||
"component": "C-4",
|
||||
"name": "Validated Analytical Data Dataset",
|
||||
"id": "E-4",
|
||||
"resource_type": "google_bigquery_dataset",
|
||||
"product": "BigQuery"
|
||||
},
|
||||
{
|
||||
"id": "E-5",
|
||||
"resource_type": "google_storage_bucket",
|
||||
"component": "C-5",
|
||||
"name": "Quarantine Storage Bucket",
|
||||
"product": "Cloud Storage"
|
||||
},
|
||||
{
|
||||
"product": "Cloud Logging and Monitoring",
|
||||
"name": "Queue Lag Alert Policy",
|
||||
"resource_type": "google_monitoring_alert_policy",
|
||||
"id": "E-6",
|
||||
"component": "C-6"
|
||||
},
|
||||
{
|
||||
"component": "C-6",
|
||||
"name": "Error Rate Alert Policy",
|
||||
"product": "Cloud Logging and Monitoring",
|
||||
"resource_type": "google_monitoring_alert_policy",
|
||||
"id": "E-7"
|
||||
},
|
||||
{
|
||||
"resource_type": "google_pubsub_subscription",
|
||||
"component": "C-2",
|
||||
"name": "Asynchronous Ingestion Subscription",
|
||||
"id": "E-8",
|
||||
"product": "Cloud Pub/Sub"
|
||||
},
|
||||
{
|
||||
"resource_type": "google_pubsub_topic",
|
||||
"product": "Cloud Pub/Sub",
|
||||
"id": "E-9",
|
||||
"component": "C-2",
|
||||
"name": "Dead-Letter Topic"
|
||||
},
|
||||
{
|
||||
"name": "Ingestion Service Account",
|
||||
"id": "E-10",
|
||||
"resource_type": "google_service_account",
|
||||
"product": "IAM"
|
||||
},
|
||||
{
|
||||
"product": "IAM",
|
||||
"id": "E-11",
|
||||
"name": "Validation Worker Service Account",
|
||||
"resource_type": "google_service_account"
|
||||
},
|
||||
{
|
||||
"product": "IAM",
|
||||
"id": "E-12",
|
||||
"resource_type": "google_service_account",
|
||||
"name": "Pub/Sub Push Service Account"
|
||||
},
|
||||
{
|
||||
"product": "Cloud KMS",
|
||||
"name": "CMEK Crypto Key",
|
||||
"id": "E-13",
|
||||
"resource_type": "google_kms_crypto_key"
|
||||
},
|
||||
{
|
||||
"name": "Load Balancer Backend",
|
||||
"product": "Cloud Load Balancing",
|
||||
"resource_type": "google_compute_backend_service",
|
||||
"id": "E-14"
|
||||
},
|
||||
{
|
||||
"product": "Cloud Armor",
|
||||
"id": "E-15",
|
||||
"name": "Cloud Armor Security Policy",
|
||||
"resource_type": "google_compute_security_policy"
|
||||
},
|
||||
{
|
||||
"id": "E-16",
|
||||
"product": "Cloud Logging and Monitoring",
|
||||
"name": "Audit Log Bucket Config",
|
||||
"resource_type": "google_logging_project_bucket_config"
|
||||
},
|
||||
{
|
||||
"resource_type": "google_monitoring_dashboard",
|
||||
"product": "Cloud Logging and Monitoring",
|
||||
"id": "E-17",
|
||||
"component": "C-6",
|
||||
"name": "Operational Dashboard"
|
||||
},
|
||||
{
|
||||
"id": "E-18",
|
||||
"component": "C-1",
|
||||
"resource_type": "google_compute_ssl_policy",
|
||||
"product": "Cloud Load Balancing",
|
||||
"name": "Load Balancer SSL Policy"
|
||||
},
|
||||
{
|
||||
"resource_type": "google_compute_target_https_proxy",
|
||||
"product": "Cloud Load Balancing",
|
||||
"name": "Load Balancer Target HTTPS Proxy",
|
||||
"component": "C-1",
|
||||
"id": "E-19"
|
||||
},
|
||||
{
|
||||
"name": "Load Balancer URL Map",
|
||||
"resource_type": "google_compute_url_map",
|
||||
"id": "E-20",
|
||||
"product": "Cloud Load Balancing",
|
||||
"component": "C-1"
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"to": "E-2",
|
||||
"id": "R-1",
|
||||
"kind": "publishes to",
|
||||
"from": "E-1"
|
||||
},
|
||||
{
|
||||
"id": "R-2",
|
||||
"kind": "runs as",
|
||||
"from": "E-1",
|
||||
"to": "E-10"
|
||||
},
|
||||
{
|
||||
"to": "E-2",
|
||||
"kind": "is granted access to",
|
||||
"id": "R-3",
|
||||
"from": "E-10"
|
||||
},
|
||||
{
|
||||
"kind": "has subscription",
|
||||
"from": "E-2",
|
||||
"to": "E-8",
|
||||
"id": "R-4"
|
||||
},
|
||||
{
|
||||
"to": "E-3",
|
||||
"id": "R-5",
|
||||
"kind": "pushes to",
|
||||
"from": "E-8"
|
||||
},
|
||||
{
|
||||
"kind": "authenticates as",
|
||||
"from": "E-8",
|
||||
"id": "R-6",
|
||||
"to": "E-12"
|
||||
},
|
||||
{
|
||||
"kind": "is granted access to",
|
||||
"from": "E-12",
|
||||
"to": "E-3",
|
||||
"id": "R-7"
|
||||
},
|
||||
{
|
||||
"id": "R-8",
|
||||
"to": "E-11",
|
||||
"from": "E-3",
|
||||
"kind": "runs as"
|
||||
},
|
||||
{
|
||||
"to": "E-4",
|
||||
"id": "R-9",
|
||||
"from": "E-3",
|
||||
"kind": "writes to"
|
||||
},
|
||||
{
|
||||
"kind": "writes to",
|
||||
"from": "E-3",
|
||||
"to": "E-5",
|
||||
"id": "R-10"
|
||||
},
|
||||
{
|
||||
"id": "R-11",
|
||||
"kind": "is granted access to",
|
||||
"to": "E-4",
|
||||
"from": "E-11"
|
||||
},
|
||||
{
|
||||
"kind": "is granted access to",
|
||||
"from": "E-11",
|
||||
"to": "E-5",
|
||||
"id": "R-12"
|
||||
},
|
||||
{
|
||||
"kind": "routes dead letters to",
|
||||
"from": "E-8",
|
||||
"id": "R-13",
|
||||
"to": "E-9"
|
||||
},
|
||||
{
|
||||
"to": "E-13",
|
||||
"id": "R-14",
|
||||
"kind": "is encrypted with",
|
||||
"from": "E-2"
|
||||
},
|
||||
{
|
||||
"kind": "is encrypted with",
|
||||
"from": "E-5",
|
||||
"to": "E-13",
|
||||
"id": "R-15"
|
||||
},
|
||||
{
|
||||
"id": "R-16",
|
||||
"from": "E-14",
|
||||
"kind": "fronts",
|
||||
"to": "E-1"
|
||||
},
|
||||
{
|
||||
"kind": "uses policy",
|
||||
"from": "E-14",
|
||||
"to": "E-15",
|
||||
"id": "R-17"
|
||||
},
|
||||
{
|
||||
"from": "E-6",
|
||||
"kind": "monitors",
|
||||
"to": "E-8",
|
||||
"id": "R-18"
|
||||
},
|
||||
{
|
||||
"to": "E-1",
|
||||
"kind": "monitors",
|
||||
"from": "E-7",
|
||||
"id": "R-19"
|
||||
},
|
||||
{
|
||||
"id": "R-20",
|
||||
"from": "E-7",
|
||||
"kind": "monitors",
|
||||
"to": "E-3"
|
||||
},
|
||||
{
|
||||
"id": "R-21",
|
||||
"kind": "routes to",
|
||||
"from": "E-20",
|
||||
"to": "E-14"
|
||||
},
|
||||
{
|
||||
"to": "E-20",
|
||||
"from": "E-19",
|
||||
"kind": "uses map",
|
||||
"id": "R-22"
|
||||
},
|
||||
{
|
||||
"to": "E-18",
|
||||
"kind": "uses policy",
|
||||
"from": "E-19",
|
||||
"id": "R-23"
|
||||
}
|
||||
],
|
||||
"checks": [
|
||||
{
|
||||
"applies_to": "E-1",
|
||||
"source": "DR-1",
|
||||
"statement": "ingress is INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER",
|
||||
"id": "K-1"
|
||||
},
|
||||
{
|
||||
"id": "K-2",
|
||||
"applies_to": "E-1",
|
||||
"statement": "template.service_account points to E-10",
|
||||
"source": "DR-2"
|
||||
},
|
||||
{
|
||||
"source": "DR-12",
|
||||
"applies_to": "E-1",
|
||||
"statement": "template.containers.resources.startup_cpu_boost is true",
|
||||
"id": "K-3"
|
||||
},
|
||||
{
|
||||
"source": "DR-13",
|
||||
"applies_to": "E-1",
|
||||
"statement": "template.max_instance_request_concurrency is at least 80",
|
||||
"id": "K-4"
|
||||
},
|
||||
{
|
||||
"id": "K-5",
|
||||
"applies_to": "E-1",
|
||||
"statement": "location is a Low-CO2 region",
|
||||
"source": "DR-14"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-2",
|
||||
"statement": "kms_key_name points to E-13",
|
||||
"source": "DR-5",
|
||||
"id": "K-6"
|
||||
},
|
||||
{
|
||||
"source": "DR-3",
|
||||
"applies_to": "E-3",
|
||||
"id": "K-7",
|
||||
"statement": "template.service_account points to E-11"
|
||||
},
|
||||
{
|
||||
"id": "K-8",
|
||||
"statement": "template.containers.resources.startup_cpu_boost is true",
|
||||
"applies_to": "E-3",
|
||||
"source": "DR-12"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-3",
|
||||
"id": "K-9",
|
||||
"statement": "template.max_instance_request_concurrency is at least 80",
|
||||
"source": "DR-13"
|
||||
},
|
||||
{
|
||||
"id": "K-10",
|
||||
"statement": "location is a Low-CO2 region",
|
||||
"applies_to": "E-3",
|
||||
"source": "DR-14"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-4",
|
||||
"id": "K-11",
|
||||
"statement": "location is set to a multi-region",
|
||||
"source": "DR-9"
|
||||
},
|
||||
{
|
||||
"id": "K-12",
|
||||
"statement": "encryption.default_kms_key_name points to E-13",
|
||||
"applies_to": "E-5",
|
||||
"source": "DR-5"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-5",
|
||||
"id": "K-13",
|
||||
"statement": "uniform_bucket_level_access is true",
|
||||
"source": "DR-6"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-5",
|
||||
"statement": "public_access_prevention is enforced",
|
||||
"source": "DR-6",
|
||||
"id": "K-14"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-5",
|
||||
"id": "K-15",
|
||||
"statement": "location is set to a multi-region",
|
||||
"source": "DR-9"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-5",
|
||||
"source": "DR-11",
|
||||
"id": "K-16",
|
||||
"statement": "lifecycle_rule has action.type = Delete and condition.age = 30"
|
||||
},
|
||||
{
|
||||
"source": "DR-10",
|
||||
"id": "K-17",
|
||||
"applies_to": "E-6",
|
||||
"statement": "conditions block filters on oldest_unacked_message_age"
|
||||
},
|
||||
{
|
||||
"id": "K-19",
|
||||
"source": "DR-4",
|
||||
"statement": "push_config.oidc_token.service_account_email points to E-12",
|
||||
"applies_to": "E-8"
|
||||
},
|
||||
{
|
||||
"id": "K-20",
|
||||
"statement": "dead_letter_policy points to E-9",
|
||||
"applies_to": "E-8",
|
||||
"source": "DR-7"
|
||||
},
|
||||
{
|
||||
"id": "K-21",
|
||||
"source": "DR-7",
|
||||
"applies_to": "E-8",
|
||||
"statement": "dead_letter_policy.max_delivery_attempts is 5"
|
||||
},
|
||||
{
|
||||
"statement": "retry_policy block is configured with minimum_backoff and maximum_backoff",
|
||||
"id": "K-22",
|
||||
"applies_to": "E-8",
|
||||
"source": "DR-8"
|
||||
},
|
||||
{
|
||||
"statement": "retention_days is at least 365",
|
||||
"applies_to": "E-16",
|
||||
"id": "K-23",
|
||||
"source": "NFR-4"
|
||||
},
|
||||
{
|
||||
"source": "NFR-4",
|
||||
"statement": "locked is true",
|
||||
"applies_to": "E-16",
|
||||
"id": "K-24"
|
||||
},
|
||||
{
|
||||
"id": "K-25",
|
||||
"source": "NFR-10",
|
||||
"applies_to": "E-17",
|
||||
"statement": "dashboard_json contains widgets for queue depth, processing latency, and validation failure rate"
|
||||
},
|
||||
{
|
||||
"statement": "role is roles/pubsub.publisher",
|
||||
"id": "K-26",
|
||||
"source": "NFR-2",
|
||||
"applies_to": "R-3"
|
||||
},
|
||||
{
|
||||
"id": "K-27",
|
||||
"statement": "role is roles/run.invoker",
|
||||
"source": "DR-4",
|
||||
"applies_to": "R-7"
|
||||
},
|
||||
{
|
||||
"source": "DR-3",
|
||||
"id": "K-28",
|
||||
"statement": "role is roles/bigquery.dataEditor",
|
||||
"applies_to": "R-11"
|
||||
},
|
||||
{
|
||||
"id": "K-29",
|
||||
"source": "NFR-2",
|
||||
"statement": "role is roles/storage.objectCreator",
|
||||
"applies_to": "R-12"
|
||||
},
|
||||
{
|
||||
"id": "K-30",
|
||||
"source": "NFR-1",
|
||||
"statement": "default_encryption_configuration is configured",
|
||||
"applies_to": "E-4"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-1",
|
||||
"statement": "template.scaling.min_instance_count is 0 or not set",
|
||||
"source": "NFR-8",
|
||||
"id": "K-31"
|
||||
},
|
||||
{
|
||||
"source": "NFR-12",
|
||||
"id": "K-32",
|
||||
"statement": "template.scaling.max_instance_count is at least 100",
|
||||
"applies_to": "E-1"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-9",
|
||||
"id": "K-33",
|
||||
"source": "DR-7",
|
||||
"statement": "message_retention_duration is configured"
|
||||
},
|
||||
{
|
||||
"statement": "account_id is configured",
|
||||
"source": "DR-2",
|
||||
"applies_to": "E-10",
|
||||
"id": "K-34"
|
||||
},
|
||||
{
|
||||
"statement": "account_id is configured",
|
||||
"id": "K-35",
|
||||
"applies_to": "E-11",
|
||||
"source": "DR-3"
|
||||
},
|
||||
{
|
||||
"id": "K-36",
|
||||
"statement": "account_id is configured",
|
||||
"applies_to": "E-12",
|
||||
"source": "DR-4"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-13",
|
||||
"statement": "rotation_period is configured",
|
||||
"id": "K-37",
|
||||
"source": "DR-5"
|
||||
},
|
||||
{
|
||||
"source": "DR-1",
|
||||
"id": "K-38",
|
||||
"applies_to": "E-14",
|
||||
"statement": "security_policy points to E-15"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-15",
|
||||
"statement": "type is CLOUD_ARMOR",
|
||||
"id": "K-39",
|
||||
"source": "DR-1"
|
||||
},
|
||||
{
|
||||
"source": "NFR-3",
|
||||
"id": "K-40",
|
||||
"applies_to": "E-5",
|
||||
"statement": "public_access_prevention is enforced"
|
||||
},
|
||||
{
|
||||
"statement": "location is configured",
|
||||
"source": "NFR-5",
|
||||
"id": "K-41",
|
||||
"applies_to": "E-1"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-2",
|
||||
"statement": "message_retention_duration is configured",
|
||||
"source": "NFR-6",
|
||||
"id": "K-42"
|
||||
},
|
||||
{
|
||||
"applies_to": "E-4",
|
||||
"statement": "location is set to a multi-region",
|
||||
"id": "K-43",
|
||||
"source": "NFR-7"
|
||||
},
|
||||
{
|
||||
"statement": "lifecycle_rule has action.type = Delete and condition.age = 30",
|
||||
"source": "NFR-9",
|
||||
"id": "K-44",
|
||||
"applies_to": "E-5"
|
||||
},
|
||||
{
|
||||
"source": "NFR-11",
|
||||
"applies_to": "E-6",
|
||||
"id": "K-45",
|
||||
"statement": "conditions block filters on oldest_unacked_message_age"
|
||||
},
|
||||
{
|
||||
"statement": "template.containers.resources.startup_cpu_boost is true",
|
||||
"source": "NFR-13",
|
||||
"applies_to": "E-1",
|
||||
"id": "K-46"
|
||||
},
|
||||
{
|
||||
"source": "NFR-14",
|
||||
"applies_to": "E-8",
|
||||
"statement": "push_config is configured",
|
||||
"id": "K-47"
|
||||
},
|
||||
{
|
||||
"source": "NFR-8",
|
||||
"id": "K-50",
|
||||
"statement": "template.scaling.min_instance_count is 0",
|
||||
"applies_to": "E-3"
|
||||
},
|
||||
{
|
||||
"statement": "template.scaling.max_instance_count is configured to a baseline max",
|
||||
"applies_to": "E-3",
|
||||
"source": "NFR-8",
|
||||
"id": "K-51"
|
||||
},
|
||||
{
|
||||
"statement": "condition filters on validation worker Cloud Run 5xx errors",
|
||||
"source": "DR-10",
|
||||
"applies_to": "R-20",
|
||||
"id": "K-52"
|
||||
},
|
||||
{
|
||||
"source": "NFR-1",
|
||||
"statement": "min_tls_version is TLS_1_2",
|
||||
"id": "K-53",
|
||||
"applies_to": "E-18"
|
||||
},
|
||||
{
|
||||
"id": "K-54",
|
||||
"source": "NFR-1",
|
||||
"applies_to": "E-19",
|
||||
"statement": "ssl_policy points to E-18"
|
||||
},
|
||||
{
|
||||
"source": "NFR-1",
|
||||
"statement": "url_map points to E-20",
|
||||
"applies_to": "E-19",
|
||||
"id": "K-55"
|
||||
},
|
||||
{
|
||||
"id": "K-56",
|
||||
"source": "NFR-1",
|
||||
"applies_to": "E-20",
|
||||
"statement": "default_service points to E-14"
|
||||
},
|
||||
{
|
||||
"statement": "enabled is true",
|
||||
"applies_to": "E-7",
|
||||
"id": "K-57",
|
||||
"source": "NFR-11"
|
||||
}
|
||||
],
|
||||
"not_applicable": []
|
||||
}
|
||||
Reference in New Issue
Block a user