feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -9,9 +9,9 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||
|
||||
@@ -204,3 +204,27 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
||||
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
|
||||
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.
|
||||
|
||||
## Step 25: An Execution that ends with warnings keeps its files
|
||||
|
||||
- **Report ref:** none; found by the runs of 2026-10-07 and 08
|
||||
- **Capability:** What an unfinished Execution produced can be read afterwards.
|
||||
- **Rationale:** Only a completed Execution wrote its Deliverables. Case 04's first run ended with warnings after 53 minutes and left nothing on disk: nine Deliverables' worth of work, and the evidence of what went wrong, existed only in the report's conversation log.
|
||||
- **Where:** `_write_outputs` and `end_with_warnings` in `app/nodes/execution_nodes.py`.
|
||||
- **Acceptance criteria:** An Execution that ends with warnings writes every Deliverable it holds, with a manifest that carries the warnings, each Deliverable's status and each Phase's status, so nothing unapproved reads as final. The case's history points to the files.
|
||||
|
||||
## Step 26: Each pillar's checklist is reviewed and the review is in the Deliverable
|
||||
|
||||
- **Report ref:** none; found by the judged runs of 2026-10-08
|
||||
- **Capability:** An architect sees every item of each pillar's checklist accounted for: addressed, and how, or not applicable, and why.
|
||||
- **Rationale:** The judged check `pillar_checklists_covered` failed in all four cases of the judged batch, at 0% to 55% a pillar, while `recommendations_quality` passed at 0.83 to 0.90. The six checklists hold 78 items; a run produces 11 to 14 recommendations, each one configurable decision. Twelve recommendations cannot answer 78 items, many of which are an organisation's process and not a design decision, so the check could not pass and nothing told an architect which items had been considered.
|
||||
- **Where:** `app/tools/pillars.py`, `submit_deliverable` in `app/tools/execution.py`, `agents/pillar-advisor.md`, `agents/design-advisor.md`.
|
||||
- **Acceptance criteria:** Each pillar advisor returns, with its advice, a status and a note for every item of its Skill's checklist. The reviews are kept with the advice across Revisions and written into the design recommendations by code, once. On a judged run `pillar_checklists_covered` passes.
|
||||
|
||||
## Step 27: A user sees a Deliverable before it is first submitted
|
||||
|
||||
- **Report ref:** none; found by evaluation case 04 on 2026-10-08
|
||||
- **Capability:** With a user present, no Deliverable is submitted that they have not been shown.
|
||||
- **Rationale:** The Specialist protocol tells every Specialist to confirm its draft with the user before submitting. In case 04 the validator did not (first run), and the target architect did not after running out of tool calls on an earlier attempt (batch A). An instruction was not enough.
|
||||
- **Where:** `submit_deliverable` in `app/tools/execution.py`, `specialist_task` in `app/nodes/execution_nodes.py`, `CONFIRM_DRAFTS` in `app/config.py`.
|
||||
- **Acceptance criteria:** In an Execution with a user, a Specialist's submission is refused until the user has replied to it about that Deliverable at least once, and the refusal says how to ask. A pre-approved Execution is unaffected. It can be switched off.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 complete (ADR-0004 accepted 2026-10-08), which unblocks steps 10 and 12; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
||||
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
||||
**This directory covers:** steps 1 to 24
|
||||
**This directory covers:** steps 1 to 27
|
||||
|
||||
## Sources of truth
|
||||
|
||||
@@ -19,12 +19,12 @@
|
||||
|---|---|---|---|---|
|
||||
| 1 | Structured validator findings | A | S | replaced by iteration 6 step 9 (verdicts per Check) |
|
||||
| 2 | Carry the draft through `needs_input` | A | S | complete |
|
||||
| 3 | Reuse pillar advice across Revisions | A | S | built; evaluation run outstanding |
|
||||
| 3 | Reuse pillar advice across Revisions | A | S | built; no run has revised design recommendations, so unexercised by evaluation |
|
||||
| 4 | Single source for Deliverable contracts | A | S | complete |
|
||||
| 5 | Cap parallel Specialists | A | S | complete |
|
||||
| 6 | Orchestrator step order | A | S | built; seen working in one stopped run, no completed evaluation |
|
||||
| 7 | Capture lesson candidates | B | M | complete |
|
||||
| 8 | Evaluation that can see improvement | B | M | built; thresholds and a judged run outstanding |
|
||||
| 8 | Evaluation that can see improvement | B | M | complete: thresholds set and four judged runs done on 2026-10-08 |
|
||||
| 9 | ADR: the lessons loop and overlay Skills | B | S | complete: ADR-0004 accepted 2026-10-08 |
|
||||
| 10 | Lessons Skill and lesson curator | B | L | planned |
|
||||
| 11 | IaC patterns Skill | B | M | built; in case 01 the IaC author fell from 57 requests to 11 with no Revision; other cases unmeasured |
|
||||
@@ -41,6 +41,9 @@
|
||||
| 22 | Validation reports every gap in one pass | added | S | instructions changed; with conditions listed, case 02 ran with 102 Checks and no Revision |
|
||||
| 23 | Validation recorded a few verdicts at a time | added | S | built; in case 02 a validator run cut short carried on and submitted in 68 seconds, with no timeout |
|
||||
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; raised nothing in case 02, which had no unimplementable Check |
|
||||
| 25 | An Execution that ends with warnings keeps its files | added | S | complete |
|
||||
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
|
||||
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; a hole found by a run and closed; not rerun |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
|
||||
@@ -10,4 +10,7 @@ Differs from the plan: the plan's acceptance is a fall in IaC author Revisions o
|
||||
|
||||
Tests: `tests/test_iac_patterns.py` runs `terraform validate` on every reference's pattern with the Skill's own versions file (13 pass against the real provider from the plugin cache; skipped when the provider cannot be installed), checks every reference is listed in `SKILL.md`, and checks the Skill and the bundle pin the same version.
|
||||
|
||||
Not done: load balancing, Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
|
||||
Not done: Firestore, BigQuery, Eventarc and GKE have no reference. A product with none is written from `terraform_schema` and the documentation.
|
||||
|
||||
Added on 2026-10-08: `references/load-balancer.md`, a fourteenth pattern: an external Application Load Balancer in front of Cloud Run with a TLS policy, a firewall policy, Identity-Aware Proxy and an HTTP redirect. Cases 02, 03 and 04 all used a load balancer with no pattern, and one run put `ssl_policy` on a backend service, which has no such setting; the reference says which link of the chain each setting lives on. It validates against the pinned provider.
|
||||
|
||||
|
||||
14
.agents/plans/iter-04-learning-and-skills/steps/step-25.md
Normal file
14
.agents/plans/iter-04-learning-and-skills/steps/step-25.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 25 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `_write_outputs` takes the Execution's final status and its Ledger. `end_with_warnings` now calls it, as `finalize` does.
|
||||
2. `execution.json` gains `status` (`completed` or `completed_with_warnings`), `warnings`, `phases` (each Phase's status), and a `status` on every Deliverable: `submitted`, or `draft` when something it depends on was reopened after it was written.
|
||||
3. With no Deliverable at all nothing is written, and the case's history records no files, as before.
|
||||
4. The solution guide is written only when it exists; an Execution that stopped before packaging has none, and the manifest says so.
|
||||
|
||||
Not covered: an Execution whose process is killed. It never reaches either end node, so it still writes nothing. The evaluation's progress file (iteration 6 step 11) records its time and tokens, not its Deliverables.
|
||||
|
||||
Also on 2026-10-08, outside any step: the CI workflow installs `requirements.txt` and `requirements-dev.txt` before linting and testing. It installed only ruff, mypy and pytest, so its test step could not import the application.
|
||||
|
||||
Tests: `tests/execution/test_engine.py`, two updated.
|
||||
17
.agents/plans/iter-04-learning-and-skills/steps/step-26.md
Normal file
17
.agents/plans/iter-04-learning-and-skills/steps/step-26.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Step 26 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `submit_advice` takes a `checklist`: one entry per item of the pillar Skill's validation checklist, with `status` (`addressed`, `not_applicable` or `not_addressed`) and a `note`. An entry with no item or another status is dropped.
|
||||
2. The consultation keeps each pillar's review in the Deliverable's working memory (`pillar_checklists`) beside its advice, so a Revision that re-consults some pillars keeps the others' reviews. The design advisor is given the advice only, not the review.
|
||||
3. `render_checklist_review` writes the reviews as one section, "Architecture framework checklist review", a table per pillar. `submit_deliverable` appends it to the design recommendations and replaces an earlier copy, so a Revision never stacks two.
|
||||
4. A Skill with no checklist gets no review and the Deliverable no section: the mechanism is the provider's Skills' own, not Google Cloud's.
|
||||
|
||||
Checked on 2026-10-08 with two real advisors on case 03's brief: reliability returned all 8 of its items (6 addressed, 2 not applicable) and cost optimisation all 11 (11 addressed), in 74 seconds for the pair.
|
||||
|
||||
Two limits:
|
||||
|
||||
- A review says what the advisor's own advice covers. The design advisor reconciles the pillars and may leave something out, so "addressed" there is the advisor's claim, not a check of the final recommendations.
|
||||
- A judged run of case 04 on 2026-10-08 scored `pillar_checklists_covered` at 100% on all six pillars, against 9% to 45% before. One run.
|
||||
|
||||
Tests: `tests/execution/test_pillars.py`, "each advisor reviews its pillar's checklist" (3).
|
||||
13
.agents/plans/iter-04-learning-and-skills/steps/step-27.md
Normal file
13
.agents/plans/iter-04-learning-and-skills/steps/step-27.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# Step 27 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `specialist_tools` takes `confirm_first`. While it is set, `submit_deliverable` checks the submission as usual, keeps the working copy, runs any review, and then refuses: the user has not seen this Deliverable; call `needs_input` with a summary and pass the content as `draft`.
|
||||
2. `specialist_task` sets it when the Execution is not pre-approved and the user has not been shown a draft of the Deliverable. The Orchestrator's turn records a Deliverable in `drafts_shown` when a Specialist's question carries a draft. Once shown, submission is allowed, so a Revision is not asked about again.
|
||||
|
||||
The first version opened the rule once the user had answered any question about the Deliverable. The verification run of 2026-10-08 showed the hole: the target architect ran out of tool calls, the Orchestrator asked the user how to proceed, and that answer let the next attempt submit unseen.
|
||||
3. `CONFIRM_DRAFTS` (default `true`) turns the rule on in the assembly. The engine's own default is off, so an engine built directly, as in tests, behaves as before.
|
||||
|
||||
**No interactive evaluation has run since the second version.** The case 04 check `specialists_ask_before_submitting` is the measure.
|
||||
|
||||
Tests: `tests/execution/test_engine.py`, "a user sees a Deliverable before it is first submitted" (3).
|
||||
@@ -78,7 +78,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 1 | Resolve the open questions | none | open, owner's input |
|
||||
| 2 | Decider interface and Jev client with limits | none | built on the TypeSafe SDK; unverified against the live service |
|
||||
| 3 | Pillars a Revision touches | J3 | planned |
|
||||
| 4 | Semantic checks at submission | J4 | a coverage review of the Target architecture built: one choice per listed condition; passes a live test on case 02's graph; off until a full case has run with it (`steps/step-4.md`); other checks planned |
|
||||
| 4 | Semantic checks at submission | J4 | two reviews built, both off until a full case has run with them: coverage of the Target architecture, one choice per condition; and requirements name no products (`steps/step-4.md`) |
|
||||
| 5 | Finding-to-Deliverable routing | J2 | planned |
|
||||
| 6 | Model tier per delegation | J5 | planned |
|
||||
| 7 | Context selection | J6 | planned |
|
||||
@@ -86,11 +86,11 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 9 | In-run context compaction | J10 | planned |
|
||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults, both fixed the same day (`steps/step-12.md`): a reply that echoes the question was read as a change request, and the router looped at the Revision limit. The rerun did not finish; it showed both fixes working and a third fault: a reply restating a requirement reopens `requirements` |
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge built; calibration against human labels open |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned |
|
||||
| 19 | Online evaluation of live Executions | none | planned; needs step 17 |
|
||||
|
||||
@@ -49,3 +49,30 @@ Tests: `tests/execution/test_router.py`, "the two faults of evaluation case 04"
|
||||
|
||||
Rerun on 2026-10-08 (unfinished, 86 minutes): changes 1 and 2 held. A third fault appeared and is not fixed: a reply that restates a requirement, given in answer to a draft-confirmation question, is read as a request to change `requirements`. See the run record in `../../iter-06-target-architecture-graph/steps/step-11.md`.
|
||||
|
||||
## Finished on 2026-10-08: the third case 04 fault, and intake
|
||||
|
||||
**A reply that restates what is agreed is an answer.** `_read_message` now shows the decider the agreed `requirements.json` and asks one more noul, `already_agreed`: does the message only confirm or repeat what those requirements say? At 0.75 or more the reply is closed as an answer, whatever `asks_change` says. Measured against live Jev with requirements holding the facts being restated:
|
||||
|
||||
| Reply | asks_change | already_agreed | Disposition |
|
||||
|---|---|---|---|
|
||||
| Restates the GDPR and secrets requirement | 0.12 | 0.78 | close |
|
||||
| Restates the availability requirement | 0.26 | 0.48 | queue (asked once, then let go) |
|
||||
| Restates the retention requirement | 0.52 | 0.81 | close |
|
||||
| "Approved. Please proceed." | 0.04 | 0.70 | close |
|
||||
| Raises availability to 99.99% | 0.85 | 0.01 | act |
|
||||
| Changes retention from 2 years to 7 | 0.89 | 0.01 | act |
|
||||
| Adds SMS notifications and a mobile app | 0.85 | 0.01 | act |
|
||||
| Changes secret rotation to 30 days | 0.87 | 0.02 | act |
|
||||
|
||||
**Intake: a message that starts something is read first.** `app/nodes/intake.py` asks one choice of a message that arrives with no Execution in progress: new request, change request, or question. The A2A server (`app/a2a_server.py`) acts on it:
|
||||
|
||||
- a question, at 0.75 confidence or more, is answered with a short statement of what the agent does and starts nothing;
|
||||
- a change request, when the Customer case has a completed Execution, is started as a new request with a note saying Change requests are not available yet (iteration 7 builds them);
|
||||
- anything else, any reading that is not confident, and every pre-approved task starts an Execution as before.
|
||||
|
||||
Measured against live Jev on ten messages: two new requests, two questions, two change requests with an earlier Execution, a question about earlier work, a different new request for the same case, a change with nothing delivered (taken as new), and "hello" (0.53, taken as new). All ten as intended.
|
||||
|
||||
Not built: acting on a change request (iteration 7 step 2), and answering a question about earlier work from that work.
|
||||
|
||||
Tests: `tests/execution/test_intake.py` (4), `tests/a2a/test_server.py` "intake" (4), `tests/execution/test_router.py` (1).
|
||||
|
||||
|
||||
20
.agents/plans/iter-05-jev-decisions/steps/step-16.md
Normal file
20
.agents/plans/iter-05-jev-decisions/steps/step-16.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Step 16 — Implementation Detail
|
||||
|
||||
The judge (`eval/judge.py`: `DeciderJudge` for Jev, `GeminiJudge` for a language model) was built earlier. Built on 2026-10-08: the calibration.
|
||||
|
||||
1. `eval/datasets/judge/` holds two saved design recommendations (cases 02 and 03, from runs that passed) and `labels.json`: 32 questions about them, 12 checklist items and the 4 quality-rubric questions for each document, with `label` empty.
|
||||
2. `eval/calibrate_judge.py` asks both judges every question several times and reports, per judge, stability (the same answer to the same question across runs) and agreement with the labels, and agreement between the two judges. `--build` regenerates the question set. A new Jev client is used for each run, because the client caches identical requests and a cached answer would make stability look perfect.
|
||||
|
||||
Measured on 2026-10-08, three runs, no labels yet (`eval/datasets/judge/calibration.json`):
|
||||
|
||||
| | Jev | Gemini (the bundle's default model) |
|
||||
|---|---|---|
|
||||
| Stability | 1.00 | 0.92 |
|
||||
| Mean answer | 0.37 | 0.32 |
|
||||
| Agreement between the two | 0.91 | |
|
||||
|
||||
So Jev gave the same answer every time, the language-model judge changed its answer on about one question in twelve, and the two agree on nine in ten.
|
||||
|
||||
**Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge.
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).
|
||||
@@ -48,3 +48,14 @@ Turn it on for a full case before relying on it.
|
||||
Not built from the plan's design: the other semantic checks (requirements name no products, every recommendation cites a source, the Deliverable addresses its task).
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py`, with a scripted decider.
|
||||
|
||||
## Added on 2026-10-08
|
||||
|
||||
**What code cannot show is listed apart.** The first full run with the coverage review raised conditions such as an availability percentage, which no Check can verify. A requirement or recommendation now lists under `conditions` only what infrastructure configuration can show, and under `outside_code` what it cannot: an outcome measured in operation, behaviour of the application's own code, a process. `check_requirements` and `check_recommendations` require at least one entry between the two. `chain_gaps` asks for no Check from a constraint with `"conditions": []` and something under `outside_code`. The target architect writes no Check for `outside_code`; the validator proposes nothing for it and names it as something to verify after deployment.
|
||||
|
||||
Measured before the split, on the designs of two runs that passed validation with nothing proposed, using the conditions the agents themselves wrote: the coverage review raised 14 of 66 (case 02) and 8 of 51 (case 03). About half were of the kind the split removes (availability, recovery time, throughput, latency, what the application logs). The rest look like real gaps the validator did not report (three zones, cost labels, a load balancer). It has not been measured since the split, which needs new runs.
|
||||
|
||||
**Requirements name no products.** `RequirementsReview` in `app/tools/requirements_review.py` asks one noul per requirement: does it name a cloud provider's product or managed service? It is a third review on the same seam, behind `JEV_REQUIREMENTS_REVIEW` (off), with its own threshold, `JEV_REQUIREMENTS_THRESHOLD`, 0.3. Measured against live Jev: across the 61 requirements of three passed runs the highest answer was 0.10, so nothing was raised; requirements written to name a product scored 0.93 (Cloud Run), 0.61 (Secret Manager) and 0.34 (a Pub/Sub topic), and a database engine or a region scored 0.10 or less. At 0.3 that is three of three caught and none of 65 raised wrongly. Three written examples is a small sample.
|
||||
|
||||
Tests: `tests/execution/test_coverage_review.py` (2 more), `tests/execution/test_requirements_chain.py` (2 more).
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 6, the Target architecture graph
|
||||
**Intent:** Establish the Target architecture as one explicit graph of Elements, Relationships and Checks before any infrastructure code is written, and derive and check every later Deliverable from it, so that the outcome is understood before code exists and validation has a fixed bar.
|
||||
**Status:** steps 1 to 10 built; cases 01, 02 and 03 completed at 100% with real models on 2026-10-07 and 08 (`steps/step-11.md`); case 04, the interactive one, does not complete
|
||||
**Status:** complete. Steps 1 to 10 built; all four cases, the interactive one included, completed twice with real models on 2026-10-08 and the thresholds are set (`steps/step-11.md`)
|
||||
**This directory covers:** steps 1 to 11
|
||||
|
||||
## Order of work
|
||||
@@ -56,7 +56,7 @@ flowchart TD
|
||||
| 8 | Deployment guidance implements the graph | M | complete |
|
||||
| 9 | Validation gives a verdict per Check | M | complete |
|
||||
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
||||
| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) scored 75% once and its rerun did not finish: scripted answers are read as new requirements (`steps/step-11.md`); thresholds outstanding |
|
||||
| 11 | Evaluation: prove the loop converges | M | complete: all four cases pass, each run twice on 2026-10-08; thresholds set (`steps/step-11.md`) |
|
||||
|
||||
Size: S under a day, M a few days.
|
||||
|
||||
|
||||
@@ -162,3 +162,51 @@ Not yet changed:
|
||||
|
||||
Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one.
|
||||
|
||||
## Changes to the evaluation after the unfinished run (2026-10-08)
|
||||
|
||||
- A scripted answer may name who it is for: `[pattern, answer, asker]`. Case 04's fourteen facts are tagged for `requirements`; any other Specialist, and the Orchestrator, gets "Approved. Please proceed." (`automatic_reply`, `eval/datasets/solution_cases.json`).
|
||||
- The evaluation writes `<case>-progress.json` in the run folder after every agent run: seconds, runs, outcomes, tokens and the timeline so far. A run that is cut off leaves it behind with `"finished": false`.
|
||||
|
||||
## Runs of 2026-10-08, 14:25 UTC: all four cases, twice (`eval/results/20261008T142509Z/` and `...142514Z/`)
|
||||
|
||||
Two batches run side by side on the build with the case 04 changes, intake and the `outside_code` split (uncommitted at the time). Batch B added the judged checks. All eight runs completed; none ended with warnings.
|
||||
|
||||
| Case | Batch A | Batch B (judged) |
|
||||
|---|---|---|
|
||||
| 01 | 1.00, 31 min, 3.42 M tokens, 1 Revision | 0.92, 39 min, 4.06 M, 1 Revision |
|
||||
| 02 | 1.00, 47 min, 7.02 M, 3 Revisions | 0.92, 24 min, 3.12 M, none |
|
||||
| 03 | 1.00, 41 min, 6.33 M, 2 Revisions | 0.92, 29 min, 4.06 M, 1 Revision |
|
||||
| 04 | 0.92, 45 min, 5.88 M, 1 Revision | 0.93, 33 min, 3.63 M, 1 Revision |
|
||||
|
||||
Estimated cost: $22.16 for batch A and $16.40 for batch B. Running two batches at once shares the machine and the model quota, so these times are longer than a case run alone.
|
||||
|
||||
- **Case 04 completes.** Both runs went through all five Phases with every draft confirmed by the user and no reply misread as a change. One Revision each: the code for a failed Check in batch B, the design for a proposed addition in batch A.
|
||||
- **Every structural check passed in all eight runs**, with no model timeout, no loop and no `terraform validate` failure.
|
||||
- **`pillar_checklists_covered` failed in all four judged runs**, at 0% to 55% a pillar, while `recommendations_quality` passed at 0.83 to 0.90. The cause is structural: 78 checklist items against 11 to 14 recommendations. Iteration 4 step 26 was built in answer; no judged run has happened since.
|
||||
- **`specialists_ask_before_submitting` failed once**, in batch A's case 04: the target architect exhausted its 45 tool calls, and on its second attempt submitted without asking. The first submission with a user present is now refused until the draft has been put to them (`CONFIRM_DRAFTS`, on by default).
|
||||
- **The same case varies widely.** Case 02 took 24 minutes with no Revision in one batch and 47 minutes with three in the other, at the same time on the same build. Revisions of the Target architecture for additions validation proposes remain the main cost.
|
||||
|
||||
### Thresholds
|
||||
|
||||
Set in `eval/datasets/solution_cases.json` from every completed run of each case on this build, at about a quarter above the worst seen:
|
||||
|
||||
| Case | Revisions | Seconds | Tokens |
|
||||
|---|---|---|---|
|
||||
| 01 | 2 | 3,000 | 5.0 M |
|
||||
| 02 | 3 | 3,600 | 8.5 M |
|
||||
| 03 | 3 | 3,200 | 8.0 M |
|
||||
| 04 | 2 | 3,400 | 7.5 M |
|
||||
|
||||
They are alarms for a run that is clearly worse than any seen, not targets. With two to four runs a case and this much spread they are loose; tighten them as runs accumulate.
|
||||
|
||||
Acceptance criteria: met. All four cases complete and score at least 0.8; deployment guidance needed at most one Revision in every run; the thresholds are set. One criterion is met only loosely: the set of gaps did not always shrink from one validation to the next (case 02, 2026-10-07).
|
||||
|
||||
## Run of 2026-10-08, 17:20 UTC (`eval/results/20261008T172049Z/`): case 04, judged
|
||||
|
||||
To verify two changes made after the batches began: the pillar checklist review (iteration 4 step 26) and the confirm-first rule (step 27). Completed, 86% (12 of 14 checks), no Revision, 36 minutes, 4.10 million tokens.
|
||||
|
||||
- **`pillar_checklists_covered` passes: 100% on all six pillars**, against 9% to 45% in batch B's case 04. The review is in the Deliverable.
|
||||
- `recommendations_quality` passed at 0.80 (specific 0.84, actionable 0.81, trade-offs 0.90, traceable 0.66).
|
||||
- **`specialists_ask_before_submitting` failed again**, for the Target architecture. The rule opened once the user had answered any question about the Deliverable; the target architect ran out of tool calls, the Orchestrator asked the user what to do, and that answer opened it. Fixed after the run: the rule now opens only when the user has been shown a draft of that Deliverable (`drafts_shown`). Not rerun.
|
||||
- **`requirements_name_no_products` failed** for the first time in any run: the requirements named Secret Manager. This is what the requirements review of iteration 5 step 4 looks for; it is off by default and was off here.
|
||||
|
||||
|
||||
@@ -39,6 +39,8 @@ TF_PLUGIN_CACHE_DIR=.cache/terraform-plugins
|
||||
ORCHESTRATOR=router # router | model
|
||||
ORCHESTRATOR_BUDGET=40
|
||||
SPECIALIST_BUDGET=15
|
||||
# With a user present, a Deliverable is shown to them before it is first submitted.
|
||||
CONFIRM_DRAFTS=true
|
||||
MAX_REVISIONS=3
|
||||
MAX_ORCHESTRATOR_TURNS=100
|
||||
MAX_PARALLEL_SPECIALISTS=4
|
||||
@@ -63,6 +65,10 @@ JEV_CONFIDENCE_THRESHOLD=0.6
|
||||
JEV_COVERAGE_REVIEW=false
|
||||
# How likely "no Check verifies this" must be before the condition is raised.
|
||||
JEV_COVERAGE_THRESHOLD=0.6
|
||||
# Review the requirements with Jev before they are submitted: does any name a cloud
|
||||
# product? Off until a full case has run with it.
|
||||
JEV_REQUIREMENTS_REVIEW=false
|
||||
JEV_REQUIREMENTS_THRESHOLD=0.3
|
||||
|
||||
# PostgreSQL (Execution records, and checkpoints when CHECKPOINTER=postgres)
|
||||
POSTGRES_HOST=localhost
|
||||
|
||||
9
.github/workflows/ci.yml
vendored
9
.github/workflows/ci.yml
vendored
@@ -18,8 +18,13 @@ jobs:
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- name: Install quality tools
|
||||
run: python -m pip install --disable-pip-version-check ruff==0.6.9 mypy==1.11.2 pytest==8.3.3
|
||||
cache: pip
|
||||
cache-dependency-path: |
|
||||
requirements.txt
|
||||
requirements-dev.txt
|
||||
# The tests import the application, so its dependencies are installed too.
|
||||
- name: Install dependencies
|
||||
run: python -m pip install --disable-pip-version-check -r requirements.txt -r requirements-dev.txt
|
||||
- name: Lint
|
||||
run: ruff check .
|
||||
- name: Type-check
|
||||
|
||||
@@ -10,6 +10,6 @@ You produce the **design recommendations** Deliverable: the Skill's task for rec
|
||||
- When `feedback` is not empty you are revising: pass `pillars` naming only the pillars the feedback touches. The guidance you were given before is returned for the others.
|
||||
- Reconcile the pillars' guidance into recommendations for each component. Where pillars conflict, name the trade-off and recommend a resolution tied to the requirements.
|
||||
- When an advisor's guidance is missing, derive that pillar's recommendations from the best-practices references in the solution-architecture Skill.
|
||||
- Give every recommendation an id and attach them as `recommendations.json`, in the format the requirements-chain Skill gives. Write one configurable decision per recommendation, so each can become a Check in the Target architecture. Give each its `conditions`: one entry for each separate thing it asks to be configured, and for each component it must hold on when it names several.
|
||||
- Give every recommendation an id and attach them as `recommendations.json`, in the format the requirements-chain Skill gives. Write one configurable decision per recommendation, so each can become a Check in the Target architecture. Give each its `conditions`: one entry for each separate thing it asks to be configured, and for each component it must hold on when it names several. Put under `outside_code` anything it asks of the application's own code or of people, which no infrastructure configuration can show.
|
||||
|
||||
Your contract's sections are one per pillar.
|
||||
Your contract's sections are one per pillar. The advisors' review of each pillar's checklist is added to your Deliverable for you when you submit, so do not write it yourself.
|
||||
|
||||
@@ -11,4 +11,6 @@ Your task message carries the design advisor's brief in `brief` and the pillar's
|
||||
3. Write prioritised, actionable recommendations for each relevant component, naming the products the Skill lists for each gap and citing its guidance.
|
||||
4. List any assessment questions from the Skill that the brief leaves unanswered, as open questions; don't ask the user.
|
||||
|
||||
Call `submit_advice` with your recommendations as markdown. You are done when it accepts them.
|
||||
5. Review the Skill's validation checklist item by item, when it has one. For every item give its `status`: `addressed` when your recommendations cover it, `not_applicable` when it does not apply to this workload or is not something an architecture can settle (an organisation's process, for example), or `not_addressed` when it applies and you recommend nothing for it. In `note`, say in a few words how it is addressed or why it does not apply. Copy each item's text as the Skill has it, shortened to its first sentence.
|
||||
|
||||
Call `submit_advice` with your recommendations as markdown in `content` and the review in `checklist`. You are done when it accepts them.
|
||||
|
||||
@@ -13,6 +13,7 @@ You produce the **requirements** Deliverable: the Skill's requirements-discovery
|
||||
- When the user runs workloads in a cloud environment today, use your cloud discovery tools to describe the current state from what actually exists, and record anything you couldn't read as a discovery gap.
|
||||
- Name no products, services or architectures. This Deliverable describes the problem, not the solution.
|
||||
- Give every requirement an id and attach them all as `requirements.json`, in the format the requirements-chain Skill gives. Use the same ids in your content.
|
||||
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
|
||||
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met and that the configuration of infrastructure can show. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
|
||||
- Put under `outside_code` what the requirement asks for that no infrastructure configuration can show: an outcome measured in operation (an availability percentage, a response time), behaviour of the application's own code, or a process people follow. Those are recorded and reported, and are not checked against the infrastructure code.
|
||||
|
||||
Add `Open questions` and `Assumptions` headings when they apply.
|
||||
|
||||
@@ -11,7 +11,7 @@ You produce the **Target architecture** Deliverable: the solution as one explici
|
||||
- Build the graph with `graph_put`, a few items per call: the Elements first, then the Relationships, then the Checks one Element at a time. Short calls are what keep your replies from timing out.
|
||||
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
|
||||
- Give every connection between Elements a Relationship.
|
||||
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have. One Check per requirement is rarely enough: write a Check for every entry in a requirement's or recommendation's `conditions`, on every Element that condition applies to. A condition left without a Check is found by validation and sent back to you.
|
||||
- Turn every non-functional requirement and every design recommendation into Checks on the Elements and Relationships they constrain. Write each Check so that it can be verified from infrastructure code alone, naming the setting and the value it must have. One Check per requirement is rarely enough: write a Check for every entry in a requirement's or recommendation's `conditions`, on every Element that condition applies to. Write no Check for an entry under `outside_code`: it names something infrastructure code cannot show, and a Check for it could never be met. A condition left without a Check is found by validation and sent back to you.
|
||||
- Name only settings the Element's resource type has. When `terraform_schema` is available, call it once for each distinct resource type in the graph, with `resource_type` alone: the reply lists every argument and block that type accepts, and you write that type's Checks from it. Never look settings up one at a time. A setting that one resource type has may not exist on a similar one, and a Check on a setting the resource type lacks cannot be met and is sent back to you. Write each setting's name in backticks, exactly as the schema spells it. When you submit, every setting a Check names is looked up for you on its Element's resource type, and any that is missing is returned to you once.
|
||||
- Decide the implementation detail yourself from the design recommendations and the Cloud provider's documentation, through your documentation tools. The user states intent at a high level; this level of detail is your job.
|
||||
- When a recommendation does not apply to this workload, list it as not applicable with the reason. The user sees that list when they approve the design.
|
||||
|
||||
@@ -10,7 +10,7 @@ You produce the **validation results** Deliverable: the Skill's pre-deployment v
|
||||
- Run the deployment dry-run with your tools: validate the Terraform, and preview it with a plan when you have permission to read live cloud resources. The tools already hold the submitted files, so pass them no files.
|
||||
- Give a verdict on every Check in the Target architecture, from the infrastructure code: pass, fail or not verifiable, each with the evidence you read. The Checks are the whole bar: judge the code against them and nothing else.
|
||||
- Before you fail a Check because the code lacks a setting, confirm with `terraform_schema` that the resource type has that setting, when the tool is available. When it does not, the Check cannot be met by any code: give it not verifiable, with what the schema shows as the evidence. Give the same verdict to a Check that `implementation.json` records as unimplementable.
|
||||
- Review the Target architecture's coverage once, completely. Go through every requirement and every design recommendation in turn; for each, list the conditions its text states and the Elements it applies to, and record a proposed addition for each one no Check verifies, naming the requirement or recommendation it serves. Report every gap in this pass: each later pass that finds one more costs a whole Revision of the design, the code and this validation. A proposed addition is not a failure of the code.
|
||||
- Review the Target architecture's coverage once, completely. Go through every requirement and every design recommendation in turn; for each, list the conditions its text states and the Elements it applies to, and record a proposed addition for each one no Check verifies, naming the requirement or recommendation it serves. Report every gap in this pass: each later pass that finds one more costs a whole Revision of the design, the code and this validation. Propose nothing for what a requirement or recommendation lists under `outside_code`: that is application behaviour, an outcome measured in operation, or a process, and no Check on infrastructure code can verify it. Name those under your analysis heading as things to verify after deployment. A proposed addition is not a failure of the code.
|
||||
- When `feedback` or your earlier verdicts show the Target architecture was revised for your proposed additions, propose only what the revision still leaves uncovered. Never propose what a Check already verifies in other words.
|
||||
- Record your verdicts with `verdict_put`, a few Checks per call, as you reach them: the Checks of one Element, then the next. Record the dry-run's result and each proposed addition the same way. Short calls are what keep your replies from timing out, and what you record is kept if your run is cut short. `verdict_read` shows which Checks still have none.
|
||||
- When `working_copy` shows verdicts recorded, your last run was cut short on this same code: call `verdict_read` and carry on from there.
|
||||
|
||||
@@ -3,6 +3,10 @@
|
||||
A new task starts an Execution; a message on an input-required task answers the
|
||||
Execution's prompt. Approvals and permissions count only as structured data parts,
|
||||
never as free text.
|
||||
|
||||
Before a new task starts anything its message is read (app/nodes/intake.py): a
|
||||
question is answered and starts nothing, and a change to delivered work is started
|
||||
as a new request with a note saying so, until Change requests are built.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -25,6 +29,7 @@ from google.protobuf import json_format
|
||||
|
||||
from app.agent import ExecutionEngine, Interaction
|
||||
from app.nodes.execution_nodes import GUIDE_FILENAME
|
||||
from app.nodes.intake import ABOUT, CHANGE_NOTE, CHANGE_REQUEST, QUESTION, Intake
|
||||
from app.presentation import describe_prompt
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
@@ -34,14 +39,17 @@ STRUCTURED_REPLY_KEYS = ("approval", "permission")
|
||||
|
||||
|
||||
class SolutionArchitectExecutor(AgentExecutor):
|
||||
def __init__(self, engine: ExecutionEngine) -> None:
|
||||
def __init__(self, engine: ExecutionEngine, intake: Intake | None = None) -> None:
|
||||
self._engine = engine
|
||||
self._intake = intake
|
||||
|
||||
async def execute(self, context: RequestContext, event_queue: EventQueue) -> None:
|
||||
task_id = context.task_id or ""
|
||||
updater = TaskUpdater(event_queue, task_id, context.context_id or "")
|
||||
new = context.current_task is None
|
||||
note: str | None = None
|
||||
try:
|
||||
if context.current_task is None:
|
||||
if new:
|
||||
history = [context.message] if context.message is not None else []
|
||||
await event_queue.enqueue_event(
|
||||
new_task(
|
||||
@@ -53,6 +61,27 @@ class SolutionArchitectExecutor(AgentExecutor):
|
||||
)
|
||||
await updater.start_work()
|
||||
options = _options(context)
|
||||
# A pre-approved caller is a batch job: it sends work, not questions.
|
||||
if self._intake and not options.get("pre_approved"):
|
||||
reading = await self._intake.read(
|
||||
context.get_user_input(),
|
||||
options.get("customer_case"),
|
||||
scope=task_id,
|
||||
)
|
||||
if reading.kind == QUESTION:
|
||||
await updater.complete(
|
||||
updater.new_agent_message(
|
||||
[
|
||||
new_text_part(ABOUT),
|
||||
new_data_part(
|
||||
{"status": "answered", "intake": QUESTION}
|
||||
),
|
||||
]
|
||||
)
|
||||
)
|
||||
return
|
||||
if reading.kind == CHANGE_REQUEST:
|
||||
note = CHANGE_NOTE
|
||||
interaction = await self._engine.start(
|
||||
context.get_user_input(),
|
||||
session_id=context.context_id,
|
||||
@@ -76,7 +105,7 @@ class SolutionArchitectExecutor(AgentExecutor):
|
||||
)
|
||||
)
|
||||
return
|
||||
await self._publish(interaction, updater)
|
||||
await self._publish(interaction, updater, note if new else None)
|
||||
|
||||
async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None:
|
||||
# The checkpoint stays; cancelling only ends the A2A task.
|
||||
@@ -85,12 +114,15 @@ class SolutionArchitectExecutor(AgentExecutor):
|
||||
)
|
||||
await updater.cancel()
|
||||
|
||||
async def _publish(self, interaction: Interaction, updater: TaskUpdater) -> None:
|
||||
async def _publish(
|
||||
self, interaction: Interaction, updater: TaskUpdater, note: str | None = None
|
||||
) -> None:
|
||||
if interaction.status == "awaiting_user" and interaction.prompt is not None:
|
||||
prompt = dict(interaction.prompt)
|
||||
await updater.requires_input(
|
||||
updater.new_agent_message(
|
||||
[
|
||||
*([new_text_part(note)] if note else []),
|
||||
new_text_part(describe_prompt(prompt)),
|
||||
new_data_part({"prompt": prompt}),
|
||||
]
|
||||
|
||||
33
app/agent.py
33
app/agent.py
@@ -35,6 +35,7 @@ from app.adk.runtime import (
|
||||
from app.checkpointing import open_checkpointer
|
||||
from app.config import Settings
|
||||
from app.customer_case import record_intent
|
||||
from app.nodes.intake import Intake
|
||||
from app.nodes.router import (
|
||||
REPLY_WRITER,
|
||||
OrchestratorPolicy,
|
||||
@@ -52,6 +53,7 @@ from app.tools.execution import ORCHESTRATOR
|
||||
from app.tools.jev import JevDecider, JevLimits
|
||||
from app.tools.local import TerraformRunner, local_tools
|
||||
from app.tools.mcp import McpCatalog, auth_from_config
|
||||
from app.tools.requirements_review import RequirementsReview
|
||||
from app.tools.schema_review import Reviews, SchemaReview
|
||||
from app.tools.toolkit import make_tool_factory
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
||||
@@ -88,6 +90,7 @@ class ExecutionEngine:
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
submission_review: SubmissionReview | None = None,
|
||||
confirm_drafts: bool = False,
|
||||
) -> None:
|
||||
required = {
|
||||
ORCHESTRATOR,
|
||||
@@ -109,6 +112,7 @@ class ExecutionEngine:
|
||||
skill_versions=skill_versions,
|
||||
orchestrator=orchestrator,
|
||||
submission_review=submission_review,
|
||||
confirm_drafts=confirm_drafts,
|
||||
).compile(checkpointer=checkpointer)
|
||||
|
||||
async def start(
|
||||
@@ -120,9 +124,9 @@ class ExecutionEngine:
|
||||
pre_approved: bool = False,
|
||||
allow_cloud_read: bool | None = None,
|
||||
execution_id: str | None = None,
|
||||
scripted_answers: Sequence[tuple[str, str]] = (),
|
||||
scripted_answers: Sequence[Sequence[str]] = (),
|
||||
) -> Interaction:
|
||||
"""Start an Execution. `scripted_answers` are (regex, answer) pairs a
|
||||
"""Start an Execution. `scripted_answers` are (regex, answer[, asker]) entries a
|
||||
pre-approved Execution uses to answer questions, for evaluation runs."""
|
||||
customer_case = check_customer_case(customer_case)
|
||||
execution_id = execution_id or str(uuid.uuid4())
|
||||
@@ -156,6 +160,7 @@ class ExecutionEngine:
|
||||
"awaiting_questions": [],
|
||||
"answers": {},
|
||||
"drafts": {},
|
||||
"drafts_shown": [],
|
||||
"working": {},
|
||||
"pending_permission": None,
|
||||
"prompt": None,
|
||||
@@ -175,9 +180,7 @@ class ExecutionEngine:
|
||||
"turns": 0,
|
||||
"status": "running",
|
||||
"warnings": [],
|
||||
"scripted_answers": [
|
||||
[pattern, answer] for pattern, answer in scripted_answers
|
||||
],
|
||||
"scripted_answers": [list(entry) for entry in scripted_answers],
|
||||
}
|
||||
await self._graph.ainvoke(initial, self._config(execution_id))
|
||||
return await self.interaction(execution_id)
|
||||
@@ -236,6 +239,7 @@ class Assembly:
|
||||
roles: dict[str, RoleSpec]
|
||||
mcp_gaps: dict[str, str]
|
||||
decider: Decider = field(default_factory=NoDecider)
|
||||
intake: Intake | None = None # reads a message that starts something
|
||||
|
||||
|
||||
def open_decider(settings: Settings) -> Decider:
|
||||
@@ -334,6 +338,11 @@ async def open_assembly(
|
||||
if bundle.terraform_providers:
|
||||
# In code: every setting a Check names exists on its resource type.
|
||||
reviews.append(SchemaReview(runner))
|
||||
if settings.JEV_REQUIREMENTS_REVIEW and isinstance(decider, JevDecider):
|
||||
# Optional, and off by default: Jev says whether a requirement names a product.
|
||||
reviews.append(
|
||||
RequirementsReview(decider, act=settings.JEV_REQUIREMENTS_THRESHOLD)
|
||||
)
|
||||
if settings.JEV_COVERAGE_REVIEW and isinstance(decider, JevDecider):
|
||||
# Optional, and off by default: Jev says which Check verifies each condition.
|
||||
reviews.append(CoverageReview(decider, none_at=settings.JEV_COVERAGE_THRESHOLD))
|
||||
@@ -354,9 +363,21 @@ async def open_assembly(
|
||||
},
|
||||
orchestrator=orchestrator,
|
||||
submission_review=Reviews(reviews) if reviews else None,
|
||||
confirm_drafts=settings.CONFIRM_DRAFTS,
|
||||
)
|
||||
try:
|
||||
yield Assembly(engine, bundle, roles, dict(catalog.gaps), decider)
|
||||
yield Assembly(
|
||||
engine,
|
||||
bundle,
|
||||
roles,
|
||||
dict(catalog.gaps),
|
||||
decider,
|
||||
Intake(
|
||||
decider,
|
||||
output_root or settings.BASE_DIR,
|
||||
act=settings.JEV_ACT_THRESHOLD,
|
||||
),
|
||||
)
|
||||
finally:
|
||||
if isinstance(decider, JevDecider):
|
||||
await decider.aclose()
|
||||
|
||||
@@ -54,6 +54,9 @@ class Settings(BaseSettings):
|
||||
A2A_PUBLIC_URL: str | None = None # advertised in the agent card
|
||||
ORCHESTRATOR_BUDGET: int = 40
|
||||
SPECIALIST_BUDGET: int = 15
|
||||
# With a user present, a Specialist's first submission of a Deliverable is refused
|
||||
# until it has put its draft to them. Off: Specialists are only told to.
|
||||
CONFIRM_DRAFTS: bool = True
|
||||
MAX_REVISIONS: int = 3
|
||||
MAX_ORCHESTRATOR_TURNS: int = 100
|
||||
# How an Orchestrator turn is decided (ADR-0007): "router" reads the Execution
|
||||
@@ -87,6 +90,10 @@ class Settings(BaseSettings):
|
||||
JEV_COVERAGE_REVIEW: bool = False
|
||||
# How likely "no Check verifies this" must be before the condition is raised.
|
||||
JEV_COVERAGE_THRESHOLD: float = 0.6
|
||||
# Review the requirements at submission: does any name a cloud product? Off
|
||||
# until a full case has run with it.
|
||||
JEV_REQUIREMENTS_REVIEW: bool = False
|
||||
JEV_REQUIREMENTS_THRESHOLD: float = 0.3
|
||||
|
||||
# Paths
|
||||
BASE_DIR: Path = Path(__file__).resolve().parent.parent
|
||||
|
||||
@@ -85,7 +85,9 @@ def create_app(
|
||||
async def lifespan(app: Starlette) -> AsyncIterator[None]:
|
||||
async with assembly_factory(settings) as assembly:
|
||||
handler = DefaultRequestHandlerV2(
|
||||
agent_executor=SolutionArchitectExecutor(assembly.engine),
|
||||
agent_executor=SolutionArchitectExecutor(
|
||||
assembly.engine, assembly.intake
|
||||
),
|
||||
task_store=InMemoryTaskStore(),
|
||||
agent_card=card,
|
||||
)
|
||||
|
||||
@@ -105,6 +105,7 @@ def make_execution_nodes(
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
submission_review: SubmissionReview | None = None,
|
||||
confirm_drafts: bool = False,
|
||||
) -> ExecutionNodes:
|
||||
tool_factory = tool_factory or _no_tools
|
||||
# How an Orchestrator turn is decided. The default drives the tools from the
|
||||
@@ -180,6 +181,7 @@ def make_execution_nodes(
|
||||
notes = list(state.get("notes", []))
|
||||
pending_permission = state.get("pending_permission")
|
||||
warnings = list(state.get("warnings", []))
|
||||
shown = list(state.get("drafts_shown", []))
|
||||
drafts = dict(state.get("drafts", {}))
|
||||
|
||||
outcomes = state.get("outcomes", [])
|
||||
@@ -217,6 +219,9 @@ def make_execution_nodes(
|
||||
"content": record["draft"],
|
||||
"attachments": record.get("draft_attachments", {}),
|
||||
}
|
||||
# The user is being shown this Deliverable.
|
||||
if kind.value not in shown:
|
||||
shown.append(kind.value)
|
||||
questions = record.get("questions") or [
|
||||
(
|
||||
f"The {record['role']} stopped: {record['summary']} "
|
||||
@@ -229,6 +234,7 @@ def make_execution_nodes(
|
||||
update: dict[str, Any] = {
|
||||
"deliverables": deliverables,
|
||||
"drafts": drafts,
|
||||
"drafts_shown": shown,
|
||||
"pending_questions": pending_questions,
|
||||
"pending_permission": pending_permission,
|
||||
"outcomes_applied": len(outcomes),
|
||||
@@ -434,6 +440,11 @@ def make_execution_nodes(
|
||||
working=working,
|
||||
library=task_input.get("library"),
|
||||
review=review if submission_review else None,
|
||||
# With a user present, their first sight of a Deliverable comes
|
||||
# before it is submitted, whatever the Specialist intends.
|
||||
confirm_first=confirm_drafts
|
||||
and not task_input["context"].get("pre_approved", True)
|
||||
and not task_input.get("shown"),
|
||||
),
|
||||
refusals,
|
||||
role.name,
|
||||
@@ -559,7 +570,7 @@ def make_execution_nodes(
|
||||
|
||||
async def finalize(state: ExecutionState) -> dict[str, Any]:
|
||||
ledger = Ledger.from_dict(state["ledger"])
|
||||
_write_outputs(output_root, state)
|
||||
_write_outputs(output_root, state, "completed", ledger)
|
||||
close(state, "completed", has_files=True)
|
||||
await record_progress(state, ledger, "completed")
|
||||
await record_lessons(state, ledger, "completed")
|
||||
@@ -567,7 +578,10 @@ def make_execution_nodes(
|
||||
|
||||
async def end_with_warnings(state: ExecutionState) -> dict[str, Any]:
|
||||
ledger = Ledger.from_dict(state["ledger"])
|
||||
close(state, "completed_with_warnings", has_files=False)
|
||||
# What was produced is kept, marked as unfinished: a person deciding what to
|
||||
# do next needs to read it, and it is the evidence of what went wrong.
|
||||
wrote = _write_outputs(output_root, state, "completed_with_warnings", ledger)
|
||||
close(state, "completed_with_warnings", has_files=wrote)
|
||||
await record_progress(state, ledger, "completed_with_warnings")
|
||||
await record_lessons(state, ledger, "completed_with_warnings")
|
||||
return {"status": "completed_with_warnings"}
|
||||
@@ -706,6 +720,9 @@ def _specialist_input(
|
||||
"attempt": attempt,
|
||||
"allow_cloud_read": state.get("allow_cloud_read"),
|
||||
"working": dict(state.get("working", {}).get(kind.value, {})),
|
||||
# Whether the user has been shown a draft of this Deliverable. Answering some
|
||||
# other question about it, such as what to do after it stopped, is not that.
|
||||
"shown": kind.value in state.get("drafts_shown", []),
|
||||
"context": {
|
||||
"workflow_request": state["workflow_request"],
|
||||
"pre_approved": bool(state.get("pre_approved")),
|
||||
@@ -762,7 +779,14 @@ def automatic_reply(
|
||||
prompt: Mapping[str, Any], scripted_answers: Sequence[Sequence[str]] = ()
|
||||
) -> dict[str, Any]:
|
||||
"""The reply in a pre-approved Execution: approve, deny cloud reads, and answer
|
||||
questions from scripted (pattern, answer) pairs where one matches."""
|
||||
questions from scripted answers where one matches.
|
||||
|
||||
A scripted answer is (pattern, answer) or (pattern, answer, asker). It answers a
|
||||
question its pattern matches; with an asker, only when that pattern also matches
|
||||
who is asking (a Deliverable kind, or the orchestrator). The first that applies
|
||||
wins. So facts written for requirements discovery are not given to a later
|
||||
Specialist that only wants its draft confirmed.
|
||||
"""
|
||||
if "approval" in prompt:
|
||||
return {
|
||||
"approval": {"phase": prompt["approval"]["phase"], "decision": "approve"}
|
||||
@@ -772,11 +796,13 @@ def automatic_reply(
|
||||
lines = []
|
||||
unanswered = False
|
||||
for item in prompt.get("questions", []):
|
||||
asker = str(item.get("from", ""))
|
||||
answer = next(
|
||||
(
|
||||
scripted
|
||||
for pattern, scripted in scripted_answers
|
||||
if re.search(pattern, item["question"], re.IGNORECASE)
|
||||
scripted[1]
|
||||
for scripted in scripted_answers
|
||||
if re.search(scripted[0], item["question"], re.IGNORECASE)
|
||||
and (len(scripted) < 3 or re.search(scripted[2], asker, re.IGNORECASE))
|
||||
),
|
||||
None,
|
||||
)
|
||||
@@ -839,13 +865,21 @@ def _case(state: ExecutionState) -> str:
|
||||
return state.get("customer_case") or UNASSIGNED_CASE
|
||||
|
||||
|
||||
def _write_outputs(output_root: Path, state: ExecutionState) -> None:
|
||||
"""Write a finished Execution's files under its Customer case, with a manifest
|
||||
that ties the request to what was generated for it."""
|
||||
def _write_outputs(
|
||||
output_root: Path, state: ExecutionState, status: str, ledger: Ledger
|
||||
) -> bool:
|
||||
"""Write an Execution's files under its Customer case, with a manifest that ties
|
||||
the request to what was generated for it. Returns whether anything was written.
|
||||
|
||||
An Execution that ended with warnings is written too. Its manifest carries the
|
||||
warnings and each Deliverable's status, so nothing unapproved reads as final.
|
||||
"""
|
||||
customer_case = _case(state)
|
||||
execution_id = state["execution_id"]
|
||||
out_dir = execution_dir(output_root, execution_id, customer_case)
|
||||
deliverables: Mapping[str, Mapping[str, Any]] = state.get("deliverables", {})
|
||||
if not deliverables:
|
||||
return False
|
||||
files: dict[str, str] = {}
|
||||
by_kind: dict[str, dict[str, Any]] = {}
|
||||
guide = deliverables.get(DeliverableKind.SOLUTION_GUIDE.value)
|
||||
@@ -856,7 +890,12 @@ def _write_outputs(output_root: Path, state: ExecutionState) -> None:
|
||||
files[document] = deliverable["content"]
|
||||
attachments = dict(deliverable.get("attachments", {}))
|
||||
files.update(attachments)
|
||||
by_kind[kind] = {"document": document, "attachments": sorted(attachments)}
|
||||
by_kind[kind] = {
|
||||
"document": document,
|
||||
"attachments": sorted(attachments),
|
||||
# submitted, or back in draft after something it depends on was reopened
|
||||
"status": ledger.status(DeliverableKind(kind)).value,
|
||||
}
|
||||
for name, content in files.items():
|
||||
path = out_dir / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
@@ -868,7 +907,11 @@ def _write_outputs(output_root: Path, state: ExecutionState) -> None:
|
||||
"session_id": state.get("session_id"),
|
||||
"intent_version": state.get("intent_version"),
|
||||
"workflow_request": state["workflow_request"],
|
||||
"status": "completed",
|
||||
"status": status,
|
||||
"warnings": list(state.get("warnings", [])),
|
||||
"phases": {
|
||||
phase.value: ledger.phase_status(phase).value for phase in PHASE_ORDER
|
||||
},
|
||||
"solution_guide": GUIDE_FILENAME if guide else None,
|
||||
"deliverables": by_kind,
|
||||
}
|
||||
@@ -876,3 +919,4 @@ def _write_outputs(output_root: Path, state: ExecutionState) -> None:
|
||||
(out_dir / MANIFEST_FILENAME).write_text(
|
||||
json.dumps(manifest, indent=2, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
return True
|
||||
|
||||
122
app/nodes/intake.py
Normal file
122
app/nodes/intake.py
Normal file
@@ -0,0 +1,122 @@
|
||||
"""Intake: what a message that starts something is asking for (ADR-0007).
|
||||
|
||||
Inside an Execution the router reads the user's replies. This is the same reading
|
||||
one step earlier, for a message that arrives with no Execution in progress:
|
||||
|
||||
new request design this workload start an Execution
|
||||
change request change what was delivered (ADR-0006; not built yet)
|
||||
question what can you do, how long... answer; start nothing
|
||||
|
||||
Without a decision, and whenever the reading is not confident, the message is taken
|
||||
as a new request: that is what every message was before, and starting work nobody
|
||||
asked for is cheaper to put right than refusing work somebody did.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import Decider, Question
|
||||
from app.customer_case import IN_PROGRESS, read_history
|
||||
from app.states.execution_state import UNASSIGNED_CASE
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
NEW_REQUEST = "new_request"
|
||||
CHANGE_REQUEST = "change_request"
|
||||
QUESTION = "question"
|
||||
|
||||
ABOUT = (
|
||||
"I design cloud solution architectures. Tell me, at a high level, what you want "
|
||||
"built, and I will work through requirements, a target architecture, "
|
||||
"infrastructure code, validation and a packaged guide, asking for your approval "
|
||||
"at each Phase. Nothing was started for this message."
|
||||
)
|
||||
CHANGE_NOTE = (
|
||||
"This reads as a change to work already delivered for this Customer case. "
|
||||
"Changing an earlier architecture in place is not available yet, so this is "
|
||||
"being designed as a new request."
|
||||
)
|
||||
|
||||
_MESSAGE_LIMIT = 8_000
|
||||
_EARLIER_SHOWN = 5
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IntakeReading:
|
||||
kind: str # new_request | change_request | question
|
||||
confidence: float = 0.0
|
||||
judged: bool = False # False when no decision was available
|
||||
earlier: int = 0 # finished Executions of the same Customer case
|
||||
|
||||
|
||||
def finished_intents(output_root: Path, customer_case: str) -> list[Mapping[str, Any]]:
|
||||
"""The case's earlier statements of intent whose Executions delivered something."""
|
||||
if not customer_case or customer_case == UNASSIGNED_CASE:
|
||||
return []
|
||||
return [
|
||||
item
|
||||
for item in read_history(output_root, customer_case)["intents"]
|
||||
if isinstance(item, Mapping)
|
||||
and str(item.get("status", IN_PROGRESS)).startswith("completed")
|
||||
]
|
||||
|
||||
|
||||
class Intake:
|
||||
def __init__(
|
||||
self,
|
||||
decider: Decider,
|
||||
output_root: Path,
|
||||
*,
|
||||
act: float = 0.75,
|
||||
) -> None:
|
||||
self._decider = decider
|
||||
self._output_root = output_root
|
||||
# How sure the reading must be before anything but a new Execution is done.
|
||||
self._act = act
|
||||
|
||||
async def read(
|
||||
self, message: str, customer_case: str | None = None, scope: str = ""
|
||||
) -> IntakeReading:
|
||||
earlier = finished_intents(self._output_root, customer_case or "")
|
||||
text = message.strip()
|
||||
if not text:
|
||||
return IntakeReading(NEW_REQUEST, earlier=len(earlier))
|
||||
answers = await self._decider.decide(
|
||||
_state(text, earlier),
|
||||
{
|
||||
"intake": Question(
|
||||
"choice",
|
||||
"What is the user's message? new_request: they describe "
|
||||
"something they want designed or built. change_request: they "
|
||||
"ask to change or extend an architecture already delivered to "
|
||||
"them. question: they ask about the service or about earlier "
|
||||
"work, and want an answer, not new work.",
|
||||
(NEW_REQUEST, CHANGE_REQUEST, QUESTION),
|
||||
)
|
||||
},
|
||||
scope=scope,
|
||||
)
|
||||
answer = (answers or {}).get("intake")
|
||||
if answer is None:
|
||||
return IntakeReading(NEW_REQUEST, earlier=len(earlier))
|
||||
kind, confidence = str(answer.value), float(answer.confidence)
|
||||
if kind != NEW_REQUEST and confidence < self._act:
|
||||
kind = NEW_REQUEST # not sure enough to do anything but start
|
||||
if kind == CHANGE_REQUEST and not earlier:
|
||||
kind = NEW_REQUEST # there is nothing delivered to change
|
||||
logger.info("[%s] intake: %s (%.2f)", scope, kind, confidence)
|
||||
return IntakeReading(kind, confidence, judged=True, earlier=len(earlier))
|
||||
|
||||
|
||||
def _state(message: str, earlier: Sequence[Mapping[str, Any]]) -> str:
|
||||
lines = [f"Architectures already delivered to this user: {len(earlier)}."]
|
||||
for item in list(earlier)[-_EARLIER_SHOWN:]:
|
||||
request = " ".join(str(item.get("workflow_request", "")).split())[:300]
|
||||
lines.append(f"- {request}")
|
||||
lines.append(f"The user's message: {message[:_MESSAGE_LIMIT]}")
|
||||
return "\n".join(lines)
|
||||
@@ -139,6 +139,9 @@ class Readings:
|
||||
judged: dict[str, Any] = field(default_factory=dict) # what the decider answered
|
||||
|
||||
|
||||
# How much of the agreed requirements is shown to the decider with a user's message.
|
||||
_AGREED_LIMIT = 24_000
|
||||
|
||||
# The actions that reopen a Deliverable, and which one.
|
||||
_REOPENS = (
|
||||
"revise_requested",
|
||||
@@ -356,6 +359,15 @@ class Router:
|
||||
# our own question about a change they already asked for.
|
||||
structured = reply.get("kind") == "changes" or bool(pending)
|
||||
|
||||
agreed = (
|
||||
""
|
||||
if structured
|
||||
else str(
|
||||
view.deliverables.get(DeliverableKind.REQUIREMENTS.value, {})
|
||||
.get("attachments", {})
|
||||
.get(REQUIREMENTS_FILE, "")
|
||||
)[:_AGREED_LIMIT]
|
||||
)
|
||||
questions = {
|
||||
"change_target": Question(
|
||||
"choice",
|
||||
@@ -375,9 +387,18 @@ class Router:
|
||||
"Is the user asking for a change to work that has already been "
|
||||
"submitted, as opposed to answering a question or adding information?",
|
||||
)
|
||||
if agreed:
|
||||
# Someone who restates what is already agreed is answering, however
|
||||
# much it sounds like a requirement.
|
||||
questions["already_agreed"] = Question(
|
||||
"noul",
|
||||
"Does the user's message only confirm or repeat what the agreed "
|
||||
"requirements above already say, asking for nothing different?",
|
||||
)
|
||||
state = (
|
||||
f"Submitted Deliverables: {', '.join(kind.value for kind in candidates)}.\n"
|
||||
f"Questions waiting for the user: {len(view.pending_questions)}.\n"
|
||||
+ (f"The agreed requirements:\n{agreed}\n" if agreed else "")
|
||||
+ f"Questions waiting for the user: {len(view.pending_questions)}.\n"
|
||||
# What we asked is background. Only the user's own words are their
|
||||
# message: our question echoed back is not something they said.
|
||||
+ (
|
||||
@@ -412,6 +433,10 @@ class Router:
|
||||
}
|
||||
probability = 1.0 if structured else float(answers["asks_change"].value)
|
||||
disposition = self._thresholds.disposition(probability)
|
||||
restated = answers.get("already_agreed")
|
||||
if restated is not None and restated.is_yes(self._thresholds.act):
|
||||
# It restates what is agreed: an answer, not a change.
|
||||
disposition = CLOSE
|
||||
target_answer = answers["change_target"]
|
||||
target = DeliverableKind(str(target_answer.value))
|
||||
readings.change_target = target
|
||||
|
||||
@@ -67,6 +67,7 @@ class ExecutionState(TypedDict, total=False):
|
||||
pending_questions: list[dict[str, Any]] # {kind, questions} not yet asked
|
||||
awaiting_questions: list[dict[str, Any]] # {kind, questions} asked, not answered
|
||||
answers: dict[str, list[dict[str, Any]]] # kind -> [{questions, reply}]
|
||||
drafts_shown: list[str] # Deliverables a draft of which the user has been shown
|
||||
drafts: dict[
|
||||
str, dict[str, Any]
|
||||
] # kind -> {content, attachments} awaiting the user
|
||||
|
||||
@@ -21,6 +21,7 @@ from app.adk.runtime import (
|
||||
files_from_argument,
|
||||
)
|
||||
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
|
||||
from app.tools.pillars import REVIEW_HEADING, render_checklist_review
|
||||
from app.tools.workspace import (
|
||||
FILES,
|
||||
basis_of,
|
||||
@@ -269,6 +270,7 @@ def specialist_tools(
|
||||
working: MutableMapping[str, Any] | None = None,
|
||||
library: Mapping[str, Mapping[str, Any]] | None = None,
|
||||
review: Callable[[Mapping[str, str]], Awaitable[Sequence[str]]] | None = None,
|
||||
confirm_first: bool = False,
|
||||
) -> list[ToolSpec]:
|
||||
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
|
||||
the cloud-read gate.
|
||||
@@ -279,6 +281,9 @@ def specialist_tools(
|
||||
given to the Specialist as data files alone, to read whole when it needs one.
|
||||
`review` looks at the meaning of a structurally complete submission; what it
|
||||
raises is returned to the Specialist once per run, and never a second time.
|
||||
`confirm_first` is set while a user is present and has not yet been asked about
|
||||
this Deliverable: a submission is then refused until the Specialist has put its
|
||||
draft to them. A rule, because an instruction alone was not always followed.
|
||||
"""
|
||||
working = {} if working is None else working
|
||||
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
||||
@@ -339,6 +344,21 @@ def specialist_tools(
|
||||
"it is already covered. This review runs once, so your next "
|
||||
"submission is accepted."
|
||||
)
|
||||
if confirm_first:
|
||||
return ToolResult(
|
||||
"Not submitted. The user has not seen this Deliverable yet. Call "
|
||||
"`needs_input` with a short summary asking them to approve it or "
|
||||
"request changes, passing this content as `draft` with its "
|
||||
"attachments. What you have recorded is kept. Submit when they "
|
||||
"have replied."
|
||||
)
|
||||
if kind is DeliverableKind.DESIGN_RECOMMENDATIONS:
|
||||
# The pillar advisors' checklist reviews, written from what they
|
||||
# recorded. A Revision replaces the section, never stacks a second.
|
||||
checklist_review = render_checklist_review(working)
|
||||
content = content.split(f"\n## {REVIEW_HEADING}")[0].rstrip() + "\n"
|
||||
if checklist_review:
|
||||
content += "\n" + checklist_review
|
||||
if is_graph:
|
||||
# The tables and the diagram are views of the graph, so the engine
|
||||
# writes them: they cannot disagree with it.
|
||||
|
||||
@@ -3,6 +3,12 @@
|
||||
Pillar advice is working material for the design recommendations Deliverable, not a
|
||||
Deliverable itself, so it never touches the Ledger. It is kept in that Deliverable's
|
||||
working memory, so a Revision re-consults only the pillars its feedback touches.
|
||||
|
||||
Each advisor also returns its review of the pillar's own checklist: for every item,
|
||||
whether its advice addresses it, or why it does not apply. The reviews are kept
|
||||
beside the advice and written into the Deliverable by code when it is submitted, so
|
||||
an architect sees the whole checklist accounted for and no model spends output
|
||||
copying it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -26,6 +32,14 @@ from app.adk.runtime import (
|
||||
CONSULT_TOOL = "consult_pillar_advisors"
|
||||
NO_GUIDANCE = "No guidance"
|
||||
ADVICE_KEY = "pillar_advice" # working memory: pillar -> the advice last given
|
||||
CHECKLIST_KEY = "pillar_checklists" # working memory: pillar -> its checklist review
|
||||
REVIEW_HEADING = "Architecture framework checklist review"
|
||||
STATUSES = ("addressed", "not_applicable", "not_addressed")
|
||||
_STATUS_WORDS = {
|
||||
"addressed": "Addressed",
|
||||
"not_applicable": "Does not apply",
|
||||
"not_addressed": "Not addressed",
|
||||
}
|
||||
|
||||
|
||||
def pillar_consultation_tool(
|
||||
@@ -42,7 +56,7 @@ def pillar_consultation_tool(
|
||||
returns the remembered advice for the rest.
|
||||
"""
|
||||
|
||||
async def advise(pillar: str, brief: str) -> str:
|
||||
async def advise(pillar: str, brief: str) -> tuple[str, list[dict[str, str]]]:
|
||||
async with AsyncExitStack() as stack:
|
||||
if limiter:
|
||||
await stack.enter_async_context(limiter.slot())
|
||||
@@ -71,12 +85,16 @@ def pillar_consultation_tool(
|
||||
remembered: dict[str, str] = (
|
||||
dict(memory.get(ADVICE_KEY, {})) if memory is not None else {}
|
||||
)
|
||||
reviews: dict[str, list[dict[str, str]]] = (
|
||||
dict(memory.get(CHECKLIST_KEY, {})) if memory is not None else {}
|
||||
)
|
||||
sections = []
|
||||
for pillar in pillar_skills:
|
||||
if pillar in fresh:
|
||||
advice = fresh[pillar]
|
||||
advice, review = fresh[pillar]
|
||||
if not advice.startswith(NO_GUIDANCE):
|
||||
remembered[pillar] = advice
|
||||
reviews[pillar] = review
|
||||
sections.append(f"## {pillar}\n\n{advice}")
|
||||
elif pillar in remembered:
|
||||
sections.append(
|
||||
@@ -85,6 +103,7 @@ def pillar_consultation_tool(
|
||||
)
|
||||
if memory is not None:
|
||||
memory[ADVICE_KEY] = remembered
|
||||
memory[CHECKLIST_KEY] = reviews
|
||||
return ToolResult("\n\n".join(sections))
|
||||
|
||||
return ToolSpec(
|
||||
@@ -115,36 +134,116 @@ async def _advise(
|
||||
skill: str,
|
||||
brief: str,
|
||||
extra_tools: Sequence[ToolSpec],
|
||||
) -> str:
|
||||
) -> tuple[str, list[dict[str, str]]]:
|
||||
"""One advisor's advice, and its review of the pillar's checklist."""
|
||||
role = dataclasses.replace(advisor, skills=(skill,))
|
||||
task = AgentTask(
|
||||
role=advisor.name,
|
||||
prompt=f"Advise on the {pillar} pillar for this workload.",
|
||||
context={"pillar": pillar, "brief": brief},
|
||||
)
|
||||
review: list[dict[str, str]] = []
|
||||
try:
|
||||
outcome = await runtime.run(role, task, [_submit_advice_tool(), *extra_tools])
|
||||
outcome = await runtime.run(
|
||||
role, task, [_submit_advice_tool(review), *extra_tools]
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - one failed advisor must not lose the others
|
||||
return f"{NO_GUIDANCE}: the advisor failed ({exc})."
|
||||
return f"{NO_GUIDANCE}: the advisor failed ({exc}).", []
|
||||
if isinstance(outcome, Submitted):
|
||||
return outcome.content.strip()
|
||||
return f"{NO_GUIDANCE}: the advisor ended with {type(outcome).__name__}."
|
||||
return outcome.content.strip(), review
|
||||
return f"{NO_GUIDANCE}: the advisor ended with {type(outcome).__name__}.", []
|
||||
|
||||
|
||||
def _submit_advice_tool() -> ToolSpec:
|
||||
async def submit_advice(content: str) -> ToolResult:
|
||||
def _submit_advice_tool(review: list[dict[str, str]]) -> ToolSpec:
|
||||
"""`review` receives the checklist review the advisor submits with its advice."""
|
||||
|
||||
async def submit_advice(content: str, checklist: Sequence[Any] = ()) -> ToolResult:
|
||||
if not content.strip():
|
||||
return ToolResult("Not submitted: the advice is empty.")
|
||||
review[:] = [
|
||||
{
|
||||
"item": " ".join(str(entry.get("item", "")).split()),
|
||||
"status": str(entry.get("status", "")),
|
||||
"note": " ".join(str(entry.get("note", "")).split()),
|
||||
}
|
||||
for entry in checklist or ()
|
||||
if isinstance(entry, Mapping)
|
||||
and str(entry.get("item", "")).strip()
|
||||
and entry.get("status") in STATUSES
|
||||
]
|
||||
return ToolResult("Advice submitted.", Submitted(content))
|
||||
|
||||
text = {"type": "string"}
|
||||
return ToolSpec(
|
||||
"submit_advice",
|
||||
"Submit your pillar recommendations as markdown. Ends your turn.",
|
||||
"Submit your pillar recommendations as markdown, with your review of the "
|
||||
"pillar's checklist. Ends your turn.",
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"content": {"type": "string"}},
|
||||
"properties": {
|
||||
"content": text,
|
||||
"checklist": {
|
||||
"type": "array",
|
||||
"description": "One entry for every item of the Skill's "
|
||||
"validation checklist.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"item": text,
|
||||
"status": {"type": "string", "enum": list(STATUSES)},
|
||||
"note": {
|
||||
"type": "string",
|
||||
"description": "How your advice addresses it, or "
|
||||
"why it does not apply to this workload.",
|
||||
},
|
||||
},
|
||||
"required": ["item", "status", "note"],
|
||||
},
|
||||
},
|
||||
},
|
||||
"required": ["content"],
|
||||
},
|
||||
submit_advice,
|
||||
tier="execution",
|
||||
)
|
||||
|
||||
|
||||
def render_checklist_review(working: Mapping[str, Any]) -> str:
|
||||
"""The advisors' checklist reviews as one section of markdown tables, or "" when
|
||||
there are none. Written by code from what the advisors recorded."""
|
||||
reviews = working.get(CHECKLIST_KEY)
|
||||
if not isinstance(reviews, Mapping) or not any(reviews.values()):
|
||||
return ""
|
||||
parts = [
|
||||
f"## {REVIEW_HEADING}",
|
||||
"",
|
||||
(
|
||||
"Each pillar advisor's review of its pillar's checklist for this "
|
||||
"workload: what its advice addresses, and what does not apply and why."
|
||||
),
|
||||
]
|
||||
for pillar, entries in reviews.items():
|
||||
if not entries:
|
||||
continue
|
||||
parts += [
|
||||
"",
|
||||
f"### {pillar}",
|
||||
"",
|
||||
"| Checklist item | Review | How, or why not |",
|
||||
"| --- | --- | --- |",
|
||||
]
|
||||
parts += [
|
||||
"| "
|
||||
+ " | ".join(
|
||||
str(value).replace("|", "\\|")
|
||||
for value in (
|
||||
entry.get("item", ""),
|
||||
_STATUS_WORDS.get(str(entry.get("status")), ""),
|
||||
entry.get("note", ""),
|
||||
)
|
||||
)
|
||||
+ " |"
|
||||
for entry in entries
|
||||
if isinstance(entry, Mapping)
|
||||
]
|
||||
return "\n".join(parts) + "\n"
|
||||
|
||||
78
app/tools/requirements_review.py
Normal file
78
app/tools/requirements_review.py
Normal file
@@ -0,0 +1,78 @@
|
||||
"""A review of the requirements: they describe the problem, not the solution.
|
||||
|
||||
Requirements discovery is kept apart from solutioning, so a requirement names no
|
||||
product. That rule was an instruction the engine could not enforce; the evaluation
|
||||
checked it only afterwards, against a list of product names. This asks the decider
|
||||
(ADR-0007), once per requirement, whether it names a cloud product or managed
|
||||
service. It never blocks: what it raises is returned to the Specialist once.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from app.adk.runtime import Decider, Question
|
||||
from app.workflows.chain import REQUIREMENTS_FILE, read_list
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TEXT = 200
|
||||
|
||||
|
||||
class RequirementsReview:
|
||||
def __init__(self, decider: Decider, *, act: float = 0.3) -> None:
|
||||
self._decider = decider
|
||||
# Lower than the router's threshold, from measurement on 2026-10-08: across
|
||||
# 61 requirements from three passed runs the highest answer was 0.10, and
|
||||
# requirements written to name a product scored 0.34, 0.61 and 0.93.
|
||||
self._act = act
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
kind: DeliverableKind,
|
||||
attachments: Mapping[str, str],
|
||||
upstream: Mapping[str, Mapping[str, Any]] | None,
|
||||
scope: str = "",
|
||||
) -> Sequence[str]:
|
||||
"""The requirements that name a product, each as a line for the Specialist."""
|
||||
if kind is not DeliverableKind.REQUIREMENTS:
|
||||
return ()
|
||||
items = [
|
||||
item
|
||||
for item in read_list(
|
||||
{"requirements": {"attachments": attachments}},
|
||||
"requirements",
|
||||
REQUIREMENTS_FILE,
|
||||
)
|
||||
if isinstance(item.get("id"), str) and isinstance(item.get("text"), str)
|
||||
]
|
||||
if not items:
|
||||
return ()
|
||||
answers = await self._decider.decide(
|
||||
"Requirements for a workload, written before any product is chosen.",
|
||||
{
|
||||
item["id"]: Question(
|
||||
"noul",
|
||||
f'Requirement: "{item["text"]}". Does it name a specific cloud '
|
||||
"provider's product or managed service, as opposed to "
|
||||
"describing what is needed?",
|
||||
)
|
||||
for item in items
|
||||
},
|
||||
scope=scope,
|
||||
)
|
||||
if not answers:
|
||||
return ()
|
||||
raised = [
|
||||
f'{item["id"]} names a product or service: "{item["text"][:_TEXT]}". '
|
||||
"Say what is needed, not what to use"
|
||||
for item in items
|
||||
if (answer := answers.get(item["id"])) is not None
|
||||
and answer.is_yes(self._act)
|
||||
]
|
||||
if raised:
|
||||
logger.info("[%s] requirements review raised %d", scope, len(raised))
|
||||
return tuple(raised)
|
||||
@@ -153,19 +153,36 @@ def conditions_of(item: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
return (text.strip(),) if isinstance(text, str) and text.strip() else ()
|
||||
|
||||
|
||||
OUTSIDE_CODE = "outside_code"
|
||||
|
||||
|
||||
def outside_code_of(item: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
"""What a requirement asks for that infrastructure code cannot show: an outcome
|
||||
measured in operation, application behaviour, or a process people follow."""
|
||||
listed = item.get(OUTSIDE_CODE)
|
||||
if not isinstance(listed, list):
|
||||
return ()
|
||||
return tuple(c.strip() for c in listed if isinstance(c, str) and c.strip())
|
||||
|
||||
|
||||
def _lacking_conditions(items: Sequence[Mapping[str, Any]], name: str) -> list[str]:
|
||||
"""A constraint must say, one by one, what would have to be true for it to be met."""
|
||||
"""A constraint must say, one by one, what would have to be true for it to be met:
|
||||
under `conditions` what infrastructure code can show, under `outside_code` what
|
||||
it cannot. At least one of the two has an entry."""
|
||||
problems = []
|
||||
for item in items:
|
||||
listed = item.get("conditions")
|
||||
if (
|
||||
not isinstance(listed, list)
|
||||
or not listed
|
||||
or not all(isinstance(c, str) and c.strip() for c in listed)
|
||||
):
|
||||
outside = item.get(OUTSIDE_CODE, [])
|
||||
well_formed = all(
|
||||
isinstance(group, list)
|
||||
and all(isinstance(c, str) and c.strip() for c in group)
|
||||
for group in (listed, outside)
|
||||
)
|
||||
if not well_formed or not (listed or outside):
|
||||
problems.append(
|
||||
f"{name}: {item['id']} needs `conditions`: a list with one entry "
|
||||
"for each separate thing it asks for"
|
||||
"for each separate thing it asks for that infrastructure code can "
|
||||
f"show (and `{OUTSIDE_CODE}` for what it cannot)"
|
||||
)
|
||||
return problems
|
||||
|
||||
|
||||
@@ -43,6 +43,7 @@ def build_execution_graph(
|
||||
skill_versions: Mapping[str, str] | None = None,
|
||||
orchestrator: OrchestratorPolicy | None = None,
|
||||
submission_review: SubmissionReview | None = None,
|
||||
confirm_drafts: bool = False,
|
||||
) -> StateGraph:
|
||||
nodes = make_execution_nodes(
|
||||
runtime=runtime,
|
||||
@@ -55,6 +56,7 @@ def build_execution_graph(
|
||||
skill_versions=skill_versions,
|
||||
orchestrator=orchestrator,
|
||||
submission_review=submission_review,
|
||||
confirm_drafts=confirm_drafts,
|
||||
)
|
||||
|
||||
graph = StateGraph(ExecutionState)
|
||||
|
||||
@@ -195,7 +195,8 @@ def chain_gaps(
|
||||
A functional requirement is carried by a component that has an Element, or by a
|
||||
Check of its own. A non-functional requirement and a design recommendation are
|
||||
each a constraint, so each must be the source of a Check (a recommendation may
|
||||
instead be marked not applicable, with a reason).
|
||||
instead be marked not applicable, with a reason). A constraint that lists
|
||||
nothing infrastructure code can show, only what lies outside it, needs none.
|
||||
"""
|
||||
gaps: list[str] = []
|
||||
requirement_ids = {item["id"] for item in requirements}
|
||||
@@ -229,6 +230,8 @@ def chain_gaps(
|
||||
)
|
||||
if requirement.get("kind") == "functional" and served:
|
||||
continue
|
||||
if _only_outside_code(requirement):
|
||||
continue
|
||||
gaps.append(f"Requirement {requirement['id']} has no Check")
|
||||
for selection in products:
|
||||
used = any(
|
||||
@@ -249,11 +252,19 @@ def chain_gaps(
|
||||
"which is not a design recommendation"
|
||||
)
|
||||
for recommendation in recommendations:
|
||||
if recommendation["id"] not in sources | excused:
|
||||
if recommendation["id"] not in sources | excused and not _only_outside_code(
|
||||
recommendation
|
||||
):
|
||||
gaps.append(f"Recommendation {recommendation['id']} has no Check")
|
||||
return tuple(gaps)
|
||||
|
||||
|
||||
def _only_outside_code(item: Mapping[str, Any]) -> bool:
|
||||
"""Whether a constraint says, explicitly, that nothing it asks for can be shown
|
||||
by infrastructure code."""
|
||||
return item.get("conditions") == [] and bool(item.get("outside_code"))
|
||||
|
||||
|
||||
# --- the diagram is a view of the graph ----------------------------------------------
|
||||
|
||||
_UNSAFE = re.compile(r'["\[\]{}()<>|`#;]')
|
||||
|
||||
184
eval/calibrate_judge.py
Normal file
184
eval/calibrate_judge.py
Normal file
@@ -0,0 +1,184 @@
|
||||
"""Calibrate the evaluation's judges against a person's labels.
|
||||
|
||||
A labelled set is questions about saved design recommendations, each with the
|
||||
answer a person gave. This asks every judge each question several times and reports:
|
||||
|
||||
agreement how often the judge's answer matches the person's
|
||||
stability how often the judge gives the same answer to the same question
|
||||
between how often the two judges agree with each other
|
||||
|
||||
Stability and agreement between judges need no labels, so they can be measured
|
||||
before anyone has marked the set. Agreement with a person cannot.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python -m eval.calibrate_judge [--runs 3] [--build]
|
||||
|
||||
--build writes eval/datasets/judge/labels.json from the documents in that folder,
|
||||
with `label` left empty for a person to fill in: 1 or 0 for a yes_no question; 0,
|
||||
0.33, 0.67 or 1 for a graded one. Costs model usage: run deliberately.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
from collections.abc import Mapping, Sequence
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.agent import open_decider
|
||||
from app.config import get_settings
|
||||
from app.providers import load_provider
|
||||
from eval.judge import DeciderJudge, GeminiJudge, Judge, JudgeUnavailable
|
||||
from eval.scoring import QUALITY_RUBRIC, checklist_items
|
||||
|
||||
FOLDER = Path(__file__).resolve().parent / "datasets" / "judge"
|
||||
LABELS = FOLDER / "labels.json"
|
||||
GEMINI_PREFIX = "gemini/"
|
||||
PER_PILLAR = 2 # checklist items sampled from each pillar, per document
|
||||
TOLERANCE = 0.17 # two graded answers within half a grade count as the same
|
||||
|
||||
|
||||
def build(seed: int = 7) -> list[dict[str, Any]]:
|
||||
"""Twenty-odd questions over the saved documents, labels left for a person."""
|
||||
bundle = load_provider(get_settings().CLOUD_PROVIDER)
|
||||
chooser = random.Random(seed)
|
||||
items: list[dict[str, Any]] = []
|
||||
for document in sorted(FOLDER.glob("*.md")):
|
||||
for pillar, skill in bundle.pillar_skills.items():
|
||||
checklist = checklist_items(bundle.skills[skill].path / "SKILL.md")
|
||||
for item in chooser.sample(checklist, min(PER_PILLAR, len(checklist))):
|
||||
items.append(
|
||||
{
|
||||
"id": f"{document.stem}.{pillar}.{len(items)}",
|
||||
"document": document.name,
|
||||
"kind": "yes_no",
|
||||
"text": f"Do the recommendations address this {pillar} "
|
||||
"checklist item, either by recommending it or by "
|
||||
f"explaining why it does not apply? {item}",
|
||||
"label": None,
|
||||
}
|
||||
)
|
||||
for key, text in QUALITY_RUBRIC.items():
|
||||
items.append(
|
||||
{
|
||||
"id": f"{document.stem}.quality.{key}",
|
||||
"document": document.name,
|
||||
"kind": "graded",
|
||||
"text": text,
|
||||
"label": None,
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
def same(a: float, b: float) -> bool:
|
||||
return abs(a - b) <= TOLERANCE
|
||||
|
||||
|
||||
def share(pairs: Sequence[tuple[float, float]]) -> float | None:
|
||||
return round(sum(same(a, b) for a, b in pairs) / len(pairs), 2) if pairs else None
|
||||
|
||||
|
||||
async def ask(
|
||||
judge: Judge, items: Sequence[Mapping[str, Any]], runs: int
|
||||
) -> dict[str, list[float]]:
|
||||
"""Every question's answers, one per run, by question id."""
|
||||
answers: dict[str, list[float]] = {item["id"]: [] for item in items}
|
||||
documents = sorted({item["document"] for item in items})
|
||||
for _ in range(runs):
|
||||
for name in documents:
|
||||
asked = {
|
||||
item["id"]: {"kind": item["kind"], "text": item["text"]}
|
||||
for item in items
|
||||
if item["document"] == name
|
||||
}
|
||||
try:
|
||||
scores = await judge.answer(
|
||||
(FOLDER / name).read_text(encoding="utf-8"), asked
|
||||
)
|
||||
except JudgeUnavailable:
|
||||
continue
|
||||
for key, value in scores.items():
|
||||
if key in answers:
|
||||
answers[key].append(float(value))
|
||||
return answers
|
||||
|
||||
|
||||
def report(
|
||||
items: Sequence[Mapping[str, Any]], by_judge: Mapping[str, dict[str, list[float]]]
|
||||
) -> dict[str, Any]:
|
||||
labelled = {i["id"]: float(i["label"]) for i in items if i["label"] is not None}
|
||||
result: dict[str, Any] = {
|
||||
"questions": len(items),
|
||||
"labelled": len(labelled),
|
||||
"judges": {},
|
||||
}
|
||||
for name, answers in by_judge.items():
|
||||
first = {key: values[0] for key, values in answers.items() if values}
|
||||
result["judges"][name] = {
|
||||
"answered": len(first),
|
||||
# Same answer on every run, per question.
|
||||
"stability": share(
|
||||
[
|
||||
(values[0], other)
|
||||
for values in answers.values()
|
||||
for other in values[1:]
|
||||
]
|
||||
),
|
||||
"agreement_with_labels": share(
|
||||
[(first[key], label) for key, label in labelled.items() if key in first]
|
||||
),
|
||||
"mean_answer": round(sum(first.values()) / len(first), 2)
|
||||
if first
|
||||
else None,
|
||||
}
|
||||
names = list(by_judge)
|
||||
if len(names) == 2:
|
||||
a, b = (by_judge[name] for name in names)
|
||||
result["between_judges"] = share(
|
||||
[(a[key][0], b[key][0]) for key in a if a[key] and b.get(key)]
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def main(arguments: Sequence[str]) -> int:
|
||||
if "--build" in arguments:
|
||||
items = build()
|
||||
LABELS.write_text(json.dumps(items, indent=2, ensure_ascii=False) + "\n")
|
||||
print(f"Wrote {len(items)} questions to {LABELS}; fill in each `label`.")
|
||||
return 0
|
||||
runs = int(arguments[arguments.index("--runs") + 1]) if "--runs" in arguments else 3
|
||||
items = json.loads(LABELS.read_text(encoding="utf-8"))
|
||||
settings = get_settings()
|
||||
model = load_provider(settings.CLOUD_PROVIDER).models.default
|
||||
by_judge: dict[str, dict[str, list[float]]] = {
|
||||
"jev": {item["id"]: [] for item in items},
|
||||
"gemini": {item["id"]: [] for item in items},
|
||||
}
|
||||
for _ in range(runs):
|
||||
# A new Jev client for each run: the client caches identical requests, and a
|
||||
# cached answer would make stability look perfect whatever the service does.
|
||||
decider = open_decider(settings)
|
||||
judges: dict[str, Judge] = {
|
||||
"jev": DeciderJudge(decider, scope="judge-calibration"),
|
||||
"gemini": GeminiJudge(model.removeprefix(GEMINI_PREFIX)),
|
||||
}
|
||||
try:
|
||||
for name, judge in judges.items():
|
||||
for key, values in (await ask(judge, items, 1)).items():
|
||||
by_judge[name][key] += values
|
||||
finally:
|
||||
aclose = getattr(decider, "aclose", None)
|
||||
if aclose:
|
||||
await aclose()
|
||||
result = {"runs": runs, **report(items, by_judge)}
|
||||
(FOLDER / "calibration.json").write_text(json.dumps(result, indent=2) + "\n")
|
||||
print(json.dumps(result, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(asyncio.run(main(sys.argv[1:])))
|
||||
20
eval/datasets/judge/calibration.json
Normal file
20
eval/datasets/judge/calibration.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"runs": 3,
|
||||
"questions": 32,
|
||||
"labelled": 0,
|
||||
"judges": {
|
||||
"jev": {
|
||||
"answered": 32,
|
||||
"stability": 1.0,
|
||||
"agreement_with_labels": null,
|
||||
"mean_answer": 0.37
|
||||
},
|
||||
"gemini": {
|
||||
"answered": 32,
|
||||
"stability": 0.92,
|
||||
"agreement_with_labels": null,
|
||||
"mean_answer": 0.32
|
||||
}
|
||||
},
|
||||
"between_judges": 0.91
|
||||
}
|
||||
44
eval/datasets/judge/case-02-design-recommendations.md
Normal file
44
eval/datasets/judge/case-02-design-recommendations.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Design Recommendations
|
||||
|
||||
This document outlines the architectural configurations and best practices applied to the components of the high-scale data ingestion pipeline. These recommendations integrate the functional and non-functional requirements (NFRs) with guidance from the Google Cloud Well-Architected Framework pillar advisors to ensure a secure, reliable, cost-optimized, and high-performing system.
|
||||
|
||||
## Security, privacy, and compliance
|
||||
|
||||
To protect the ingestion endpoint from volumetric DDoS attacks and Layer 7 exploits, a Global External Application Load Balancer with Google Cloud Armor is deployed. This fulfills the requirement for strict perimeter isolation while serving the 10,000 RPS traffic over TLS 1.2+ (**DR-1**).
|
||||
|
||||
The architecture implements a zero-trust model where all internal pipeline components (Cloud Run, Pub/Sub, Cloud Storage, BigQuery) communicate privately via VPC Service Controls and direct VPC egress (**DR-3**). All processes run under dedicated, least-privilege IAM service accounts, tightly binding permissions to only the required resources (**DR-4**). To meet regulatory data protection mandates and ensure cryptographic sovereignty, all persistent data across the Pub/Sub staging buffer, Cloud Storage sink, and BigQuery quarantine store is encrypted with Customer-Managed Encryption Keys (CMEK) (**DR-2**).
|
||||
|
||||
## Reliability
|
||||
|
||||
Reliability is anchored on minimizing data loss (RPO 0) and avoiding systemic failures under heavy load. The ingestion endpoint mitigates the risk of cold-start latency spikes by maintaining a minimum instance baseline, allowing it to sustainably absorb traffic bursts without violating the 200ms p95 latency target (**DR-5**).
|
||||
|
||||
Within the asynchronous pipeline, downstream processing degradation is isolated from ingestion. The Pub/Sub validation subscription utilizes a dead-letter policy with bounded delivery attempts (5 retries) to redirect unparseable poison-pill messages to BigQuery, preventing queue head-of-line blocking (**DR-6**). Additionally, the valid record storage sink leverages multi-region replication and object versioning to guarantee data durability even during a regional outage or accidental deletion event (**DR-7**). Finally, the Cloud Run services utilize explicit startup and liveness probes to ensure traffic is only routed to healthy, fully-initialized containers (**DR-8**).
|
||||
|
||||
## Operational excellence
|
||||
|
||||
A foundational principle of operational excellence is reproducible, reviewable, and automated changes. Every pipeline component—from IAM bindings to Pub/Sub schemas and serverless runtimes—is defined declaratively as Infrastructure as Code (Terraform) (**DR-9**). This declarative approach acts as the primary recovery mechanism to satisfy the RTO < 60 minutes disaster recovery objective.
|
||||
|
||||
## Cost optimization
|
||||
|
||||
Cost optimizations prioritize scaling tightly with business demand. Both the Ingestion and Validation Cloud Run services are configured to allocate CPU only during request processing, meaning they incur zero compute costs when idle, perfectly matching the strict scale-to-zero requirement (**DR-10**).
|
||||
|
||||
Storage costs are minimized through aggressive lifecycle management. The Pub/Sub buffer retains messages only for a shortened 48-hour window, as RPO=0 durability is fulfilled by the downstream sinks (**DR-11**). The Cloud Storage bucket transitions older validated batches to Nearline storage after 30 days (**DR-12**), and the BigQuery dead-letter quarantine datasets enforce a 30-day automated partition expiration to prevent unbound storage accumulation of malformed payloads (**DR-13**).
|
||||
|
||||
## Performance efficiency
|
||||
|
||||
To sustain the 10,000 RPS throughput requirement with sub-second latencies, the pipeline leverages advanced serverless execution features. Both Cloud Run services use Execution Environment Gen 2 and Startup CPU Boost to optimize network packet processing and eliminate initialization drag (**DR-14**). The Ingestion service's concurrency is explicitly tuned to handle high parallel request volumes (e.g., 80 concurrent requests per instance), preventing excessive container scale-out thrashing (**DR-15**).
|
||||
|
||||
Between the buffer and the processing tier, the Validation Engine relies on a Pub/Sub push subscription to immediately invoke the validation containers, minimizing pull polling overhead and reducing end-to-end event latency (**DR-16**).
|
||||
|
||||
## Sustainability
|
||||
|
||||
To minimize the workload's carbon footprint, all services are provisioned together within a Google Cloud region boasting a high Carbon-Free Energy (CFE) score (e.g., `europe-west1` or `us-central1`) (**DR-17**).
|
||||
|
||||
Energy waste is further curtailed by enforcing schema compliance at the initial Pub/Sub buffer boundary, dropping malformed payloads early and preventing wasted validation compute cycles (**DR-19**). Lastly, the Quarantine Replay Processor is architected as an on-demand Cloud Run Job rather than an always-on polling consumer, aligning energy expenditure directly to actual remediation activity (**DR-18**).
|
||||
|
||||
## Assumptions
|
||||
|
||||
As this design was produced in pre-approved mode, the following assumptions were made:
|
||||
- **Cloud Region Selection**: We assume the selection of a specific region (like `europe-west1` or `us-central1`) will balance the multi-zone availability requirements with the high CFE targets for sustainability.
|
||||
- **Micro-batching**: We assume the Validation Engine service includes internal micro-batching logic to aggregate records before writing to Cloud Storage, avoiding excessive single-object API calls that could degrade performance.
|
||||
- **Trace Sampling & Logging**: Observability costs will be governed by sampling Cloud Trace ingestion and filtering out routine Cloud Run health check (`/healthz`) logs out-of-band via standard project sinks.
|
||||
56
eval/datasets/judge/case-03-design-recommendations.md
Normal file
56
eval/datasets/judge/case-03-design-recommendations.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Design Recommendations
|
||||
|
||||
## Assumptions
|
||||
|
||||
Since `pre_approved` is `true`, the following assumptions govern these design recommendations:
|
||||
- The workload is deployed regionally in a low-carbon region (a region with CFE% >= 75%) to meet sustainability goals.
|
||||
- Cloud SQL for PostgreSQL is configured in a high-availability (multi-zone) topology to meet the 99.9% uptime requirement, avoiding cross-region replication.
|
||||
- Cloud Run handles the compute scaling, scaling down to 0 instances outside of business hours to optimize costs.
|
||||
- The Corporate Identity Provider integrates with Identity-Aware Proxy (IAP) to authenticate users at the load balancer.
|
||||
- All application and operational logs are stored in a centralized Google Cloud Storage bucket locked for 365 days.
|
||||
|
||||
## Security, privacy, and compliance
|
||||
|
||||
To enforce a zero-trust model and meet security, privacy, and compliance requirements, the architecture implements several layers of defense:
|
||||
|
||||
- **Identity-Aware Proxy (IAP)**: Protects the Cloud Run service by authenticating corporate users at the external load balancer before traffic reaches the application (DR-2).
|
||||
- **Network Security**: The Load Balancer enforces a custom SSL policy restricting traffic to TLS 1.2+ (DR-1). The Cloud Run service itself restricts ingress to `internal-and-cloud-load-balancing` so it cannot be accessed directly via the public internet (DR-3). Outbound traffic from Cloud Run is routed strictly through the VPC using Direct VPC Egress (DR-4).
|
||||
- **Least Privilege and Secrets**: Cloud Run operates using a dedicated, user-managed service account (DR-5) with permissions limited to accessing necessary secrets from Secret Manager (DR-8) and connecting to the database. These secrets are injected dynamically into the runtime environment (DR-9).
|
||||
- **Database Security**: Cloud SQL is isolated from the public internet by disabling public IPv4 and connecting via Private Service Connect/VPC peering (DR-6). It enforces IAM authentication and SSL/TLS connections (DR-7).
|
||||
- **Compliance Logging**: To meet the 365-day immutable audit log requirement, a Cloud Logging sink routes all audit events to a dedicated Cloud Storage bucket configured with a 365-day retention policy (DR-10).
|
||||
|
||||
## Reliability
|
||||
|
||||
High availability and disaster recovery are essential to meeting the 99.9% uptime and RTO/RPO requirements:
|
||||
|
||||
- **High Availability**: Cloud SQL for PostgreSQL is configured for regional high availability, maintaining a synchronous standby replica in a secondary zone to allow automatic failover during a disruption (DR-11).
|
||||
- **Disaster Recovery**: Automated daily backups are retained for at least 7 days, and Point-in-Time Recovery (PITR) is enabled via Write-Ahead Logging (WAL) to guarantee an RPO of 1 hour or less (DR-12).
|
||||
- **Health Checking**: The load balancer utilizes backend health checks to continuously monitor the health of the Cloud Run instances, routing traffic away from degraded containers (DR-13).
|
||||
|
||||
## Operational excellence
|
||||
|
||||
Operations are streamlined through comprehensive observability and automated alerting:
|
||||
|
||||
- **Monitoring and Alerting**: Cloud Monitoring captures metrics across all components. Alert policies are specifically configured to trigger when Cloud Run HTTP 5xx error rates exceed 1% over a 5-minute window, and when Cloud SQL CPU or storage utilization exceeds 80% (DR-16).
|
||||
- **Centralized Logs**: All system metrics, application logs, and traces are centralized within Google Cloud's operations suite for rapid incident response.
|
||||
|
||||
## Cost optimization
|
||||
|
||||
The architecture leverages serverless characteristics to align cloud spend with active business usage:
|
||||
|
||||
- **Scale to Zero**: Cloud Run is configured with a minimum instance count of 0, ensuring that compute resources scale down entirely during off-hours, weekends, and periods of inactivity, drastically reducing idle costs (DR-14).
|
||||
- **Right-sized Database**: While high availability is required, autoscaling storage ensures that database disk space is only paid for when consumed.
|
||||
|
||||
## Performance efficiency
|
||||
|
||||
Sub-second page loads and transactional consistency are maintained through optimized storage and networking:
|
||||
|
||||
- **Database Performance**: Cloud SQL is provisioned with high-throughput SSD storage and automatic storage expansion to maintain low-latency read/write operations for booking transactions without hitting IOPS ceilings (DR-15).
|
||||
- **Connection Optimization**: By using VPC-native private connections and keeping all traffic within the same region, network latency between Cloud Run and Cloud SQL is minimized.
|
||||
|
||||
## Sustainability
|
||||
|
||||
Environmental impact is mitigated by selecting the right region and eliminating idle resource waste:
|
||||
|
||||
- **Low-Carbon Region**: All resources are deployed in a region with a Carbon-Free Energy (CFE) score of 75% or higher, directly lowering the gross carbon emissions of the workload (DR-17).
|
||||
- **Dynamic Scaling**: Utilizing Cloud Run's scale-to-zero capabilities (DR-14) ensures that the application does not consume power for idle compute instances outside of business hours.
|
||||
226
eval/datasets/judge/labels.json
Normal file
226
eval/datasets/judge/labels.json
Normal file
@@ -0,0 +1,226 @@
|
||||
[
|
||||
{
|
||||
"id": "case-02-design-recommendations.security.0",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Shift-left security: Are automated security scans integrated into the CI/CD pipeline?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.security.1",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Zero trust: Is Cloud Identity used as a centralized identity provider for managing user lifecycles and federation?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.reliability.2",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? Game days or chaos engineering practices are regularly held to validate failure recovery.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.reliability.3",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? The system architecture incorporates patterns like circuit breakers, retries with exponential backoff, and rate limiting to support graceful degradation.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.operational-excellence.4",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Operational readiness: A formal framework or set of criteria exists to assess operational readiness before production deployment.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.operational-excellence.5",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Culture of improvement: A documented strategy is in place for regularly reviewing and adapting cloud operations to industry advancements.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.cost-optimization.6",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Network Egress: Data transfer is minimized by keeping traffic regional, using **Cloud CDN**, and leveraging **Standard Network Tier** where appropriate. High-volume on-premises traffic uses **Direct Peering** or **Cloud Interconnect** (always verify if hybrid connectivity is involved).",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.cost-optimization.7",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Granular Visibility: BigQuery billing export is enabled and used for regular cost reviews.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.performance-optimization.8",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Monitoring and continuous improvement: Automated dashboards and alerts are configured for key performance indicators (KPIs).",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.performance-optimization.9",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Elasticity: The architecture leverages serverless or managed services to dynamically match capacity to load.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.sustainability.10",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Cloud sustainability: AI is used as a catalyst for profitability and resilience to streamline operations, or sustainability is integrated into the design process to create positive feedback loops.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.sustainability.11",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Continuously measure and improve: Applications are instrumented to measure the specific carbon intensity of software features, or automated exports of Carbon Footprint data to BigQuery are configured for deep analysis.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.quality.specific",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Are the recommendations specific to this workload and its stated requirements, as opposed to generic best practice that would fit any workload?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.quality.actionable",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Does each recommendation name the concrete setting, feature or configuration to apply, so an engineer could act on it without more research?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.quality.trade_offs",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Where pillars pull in different directions (for example cost against reliability), is the trade-off named and a resolution recommended?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-02-design-recommendations.quality.traceable",
|
||||
"document": "case-02-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Can each recommendation be traced to a requirement or constraint of the workload?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.security.16",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Zero trust: Are private connectivity methods (Cloud Interconnect, VPN) used for internal traffic?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.security.17",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this security checklist item, either by recommending it or by explaining why it does not apply? Security by design: Is defense-in-depth implemented at the network, host, and application layers?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.reliability.18",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? The architecture avoids single points of failure through cross-zone or cross-region redundancy.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.reliability.19",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this reliability checklist item, either by recommending it or by explaining why it does not apply? Application and infrastructure health checks are configured to trigger automated failovers.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.operational-excellence.20",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Resource optimization: Resource utilization is regularly reviewed using recommendations from Active Assist or performance data.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.operational-excellence.21",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this operational-excellence checklist item, either by recommending it or by explaining why it does not apply? Operational readiness: A formal framework or set of criteria exists to assess operational readiness before production deployment.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.cost-optimization.22",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Rightsizing: Resources are regularly adjusted based on rightsizing suggestions provided by Active Assist Recommender.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.cost-optimization.23",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this cost-optimization checklist item, either by recommending it or by explaining why it does not apply? Granular Visibility: BigQuery billing export is enabled and used for regular cost reviews.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.performance-optimization.24",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Elasticity: Auto-scaling rules are configured and validated to handle variable demand.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.performance-optimization.25",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this performance-optimization checklist item, either by recommending it or by explaining why it does not apply? Monitoring and continuous improvement: Distributed tracing and profiling tools are used to identify code-level bottlenecks.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.sustainability.26",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Cloud sustainability: AI is used as a catalyst for profitability and resilience to streamline operations, or sustainability is integrated into the design process to create positive feedback loops.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.sustainability.27",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "yes_no",
|
||||
"text": "Do the recommendations address this sustainability checklist item, either by recommending it or by explaining why it does not apply? Promote a culture of sustainability: Abstract carbon metrics are transformed into tangible progress indicators in annual reports, or sustainability is treated as a first-class technical requirement (NFR) tied to KPIs and performance reviews.",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.quality.specific",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Are the recommendations specific to this workload and its stated requirements, as opposed to generic best practice that would fit any workload?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.quality.actionable",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Does each recommendation name the concrete setting, feature or configuration to apply, so an engineer could act on it without more research?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.quality.trade_offs",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Where pillars pull in different directions (for example cost against reliability), is the trade-off named and a resolution recommended?",
|
||||
"label": null
|
||||
},
|
||||
{
|
||||
"id": "case-03-design-recommendations.quality.traceable",
|
||||
"document": "case-03-design-recommendations.md",
|
||||
"kind": "graded",
|
||||
"text": "Can each recommendation be traced to a requirement or constraint of the workload?",
|
||||
"label": null
|
||||
}
|
||||
]
|
||||
@@ -57,7 +57,12 @@
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
]
|
||||
]
|
||||
],
|
||||
"thresholds": {
|
||||
"revisions": 2,
|
||||
"seconds": 3000,
|
||||
"tokens": 5000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "case-02-high-scale-ingestion",
|
||||
@@ -121,7 +126,12 @@
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
]
|
||||
]
|
||||
],
|
||||
"thresholds": {
|
||||
"revisions": 3,
|
||||
"seconds": 3600,
|
||||
"tokens": 8500000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "case-03-internal-web-app",
|
||||
@@ -193,7 +203,12 @@
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
]
|
||||
]
|
||||
],
|
||||
"thresholds": {
|
||||
"revisions": 3,
|
||||
"seconds": 3200,
|
||||
"tokens": 8000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "case-04-interactive-web-app",
|
||||
@@ -207,65 +222,89 @@
|
||||
"scripted_answers": [
|
||||
[
|
||||
"region|location|geograph",
|
||||
"A single region, europe-west1 (Belgium). All users are in the EU."
|
||||
"A single region, europe-west1 (Belgium). All users are in the EU.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"user|staff|employee|how many|concurren|traffic|volume|scale|peak|requests per|rps",
|
||||
"About 4,000 employees; 300 concurrent users at the 09:00 peak, around 20 requests per second."
|
||||
"About 4,000 employees; 300 concurrent users at the 09:00 peak, around 20 requests per second.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"availability|uptime|sla|rto|rpo|disaster|recover|backup",
|
||||
"99.5% during business hours (07:00-19:00 CET). RTO 4 hours, RPO 1 hour. Daily backups kept 14 days."
|
||||
"99.5% during business hours (07:00-19:00 CET). RTO 4 hours, RPO 1 hour. Daily backups kept 14 days.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"latency|response time|performance",
|
||||
"Pages should load within 1 second at p95."
|
||||
"Pages should load within 1 second at p95.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"identity|sign.?in|login|sso|auth|idp|access",
|
||||
"Employees sign in with Google Workspace accounts; no access from outside the company. Admins are a Google group."
|
||||
"Employees sign in with Google Workspace accounts; no access from outside the company. Admins are a Google group.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"security|compliance|encrypt|privacy|gdpr|secret|credential|password",
|
||||
"GDPR applies to employee names and emails. Google-managed keys are fine. Database passwords and API keys must live in a secrets store and rotate every 90 days."
|
||||
"GDPR applies to employee names and emails. Google-managed keys are fine. Database passwords and API keys must live in a secrets store and rotate every 90 days.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"data|database|schema|relational|postgres|mysql|size|storage",
|
||||
"PostgreSQL. Under 20 GB of data in the first three years; bookings, rooms, equipment and audit history."
|
||||
"PostgreSQL. Under 20 GB of data in the first three years; bookings, rooms, equipment and audit history.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"cost|budget|spend",
|
||||
"Under 600 EUR a month; the database can be small."
|
||||
"Under 600 EUR a month; the database can be small.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"current|existing|today|on-prem|migrat|legacy",
|
||||
"Bookings are kept in a shared spreadsheet today; there is no data to migrate."
|
||||
"Bookings are kept in a shared spreadsheet today; there is no data to migrate.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"depend|integrat|calendar|email|notif|downstream|upstream",
|
||||
"Send booking confirmation emails through the existing SMTP relay; no calendar integration in phase one."
|
||||
"Send booking confirmation emails through the existing SMTP relay; no calendar integration in phase one.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"operat|monitor|logging|alert|team|support|deploy|ci|pipeline",
|
||||
"Two developers own it, deploy containers from Cloud Build with Terraform, and want Cloud Monitoring alerts to a Google Chat space."
|
||||
"Two developers own it, deploy containers from Cloud Build with Terraform, and want Cloud Monitoring alerts to a Google Chat space.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"network|vpc|private|internet|egress|ingress",
|
||||
"No public database endpoint; the database must use private IP only."
|
||||
"No public database endpoint; the database must use private IP only.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"sustainab|carbon",
|
||||
"No specific requirement."
|
||||
"No specific requirement.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"retention|audit|archive|delete",
|
||||
"Keep booking history 2 years, then delete; keep audit logs 1 year."
|
||||
"Keep booking history 2 years, then delete; keep audit logs 1 year.",
|
||||
"requirements"
|
||||
],
|
||||
[
|
||||
"approve|confirm|proceed|look right|any changes|happy",
|
||||
"Approved. Please proceed."
|
||||
],
|
||||
[
|
||||
".",
|
||||
"Approved. Please proceed.",
|
||||
"^(?!requirements$)"
|
||||
]
|
||||
],
|
||||
"interactive": true
|
||||
"interactive": true,
|
||||
"thresholds": {
|
||||
"revisions": 2,
|
||||
"seconds": 3400,
|
||||
"tokens": 7500000
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
@@ -71,7 +71,7 @@ class CaseSpec:
|
||||
name: str
|
||||
workflow_request: str
|
||||
required_products: tuple[str, ...]
|
||||
scripted_answers: tuple[tuple[str, str], ...]
|
||||
scripted_answers: tuple[tuple[str, ...], ...] # (pattern, answer[, asker])
|
||||
# Run with a user in the loop (scripted): Approvals and questions are answered
|
||||
# turn by turn, and Specialists are checked for asking before they submit.
|
||||
interactive: bool = False
|
||||
|
||||
@@ -24,7 +24,7 @@ import os
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Callable, Sequence
|
||||
from dataclasses import asdict
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
@@ -58,8 +58,12 @@ RESULTS_DIR = ROOT / "results"
|
||||
class CountingRuntime:
|
||||
"""Counts agent runs and their outcomes per role, with a timeline of runs."""
|
||||
|
||||
def __init__(self, inner: AgentRuntime) -> None:
|
||||
def __init__(
|
||||
self, inner: AgentRuntime, on_run: Callable[[], None] | None = None
|
||||
) -> None:
|
||||
self._inner = inner
|
||||
# Called after every agent run, so a run cut short leaves a record behind.
|
||||
self._on_run = on_run
|
||||
self._started = time.monotonic()
|
||||
self.runs: Counter[str] = Counter()
|
||||
self.outcomes: Counter[str] = Counter()
|
||||
@@ -81,6 +85,8 @@ class CountingRuntime:
|
||||
"outcome": type(outcome).__name__,
|
||||
}
|
||||
)
|
||||
if self._on_run:
|
||||
self._on_run()
|
||||
return outcome
|
||||
|
||||
|
||||
@@ -145,9 +151,30 @@ async def run_case(
|
||||
skill_dirs={name: meta.path for name, meta in bundle.skills.items()},
|
||||
model_timeout_seconds=settings.MODEL_TIMEOUT_SECONDS,
|
||||
)
|
||||
runtime = CountingRuntime(adk_runtime)
|
||||
recorder = InMemoryRecorder()
|
||||
started = time.monotonic()
|
||||
progress_path = run_dir / f"{case.id}-progress.json"
|
||||
|
||||
def write_progress() -> None:
|
||||
"""What the run has cost so far. Overwritten after every agent run and left
|
||||
in place, so an interrupted run still has its time, tokens and timeline."""
|
||||
progress_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"case_id": case.id,
|
||||
"finished": False,
|
||||
"seconds": round(time.monotonic() - started, 1),
|
||||
"agent_runs": dict(runtime.runs),
|
||||
"outcomes": dict(runtime.outcomes),
|
||||
"tokens": token_report(adk_runtime.usage),
|
||||
"timeline": runtime.timeline,
|
||||
},
|
||||
indent=2,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
runtime = CountingRuntime(adk_runtime, on_run=write_progress)
|
||||
recorder = InMemoryRecorder()
|
||||
async with open_assembly(
|
||||
settings, runtime=runtime, recorder=recorder, output_root=run_dir
|
||||
) as assembly:
|
||||
@@ -166,6 +193,13 @@ async def run_case(
|
||||
scripted_answers=case.scripted_answers,
|
||||
)
|
||||
elapsed = time.monotonic() - started
|
||||
write_progress()
|
||||
progress_path.write_text(
|
||||
progress_path.read_text(encoding="utf-8").replace(
|
||||
'"finished": false', '"finished": true', 1
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
runner = TerraformRunner(
|
||||
binary=settings.TERRAFORM_BINARY,
|
||||
|
||||
@@ -4,7 +4,7 @@ description: >-
|
||||
Terraform patterns for Google Cloud products that validate against one pinned
|
||||
version of the hashicorp/google provider: Cloud Run services and jobs,
|
||||
Pub/Sub, Cloud Storage, IAM, Secret Manager, Cloud KMS, Cloud SQL,
|
||||
networking, Cloud Scheduler, Artifact Registry, Monitoring and API
|
||||
networking, load balancing, Cloud Scheduler, Artifact Registry, Monitoring and API
|
||||
enablement. Use when writing Terraform for a Google Cloud product, before
|
||||
writing the resource, to start from a pattern whose arguments are known to be
|
||||
in the right place.
|
||||
@@ -35,6 +35,7 @@ Every pattern here passes `terraform validate` against `hashicorp/google` 8.6.0.
|
||||
| Cloud KMS | `references/kms.md` |
|
||||
| Cloud SQL | `references/cloud-sql.md` |
|
||||
| VPC networking | `references/networking.md` |
|
||||
| External Application Load Balancer, TLS policy, firewall policy, Identity-Aware Proxy | `references/load-balancer.md` |
|
||||
| Cloud Scheduler | `references/cloud-scheduler.md` |
|
||||
| Artifact Registry | `references/artifact-registry.md` |
|
||||
| Cloud Monitoring and Logging | `references/monitoring.md` |
|
||||
|
||||
184
skills/google-cloud-iac-patterns/references/load-balancer.md
Normal file
184
skills/google-cloud-iac-patterns/references/load-balancer.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# External Application Load Balancer
|
||||
|
||||
A global HTTPS load balancer in front of a Cloud Run service, with a TLS policy, a web application firewall policy and, optionally, Identity-Aware Proxy.
|
||||
|
||||
The chain is: forwarding rule → target HTTPS proxy → URL map → backend service → serverless network endpoint group → Cloud Run service. Each setting lives on one link of it, and putting one on the wrong link is the usual mistake:
|
||||
|
||||
- **The minimum TLS version is on an SSL policy, attached to the target HTTPS proxy** (`ssl_policy`). A backend service has no `ssl_policy` setting.
|
||||
- **The certificate is on the target HTTPS proxy** (`ssl_certificates`).
|
||||
- **The firewall policy is attached to the backend service** (`security_policy`).
|
||||
- **Identity-Aware Proxy is the `iap` block of the backend service.** It decides who reaches the application; what a signed-in user may do inside it is the application's own code and cannot be shown by infrastructure.
|
||||
- **Request logging is the `log_config` block of the backend service.**
|
||||
- A backend service for Cloud Run uses a serverless NEG and takes no `health_checks`, no `port_name` and no `timeout_sec`.
|
||||
- `load_balancing_scheme = "EXTERNAL_MANAGED"` on both the backend service and the forwarding rule.
|
||||
- Set the Cloud Run service's `ingress` to `INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER`, or it stays reachable around the load balancer.
|
||||
- Redirecting HTTP to HTTPS needs a second URL map, target HTTP proxy and forwarding rule on port 80.
|
||||
|
||||
```hcl
|
||||
variable "project_id" {
|
||||
type = string
|
||||
description = "Project to deploy into."
|
||||
}
|
||||
|
||||
variable "region" {
|
||||
type = string
|
||||
description = "Region for regional resources."
|
||||
}
|
||||
|
||||
variable "domain" {
|
||||
type = string
|
||||
description = "Domain name the load balancer serves."
|
||||
}
|
||||
|
||||
resource "google_cloud_run_v2_service" "app" {
|
||||
project = var.project_id
|
||||
name = "booking-app"
|
||||
location = var.region
|
||||
ingress = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = "us-docker.pkg.dev/cloudrun/container/hello"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_region_network_endpoint_group" "app" {
|
||||
project = var.project_id
|
||||
name = "booking-app"
|
||||
region = var.region
|
||||
network_endpoint_type = "SERVERLESS"
|
||||
|
||||
cloud_run {
|
||||
service = google_cloud_run_v2_service.app.name
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_security_policy" "edge" {
|
||||
project = var.project_id
|
||||
name = "booking-edge"
|
||||
|
||||
rule {
|
||||
action = "deny(403)"
|
||||
priority = 1000
|
||||
match {
|
||||
expr {
|
||||
expression = "evaluatePreconfiguredWaf('sqli-v33-stable', {'sensitivity': 1})"
|
||||
}
|
||||
}
|
||||
description = "Block SQL injection."
|
||||
}
|
||||
|
||||
rule {
|
||||
action = "allow"
|
||||
priority = 2147483647
|
||||
match {
|
||||
versioned_expr = "SRC_IPS_V1"
|
||||
config {
|
||||
src_ip_ranges = ["*"]
|
||||
}
|
||||
}
|
||||
description = "Default rule."
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_backend_service" "app" {
|
||||
project = var.project_id
|
||||
name = "booking-app"
|
||||
protocol = "HTTPS"
|
||||
load_balancing_scheme = "EXTERNAL_MANAGED"
|
||||
security_policy = google_compute_security_policy.edge.id
|
||||
|
||||
backend {
|
||||
group = google_compute_region_network_endpoint_group.app.id
|
||||
}
|
||||
|
||||
# Only signed-in identities granted roles/iap.httpsResourceAccessor get through.
|
||||
iap {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
log_config {
|
||||
enable = true
|
||||
sample_rate = 1.0
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_url_map" "app" {
|
||||
project = var.project_id
|
||||
name = "booking-app"
|
||||
default_service = google_compute_backend_service.app.id
|
||||
}
|
||||
|
||||
resource "google_compute_ssl_policy" "modern" {
|
||||
project = var.project_id
|
||||
name = "tls-1-2-modern"
|
||||
profile = "MODERN"
|
||||
min_tls_version = "TLS_1_2"
|
||||
}
|
||||
|
||||
resource "google_compute_managed_ssl_certificate" "app" {
|
||||
project = var.project_id
|
||||
name = "booking-app"
|
||||
|
||||
managed {
|
||||
domains = [var.domain]
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_target_https_proxy" "app" {
|
||||
project = var.project_id
|
||||
name = "booking-app"
|
||||
url_map = google_compute_url_map.app.id
|
||||
ssl_certificates = [google_compute_managed_ssl_certificate.app.id]
|
||||
ssl_policy = google_compute_ssl_policy.modern.id
|
||||
}
|
||||
|
||||
resource "google_compute_global_address" "app" {
|
||||
project = var.project_id
|
||||
name = "booking-app"
|
||||
}
|
||||
|
||||
resource "google_compute_global_forwarding_rule" "https" {
|
||||
project = var.project_id
|
||||
name = "booking-app-https"
|
||||
load_balancing_scheme = "EXTERNAL_MANAGED"
|
||||
ip_address = google_compute_global_address.app.id
|
||||
port_range = "443"
|
||||
target = google_compute_target_https_proxy.app.id
|
||||
}
|
||||
|
||||
# HTTP to HTTPS.
|
||||
resource "google_compute_url_map" "redirect" {
|
||||
project = var.project_id
|
||||
name = "booking-app-redirect"
|
||||
|
||||
default_url_redirect {
|
||||
https_redirect = true
|
||||
strip_query = false
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_compute_target_http_proxy" "redirect" {
|
||||
project = var.project_id
|
||||
name = "booking-app-redirect"
|
||||
url_map = google_compute_url_map.redirect.id
|
||||
}
|
||||
|
||||
resource "google_compute_global_forwarding_rule" "http" {
|
||||
project = var.project_id
|
||||
name = "booking-app-http"
|
||||
load_balancing_scheme = "EXTERNAL_MANAGED"
|
||||
ip_address = google_compute_global_address.app.id
|
||||
port_range = "80"
|
||||
target = google_compute_target_http_proxy.redirect.id
|
||||
}
|
||||
|
||||
# Who may pass Identity-Aware Proxy.
|
||||
resource "google_iap_web_backend_service_iam_member" "staff" {
|
||||
project = var.project_id
|
||||
web_backend_service = google_compute_backend_service.app.name
|
||||
role = "roles/iap.httpsResourceAccessor"
|
||||
member = "domain:${var.domain}"
|
||||
}
|
||||
```
|
||||
@@ -26,11 +26,14 @@ A list. Number functional requirements `FR-1`, `FR-2` and non-functional ones `N
|
||||
[
|
||||
{"id": "FR-1", "kind": "functional", "text": "Accept signed webhooks over HTTPS."},
|
||||
{"id": "NFR-1", "kind": "non_functional", "text": "Raw events are encrypted at rest and retained for 30 days.",
|
||||
"conditions": ["Raw events are encrypted at rest.", "Raw events are kept for 30 days and then deleted."]}
|
||||
"conditions": ["Raw events are encrypted at rest.", "Raw events are kept for 30 days and then deleted."]},
|
||||
{"id": "NFR-2", "kind": "non_functional", "text": "The service is available 99.9% of each month, across two zones.",
|
||||
"conditions": ["The service runs in at least two zones."],
|
||||
"outside_code": ["Measured availability is at least 99.9% a month."]}
|
||||
]
|
||||
```
|
||||
|
||||
Every non-functional requirement lists its `conditions`: one entry for each separate thing that must be true for it to be met. Write each so it can be checked on its own, and so that together they say everything the text says. The Target architecture needs a Check for each one.
|
||||
Every non-functional requirement lists its `conditions`: one entry for each separate thing that must be true for it to be met and that infrastructure configuration can show. What it asks for that configuration cannot show goes under `outside_code`: an outcome measured in operation, behaviour of the application's own code, or a process people follow. A requirement has at least one entry between the two; one that is entirely outside code has `"conditions": []`. Write each so it can be checked on its own, and so that together they say everything the text says. The Target architecture needs a Check for each one.
|
||||
|
||||
`kind` is `functional` or `non_functional`. Write one requirement per entry, so each can be checked on its own.
|
||||
|
||||
@@ -59,7 +62,7 @@ A list. Each recommendation names its pillar and the components it applies to; u
|
||||
"conditions": ["The topic is encrypted with a customer-managed key.", "The messaging service's own identity may use that key."]}]
|
||||
```
|
||||
|
||||
Every recommendation lists its `conditions`, as a non-functional requirement does. When it applies to several components, give a condition for each.
|
||||
Every recommendation lists its `conditions`, and `outside_code` where it applies, as a non-functional requirement does. When it applies to several components, give a condition for each.
|
||||
|
||||
Write one configurable decision per recommendation. "Harden the service" cannot become a Check; "disable public ingress on the worker service" can.
|
||||
|
||||
@@ -87,7 +90,7 @@ Write one configurable decision per recommendation. "Harden the service" cannot
|
||||
- **Elements.** One per resource to be created. `component` is the component it realises; leave it out for a supporting Element such as an identity, a key or a network. `product` for a component's Element is exactly the product selected for that component.
|
||||
- **Relationships.** `from` and `to` are Element ids. `kind` says what the connection is: calls, publishes to, reads from, runs as, is granted access to.
|
||||
- **Checks.** `applies_to` is an Element or Relationship id. `source` is the requirement or recommendation id the Check comes from. Every Element has at least one Check.
|
||||
- **Coverage.** Every entry in the `conditions` of a non-functional requirement or a recommendation needs a Check that verifies it, with that requirement or recommendation as its `source`. A condition that applies to several Elements needs a Check on each: "processing scales to zero when idle" applies to every processing Element, not one of them.
|
||||
- **Coverage.** Every entry in the `conditions` of a non-functional requirement or a recommendation needs a Check that verifies it, with that requirement or recommendation as its `source`. An entry under `outside_code` gets no Check, and a requirement or recommendation with nothing but `outside_code` needs none. A condition that applies to several Elements needs a Check on each: "processing scales to zero when idle" applies to every processing Element, not one of them.
|
||||
- **Completeness.** Every component has an Element. Every non-functional requirement and every recommendation is the `source` of at least one Check, unless the recommendation is listed under `not_applicable` with a reason. Every selected product appears on its component's Element.
|
||||
|
||||
#### Writing a Check
|
||||
@@ -135,5 +138,5 @@ Implement every Element, Relationship and Check, and nothing the Target architec
|
||||
|
||||
- Give every Check exactly one verdict: `pass`, `fail` or `not_verifiable`, with the evidence you read in the code.
|
||||
- `dry_run.verdict` is `pass`, `fail`, or `not_run` when the tool could not run.
|
||||
- Find every gap in one pass: go through each requirement and recommendation in turn and compare what its text asks with the Checks that name it as `source`.
|
||||
- Find every gap in one pass: go through each requirement and recommendation in turn and compare its `conditions` with the Checks that name it as `source`. Propose nothing for its `outside_code`.
|
||||
- Judge the code against the Checks and nothing else. Something the Target architecture lacks is a `proposed_additions` entry, with the requirement or recommendation id it serves in `reason`. It is never a failed Check.
|
||||
|
||||
@@ -8,11 +8,18 @@ import pytest
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from app.adk.fake_runtime import Call, FakeAgentRuntime, sample_submission
|
||||
from app.adk.fake_runtime import (
|
||||
Call,
|
||||
FakeAgentRuntime,
|
||||
FakeDecider,
|
||||
sample_submission,
|
||||
)
|
||||
from app.adk.roles import load_roles
|
||||
from app.adk.runtime import Answer
|
||||
from app.agent import Assembly, ExecutionEngine
|
||||
from app.config import get_settings
|
||||
from app.main import create_app
|
||||
from app.nodes.intake import Intake
|
||||
from app.providers import load_provider
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind, PhaseId
|
||||
@@ -67,7 +74,9 @@ def complete_run_turns() -> dict:
|
||||
}
|
||||
|
||||
|
||||
def client_for(tmp_path: Path, turns: dict) -> tuple[TestClient, FakeAgentRuntime]:
|
||||
def client_for(
|
||||
tmp_path: Path, turns: dict, decider=None
|
||||
) -> tuple[TestClient, FakeAgentRuntime]:
|
||||
bundle = load_provider("gcp")
|
||||
roles = load_roles(AGENTS_DIR, bundle)
|
||||
runtime = FakeAgentRuntime(turns)
|
||||
@@ -81,7 +90,13 @@ def client_for(tmp_path: Path, turns: dict) -> tuple[TestClient, FakeAgentRuntim
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
)
|
||||
yield Assembly(engine, bundle, roles, {"compute": "API not enabled"})
|
||||
yield Assembly(
|
||||
engine,
|
||||
bundle,
|
||||
roles,
|
||||
{"compute": "API not enabled"},
|
||||
intake=Intake(decider, tmp_path) if decider else None,
|
||||
)
|
||||
|
||||
return TestClient(create_app(get_settings(), fake_assembly)), runtime
|
||||
|
||||
@@ -319,3 +334,56 @@ def test_generate_groups_the_execution_under_the_callers_customer_case(tmp_path)
|
||||
assert "customer_case" in refused.json()["error"]
|
||||
assert accepted.status_code == 200, accepted.json()
|
||||
assert (tmp_path / "lessons" / "candidates" / "acme-webhooks").is_dir()
|
||||
|
||||
|
||||
# --- a message that starts something is read first (intake) ----------------------------
|
||||
|
||||
|
||||
def test_a_question_is_answered_and_starts_no_execution(tmp_path):
|
||||
decider = FakeDecider({"intake": Answer("question", 0.97)})
|
||||
client, runtime = client_for(tmp_path, {}, decider)
|
||||
with client:
|
||||
task = send(client, [{"text": "What can you do?"}])
|
||||
|
||||
text, data = status_parts(task)
|
||||
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
|
||||
assert "Nothing was started" in text
|
||||
assert data == {"status": "answered", "intake": "question"}
|
||||
assert runtime.runs == []
|
||||
assert not (tmp_path / "deliverables").exists()
|
||||
|
||||
|
||||
def test_a_new_request_starts_an_execution_as_before(tmp_path):
|
||||
decider = FakeDecider({"intake": Answer("new_request", 0.99)})
|
||||
turns = {"orchestrator": [[Call("ask_user", {"questions": ["Which region?"]})]]}
|
||||
client, runtime = client_for(tmp_path, turns, decider)
|
||||
with client:
|
||||
task = send(client, [{"text": "Design an ingestion pipeline."}])
|
||||
|
||||
assert task["status"]["state"] == "TASK_STATE_INPUT_REQUIRED"
|
||||
assert "Which region?" in status_parts(task)[0]
|
||||
assert len(runtime.runs) == 1
|
||||
|
||||
|
||||
def test_an_unsure_reading_starts_an_execution(tmp_path):
|
||||
decider = FakeDecider({"intake": Answer("question", 0.5)})
|
||||
turns = {"orchestrator": [[Call("ask_user", {"questions": ["Which region?"]})]]}
|
||||
client, runtime = client_for(tmp_path, turns, decider)
|
||||
with client:
|
||||
task = send(client, [{"text": "hello"}])
|
||||
|
||||
assert task["status"]["state"] == "TASK_STATE_INPUT_REQUIRED"
|
||||
assert len(runtime.runs) == 1
|
||||
|
||||
|
||||
def test_a_pre_approved_task_is_not_read_as_a_question(tmp_path):
|
||||
decider = FakeDecider({"intake": Answer("question", 0.99)})
|
||||
client, runtime = client_for(tmp_path, complete_run_turns(), decider)
|
||||
with client:
|
||||
task = send(
|
||||
client, [{"text": "What can you do?"}], metadata={"pre_approved": True}
|
||||
)
|
||||
|
||||
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
|
||||
assert decider.asked == []
|
||||
assert runtime.runs
|
||||
|
||||
@@ -199,9 +199,11 @@ def test_benchmark_cases_load_with_valid_answer_patterns():
|
||||
assert len(cases) >= 2
|
||||
for case in cases:
|
||||
assert case.required_products
|
||||
for pattern, answer in case.scripted_answers:
|
||||
for pattern, answer, *asker in case.scripted_answers:
|
||||
re.compile(pattern)
|
||||
assert answer
|
||||
for who in asker:
|
||||
re.compile(who)
|
||||
|
||||
|
||||
# --- checks that can see quality (iteration 4, step 8) ---------------------------------
|
||||
@@ -528,3 +530,87 @@ async def test_a_judge_that_gives_no_answer_leaves_the_check_unanswered_not_pass
|
||||
"judge unavailable: Jev gave no answer"
|
||||
)
|
||||
assert not check(result, "recommendations_quality").passed
|
||||
|
||||
|
||||
# --- the scripted user (evaluation case 04) --------------------------------------------
|
||||
|
||||
|
||||
def test_a_scripted_fact_is_given_only_to_who_it_was_written_for():
|
||||
from app.nodes.execution_nodes import automatic_reply
|
||||
|
||||
scripted = [
|
||||
["region|location", "europe-west1.", "requirements"],
|
||||
["approve|confirm", "Approved. Please proceed."],
|
||||
]
|
||||
|
||||
def reply(asker: str, question: str) -> str:
|
||||
prompt = {"questions": [{"from": asker, "question": question}]}
|
||||
return automatic_reply(prompt, scripted)["reply"]
|
||||
|
||||
assert reply("requirements", "Which region?").endswith("A: europe-west1.")
|
||||
# A later Specialist confirming a draft that mentions the region gets an approval.
|
||||
assert reply(
|
||||
"validation_results", "Region is europe-west1. Please confirm my draft."
|
||||
).endswith("A: Approved. Please proceed.")
|
||||
# With no answer for it, the reply says no user is available.
|
||||
assert "No user is available" in reply("product_selection", "Which region?")
|
||||
|
||||
|
||||
def test_case_04_answers_a_draft_confirmation_with_an_approval():
|
||||
from app.nodes.execution_nodes import automatic_reply
|
||||
from eval.scoring import load_cases
|
||||
|
||||
cases = load_cases(Path(__file__).parents[2] / "eval/datasets/solution_cases.json")
|
||||
case = next(c for c in cases if c.id == "case-04-interactive-web-app")
|
||||
|
||||
def reply(asker: str, question: str) -> str:
|
||||
prompt = {"questions": [{"from": asker, "question": question}]}
|
||||
return automatic_reply(prompt, [list(a) for a in case.scripted_answers])[
|
||||
"reply"
|
||||
]
|
||||
|
||||
question = "I checked encryption, GDPR and password rotation. Please confirm."
|
||||
assert reply("validation_results", question).endswith(
|
||||
"A: Approved. Please proceed."
|
||||
)
|
||||
assert "europe-west1" in reply("requirements", "Which region should it run in?")
|
||||
|
||||
|
||||
# --- calibrating the judges ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_calibration_reports_agreement_stability_and_between_judges():
|
||||
from eval.calibrate_judge import report
|
||||
|
||||
items = [
|
||||
{"id": "a", "document": "d.md", "kind": "yes_no", "text": "?", "label": 1},
|
||||
{"id": "b", "document": "d.md", "kind": "graded", "text": "?", "label": 0.67},
|
||||
{"id": "c", "document": "d.md", "kind": "yes_no", "text": "?", "label": None},
|
||||
]
|
||||
by_judge = {
|
||||
"jev": {"a": [1.0, 1.0], "b": [0.33, 0.67], "c": [0.0, 0.0]},
|
||||
"gemini": {"a": [1.0, 1.0], "b": [0.67, 0.67], "c": [1.0, 1.0]},
|
||||
}
|
||||
|
||||
result = report(items, by_judge)
|
||||
|
||||
assert result["questions"] == 3 and result["labelled"] == 2
|
||||
assert result["judges"]["jev"]["agreement_with_labels"] == 0.5
|
||||
assert result["judges"]["gemini"]["agreement_with_labels"] == 1.0
|
||||
assert result["judges"]["jev"]["stability"] == 0.67
|
||||
assert result["judges"]["gemini"]["stability"] == 1.0
|
||||
assert result["between_judges"] == 0.33
|
||||
|
||||
|
||||
def test_calibration_without_labels_still_measures_stability():
|
||||
from eval.calibrate_judge import report
|
||||
|
||||
items = [
|
||||
{"id": "a", "document": "d.md", "kind": "yes_no", "text": "?", "label": None}
|
||||
]
|
||||
|
||||
result = report(items, {"jev": {"a": [1.0, 1.0, 1.0]}})
|
||||
|
||||
assert result["labelled"] == 0
|
||||
assert result["judges"]["jev"]["agreement_with_labels"] is None
|
||||
assert result["judges"]["jev"]["stability"] == 1.0
|
||||
|
||||
@@ -399,3 +399,44 @@ async def test_feedback_on_the_description_itself_has_it_written_again(tmp_path)
|
||||
assert asked["working"]["architecture_description"][STANDING]["feedback"] == [
|
||||
"Make the flow clearer."
|
||||
]
|
||||
|
||||
|
||||
# --- requirements name no products ------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_requirement_that_names_a_product_is_raised():
|
||||
from app.tools.requirements_review import RequirementsReview
|
||||
|
||||
requirements = [
|
||||
{"id": "FR-1", "kind": "functional", "text": "Run the API on Cloud Run."},
|
||||
{"id": "FR-2", "kind": "functional", "text": "Accept signed webhooks."},
|
||||
]
|
||||
decider = FakeDecider({"FR-1": Answer(0.93, 0.86), "FR-2": Answer(0.05, 0.9)})
|
||||
|
||||
raised = await RequirementsReview(decider)(
|
||||
K.REQUIREMENTS, {"requirements.json": json.dumps(requirements)}, None, "e1"
|
||||
)
|
||||
|
||||
assert len(raised) == 1 and raised[0].startswith("FR-1 names a product")
|
||||
_, questions, scope = decider.asked[0]
|
||||
assert set(questions) == {"FR-1", "FR-2"} and scope == "e1"
|
||||
assert "Accept signed webhooks." in questions["FR-2"].instructions
|
||||
|
||||
|
||||
async def test_the_requirements_review_looks_only_at_requirements_and_never_blocks():
|
||||
from app.tools.requirements_review import RequirementsReview
|
||||
|
||||
decider = FakeDecider({})
|
||||
review = RequirementsReview(decider)
|
||||
|
||||
assert await review(K.TARGET_ARCHITECTURE, GRAPH, UPSTREAM) == ()
|
||||
assert await review(K.REQUIREMENTS, {"requirements.json": "not json"}, None) == ()
|
||||
assert decider.asked == []
|
||||
assert (
|
||||
await RequirementsReview(NoDecider())(
|
||||
K.REQUIREMENTS,
|
||||
{"requirements.json": SAMPLE_DATA["requirements.json"]},
|
||||
None,
|
||||
)
|
||||
== ()
|
||||
)
|
||||
|
||||
@@ -488,12 +488,21 @@ async def test_pre_approved_escalation_ends_with_warnings(tmp_path):
|
||||
|
||||
assert result.status == "completed_with_warnings"
|
||||
assert "revision limit" in result.state["warnings"][0]
|
||||
# No Deliverable files are written, but the request stays on record.
|
||||
# What was produced is kept, marked as unfinished, and the request is on record.
|
||||
case = tmp_path / "deliverables" / "cases" / "unassigned"
|
||||
assert not (case / "executions").exists()
|
||||
[intent] = json.loads((case / "intent" / "history.json").read_text())["intents"]
|
||||
assert intent["status"] == "completed_with_warnings"
|
||||
assert intent["execution"] is None
|
||||
assert intent["execution"] == f"executions/{result.execution_id}"
|
||||
out_dir = case / intent["execution"]
|
||||
manifest = json.loads((out_dir / "execution.json").read_text())
|
||||
assert manifest["status"] == "completed_with_warnings"
|
||||
assert "revision limit" in manifest["warnings"][0]
|
||||
assert manifest["solution_guide"] is None
|
||||
assert not (out_dir / "solution-architecture-guide.md").exists()
|
||||
assert (out_dir / "deliverables" / "requirements.md").is_file()
|
||||
# Nothing unapproved reads as final: each Deliverable and Phase has its status.
|
||||
assert manifest["deliverables"]["requirements"]["status"] == "submitted"
|
||||
assert manifest["phases"]["packaging"] != "approved"
|
||||
|
||||
|
||||
# --- budgets -------------------------------------------------------------------------------------
|
||||
@@ -880,7 +889,10 @@ async def test_an_execution_in_a_customer_case_writes_under_that_case(tmp_path):
|
||||
"terraform/main.tf",
|
||||
"terraform/variables.tf",
|
||||
],
|
||||
"status": "submitted",
|
||||
}
|
||||
assert manifest["status"] == "completed" and manifest["warnings"] == []
|
||||
assert set(manifest["phases"].values()) == {"approved"}
|
||||
for entry in manifest["deliverables"].values():
|
||||
assert (out / entry["document"]).is_file()
|
||||
assert all((out / name).is_file() for name in entry["attachments"])
|
||||
@@ -969,3 +981,92 @@ async def test_the_intent_is_on_record_as_soon_as_an_execution_starts(tmp_path):
|
||||
assert entry["status"] == "in_progress"
|
||||
assert entry["statement"] == "intent/001.md"
|
||||
assert entry["execution_id"] == paused.execution_id
|
||||
|
||||
|
||||
# --- a user sees a Deliverable before it is first submitted -----------------------------
|
||||
|
||||
|
||||
async def test_with_a_user_present_a_first_submission_waits_for_their_reply(tmp_path):
|
||||
"""Evaluation case 04: a Specialist that ran out of budget submitted without asking."""
|
||||
runtime = FakeAgentRuntime(
|
||||
{
|
||||
"orchestrator": [
|
||||
[delegate(K.REQUIREMENTS)],
|
||||
[ask_user()],
|
||||
[delegate(K.REQUIREMENTS)],
|
||||
[ask_user("Anything else?")],
|
||||
],
|
||||
specialist(K.REQUIREMENTS): [
|
||||
# It tries to submit straight away, is refused, and asks.
|
||||
[
|
||||
sample_submission(K.REQUIREMENTS),
|
||||
Call(
|
||||
"needs_input",
|
||||
{"questions": ["Please approve the draft."], "draft": "# r"},
|
||||
),
|
||||
],
|
||||
submits(K.REQUIREMENTS),
|
||||
],
|
||||
}
|
||||
)
|
||||
engine = make_engine(tmp_path, runtime, confirm_drafts=True)
|
||||
|
||||
first = await engine.start("Ingestion")
|
||||
refusal = runtime.runs[1].results[0][1]
|
||||
await engine.resume(first.execution_id, {"reply": "Approved."})
|
||||
|
||||
assert refusal.startswith("Not submitted. The user has not seen this Deliverable")
|
||||
assert first.prompt["questions"][0]["question"] == "Please approve the draft."
|
||||
# Once they have replied, the submission is accepted.
|
||||
assert runtime.runs[-2].results[0][1].startswith("Submitted")
|
||||
|
||||
|
||||
async def test_a_pre_approved_execution_is_never_made_to_ask(tmp_path):
|
||||
runtime = FakeAgentRuntime(happy_path_turns())
|
||||
|
||||
result = await make_engine(tmp_path, runtime, confirm_drafts=True).start(
|
||||
"Ingestion", pre_approved=True
|
||||
)
|
||||
|
||||
assert result.status == "completed"
|
||||
|
||||
|
||||
async def test_answering_another_question_is_not_being_shown_the_draft(tmp_path):
|
||||
"""Evaluation case 04, 2026-10-08: a Specialist that stopped, was asked about by
|
||||
the Orchestrator and then submitted. The user had answered a question about the
|
||||
Deliverable but had never seen it."""
|
||||
draft = Call(
|
||||
"needs_input", {"questions": ["Please approve the draft."], "draft": "# r"}
|
||||
)
|
||||
runtime = FakeAgentRuntime(
|
||||
{
|
||||
"orchestrator": [
|
||||
[delegate(K.REQUIREMENTS)],
|
||||
[ask_user()], # relays "the requirements analyst stopped"
|
||||
[delegate(K.REQUIREMENTS)],
|
||||
[ask_user()], # relays the draft
|
||||
[delegate(K.REQUIREMENTS)],
|
||||
[ask_user("Anything else?")],
|
||||
],
|
||||
specialist(K.REQUIREMENTS): [
|
||||
[], # stops without ending its turn
|
||||
[sample_submission(K.REQUIREMENTS), draft],
|
||||
submits(K.REQUIREMENTS),
|
||||
],
|
||||
}
|
||||
)
|
||||
engine = make_engine(tmp_path, runtime, confirm_drafts=True)
|
||||
|
||||
first = await engine.start("Ingestion")
|
||||
assert "stopped" in first.prompt["questions"][0]["question"]
|
||||
second = await engine.resume(first.execution_id, {"reply": "Try again."})
|
||||
await engine.resume(first.execution_id, {"reply": "Approved."})
|
||||
|
||||
analyst = [run for run in runtime.runs if run.role == specialist(K.REQUIREMENTS)]
|
||||
assert (
|
||||
analyst[1]
|
||||
.results[0][1]
|
||||
.startswith("Not submitted. The user has not seen this Deliverable")
|
||||
)
|
||||
assert second.prompt["questions"][0]["question"] == "Please approve the draft."
|
||||
assert analyst[2].results[0][1].startswith("Submitted")
|
||||
|
||||
58
tests/execution/test_intake.py
Normal file
58
tests/execution/test_intake.py
Normal file
@@ -0,0 +1,58 @@
|
||||
"""Intake: what a message that starts something is asking for."""
|
||||
|
||||
import json
|
||||
|
||||
from app.adk.fake_runtime import FakeDecider
|
||||
from app.adk.runtime import Answer, NoDecider
|
||||
from app.nodes.intake import CHANGE_REQUEST, NEW_REQUEST, QUESTION, Intake
|
||||
|
||||
|
||||
def delivered(tmp_path, case: str, statuses: list[str]) -> None:
|
||||
folder = tmp_path / "deliverables" / "cases" / case / "intent"
|
||||
folder.mkdir(parents=True)
|
||||
intents = [
|
||||
{"version": n, "workflow_request": f"Design thing {n}.", "status": status}
|
||||
for n, status in enumerate(statuses, start=1)
|
||||
]
|
||||
(folder / "history.json").write_text(
|
||||
json.dumps({"customer_case": case, "intents": intents})
|
||||
)
|
||||
|
||||
|
||||
async def test_without_a_decision_every_message_is_a_new_request(tmp_path):
|
||||
reading = await Intake(NoDecider(), tmp_path).read("What can you do?")
|
||||
|
||||
assert reading.kind == NEW_REQUEST and not reading.judged
|
||||
|
||||
|
||||
async def test_a_confident_question_is_read_as_one(tmp_path):
|
||||
decider = FakeDecider({"intake": Answer(QUESTION, 0.98)})
|
||||
|
||||
reading = await Intake(decider, tmp_path).read("How long does it take?", scope="t1")
|
||||
|
||||
state, questions, scope = decider.asked[0]
|
||||
assert reading.kind == QUESTION and reading.judged
|
||||
assert scope == "t1"
|
||||
assert questions["intake"].options == (NEW_REQUEST, CHANGE_REQUEST, QUESTION)
|
||||
assert state.endswith("The user's message: How long does it take?")
|
||||
|
||||
|
||||
async def test_a_reading_that_is_not_sure_enough_starts_new_work(tmp_path):
|
||||
decider = FakeDecider({"intake": Answer(QUESTION, 0.6)})
|
||||
|
||||
assert (await Intake(decider, tmp_path, act=0.75).read("hello")).kind == NEW_REQUEST
|
||||
|
||||
|
||||
async def test_a_change_needs_something_delivered_to_change(tmp_path):
|
||||
decider = FakeDecider({"intake": Answer(CHANGE_REQUEST, 0.99)})
|
||||
delivered(tmp_path, "acme", ["completed", "in_progress"])
|
||||
intake = Intake(decider, tmp_path)
|
||||
|
||||
with_history = await intake.read("Raise availability to 99.99%.", "acme")
|
||||
without = await intake.read("Raise availability to 99.99%.", "new-client")
|
||||
no_case = await intake.read("Raise availability to 99.99%.")
|
||||
|
||||
assert with_history.kind == CHANGE_REQUEST and with_history.earlier == 1
|
||||
assert "- Design thing 1." in decider.asked[0][0]
|
||||
assert "Design thing 2." not in decider.asked[0][0] # still in progress
|
||||
assert without.kind == NEW_REQUEST and no_case.kind == NEW_REQUEST
|
||||
@@ -308,3 +308,103 @@ async def test_pillar_advice_survives_a_revision_through_the_engine(tmp_path):
|
||||
assert "## reliability (from your earlier consultation)" in second.results[0][1]
|
||||
stored = result.state["working"]["design_recommendations"]["pillar_advice"]
|
||||
assert set(stored) == set(PILLARS)
|
||||
|
||||
|
||||
# --- each advisor reviews its pillar's checklist ----------------------------------------
|
||||
|
||||
|
||||
def reviewed(task):
|
||||
pillar = task.context["pillar"]
|
||||
return [
|
||||
Call(
|
||||
"submit_advice",
|
||||
{
|
||||
"content": f"Use strong defaults for {pillar}.",
|
||||
"checklist": [
|
||||
{
|
||||
"item": f"{pillar} item one | with a bar",
|
||||
"status": "addressed",
|
||||
"note": "Covered by the first recommendation.",
|
||||
},
|
||||
{
|
||||
"item": f"{pillar} item two",
|
||||
"status": "not_applicable",
|
||||
"note": "An organisation's process, not a design decision.",
|
||||
},
|
||||
{"item": "malformed", "status": "maybe", "note": "dropped"},
|
||||
],
|
||||
},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
async def test_each_advisors_checklist_review_is_kept_beside_its_advice():
|
||||
from app.tools.pillars import CHECKLIST_KEY
|
||||
|
||||
memory: dict = {}
|
||||
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [reviewed] * 3})
|
||||
tool = pillar_consultation_tool(runtime, ADVISOR, PILLARS, memory=memory)
|
||||
|
||||
result = await consult(tool, brief="b")
|
||||
|
||||
# The design advisor is given the advice, not the review: it does not copy it.
|
||||
assert "item one" not in result.message
|
||||
assert set(memory[CHECKLIST_KEY]) == set(PILLARS)
|
||||
review = memory[CHECKLIST_KEY]["security"]
|
||||
assert [entry["status"] for entry in review] == ["addressed", "not_applicable"]
|
||||
|
||||
|
||||
async def test_the_review_is_written_into_the_deliverable_by_code():
|
||||
from app.adk.fake_runtime import sample_submission
|
||||
from app.adk.runtime import Submitted
|
||||
from app.tools.execution import specialist_tools
|
||||
from app.tools.pillars import REVIEW_HEADING
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
memory: dict = {}
|
||||
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [reviewed] * 3})
|
||||
await consult(
|
||||
pillar_consultation_tool(runtime, ADVISOR, PILLARS, memory=memory), brief="b"
|
||||
)
|
||||
kind = DeliverableKind.DESIGN_RECOMMENDATIONS
|
||||
(submit,) = (
|
||||
tool
|
||||
for tool in specialist_tools(kind, (), allow_cloud_read=False, working=memory)
|
||||
if tool.name == "submit_deliverable"
|
||||
)
|
||||
call = sample_submission(kind)
|
||||
|
||||
first = await submit.handler(**call.args)
|
||||
# A Revision submits content that already carries the section.
|
||||
again = await submit.handler(
|
||||
content=first.outcome.content, attachments=call.args["attachments"]
|
||||
)
|
||||
|
||||
assert isinstance(first.outcome, Submitted)
|
||||
content = first.outcome.content
|
||||
assert f"## {REVIEW_HEADING}" in content and "### security" in content
|
||||
assert (
|
||||
"| security item one \\| with a bar | Addressed | Covered by the first "
|
||||
"recommendation. |"
|
||||
) in content
|
||||
assert "| security item two | Does not apply |" in content
|
||||
assert "malformed" not in content
|
||||
assert again.outcome.content.count(f"## {REVIEW_HEADING}") == 1
|
||||
|
||||
|
||||
async def test_with_no_review_nothing_is_added():
|
||||
from app.adk.fake_runtime import sample_submission
|
||||
from app.tools.execution import specialist_tools
|
||||
from app.tools.pillars import REVIEW_HEADING
|
||||
from app.workflows.deliverables import DeliverableKind
|
||||
|
||||
kind = DeliverableKind.DESIGN_RECOMMENDATIONS
|
||||
(submit,) = (
|
||||
tool
|
||||
for tool in specialist_tools(kind, (), allow_cloud_read=False, working={})
|
||||
if tool.name == "submit_deliverable"
|
||||
)
|
||||
|
||||
result = await submit.handler(**sample_submission(kind).args)
|
||||
|
||||
assert REVIEW_HEADING not in result.outcome.content
|
||||
|
||||
@@ -689,13 +689,11 @@ def test_a_non_functional_requirement_must_list_its_conditions():
|
||||
{"requirements.json": json.dumps(requirements)},
|
||||
)
|
||||
|
||||
needs = (
|
||||
"needs `conditions`: a list with one entry for each separate thing it asks for"
|
||||
)
|
||||
assert missing == (
|
||||
needs = "needs `conditions`:"
|
||||
assert [line.split(" a list")[0] for line in missing] == [
|
||||
f"requirements.json: NFR-1 {needs}",
|
||||
f"requirements.json: NFR-2 {needs}",
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_a_recommendation_must_list_its_conditions():
|
||||
@@ -709,7 +707,61 @@ def test_a_recommendation_must_list_its_conditions():
|
||||
{"recommendations.json": json.dumps(recommendations)},
|
||||
)
|
||||
|
||||
needs = (
|
||||
"needs `conditions`: a list with one entry for each separate thing it asks for"
|
||||
needs = "needs `conditions`:"
|
||||
assert [line.split(" a list")[0] for line in missing] == [
|
||||
f"recommendations.json: DR-1 {needs}"
|
||||
]
|
||||
|
||||
|
||||
def test_what_code_cannot_show_is_listed_apart_and_needs_no_check():
|
||||
"""An outcome measured in operation is recorded, and no Check is asked for it."""
|
||||
requirements = json.loads(SAMPLE_DATA["requirements.json"])
|
||||
requirements.append(
|
||||
{
|
||||
"id": "NFR-2",
|
||||
"kind": "non_functional",
|
||||
"text": "Pages load within one second at p95.",
|
||||
"conditions": [],
|
||||
"outside_code": ["Measured p95 page load is under one second."],
|
||||
}
|
||||
)
|
||||
assert missing == (f"recommendations.json: DR-1 {needs}",)
|
||||
args = sample_submission(K.REQUIREMENTS).args
|
||||
|
||||
accepted = check_submission(
|
||||
K.REQUIREMENTS,
|
||||
args["content"],
|
||||
{"requirements.json": json.dumps(requirements)},
|
||||
)
|
||||
gaps = chain_gaps(
|
||||
parse_graph(SAMPLE_DATA["target-architecture.json"]),
|
||||
requirements=requirements,
|
||||
components=json.loads(SAMPLE_DATA["components.json"]),
|
||||
products=json.loads(SAMPLE_DATA["products.json"]),
|
||||
recommendations=json.loads(SAMPLE_DATA["recommendations.json"]),
|
||||
)
|
||||
|
||||
assert accepted == ()
|
||||
assert gaps == ()
|
||||
|
||||
|
||||
def test_a_constraint_with_conditions_still_needs_a_check():
|
||||
requirements = json.loads(SAMPLE_DATA["requirements.json"])
|
||||
requirements.append(
|
||||
{
|
||||
"id": "NFR-2",
|
||||
"kind": "non_functional",
|
||||
"text": "Two zones, 99.9%.",
|
||||
"conditions": ["The service runs in two zones."],
|
||||
"outside_code": ["Measured availability is 99.9%."],
|
||||
}
|
||||
)
|
||||
|
||||
gaps = chain_gaps(
|
||||
parse_graph(SAMPLE_DATA["target-architecture.json"]),
|
||||
requirements=requirements,
|
||||
components=json.loads(SAMPLE_DATA["components.json"]),
|
||||
products=json.loads(SAMPLE_DATA["products.json"]),
|
||||
recommendations=json.loads(SAMPLE_DATA["recommendations.json"]),
|
||||
)
|
||||
|
||||
assert gaps == ("Requirement NFR-2 has no Check",)
|
||||
|
||||
@@ -529,6 +529,7 @@ async def test_a_free_text_reply_that_asks_for_a_change_is_acted_on(tmp_path):
|
||||
"asks_change": Answer(0.93, 0.86),
|
||||
"change_target": Answer("requirements", 0.9),
|
||||
"change_clarity": Answer(1.9, 0.8),
|
||||
"already_agreed": Answer(0.03, 0.94),
|
||||
}
|
||||
)
|
||||
engine, _ = engine_for(tmp_path, turns, decider)
|
||||
@@ -755,3 +756,46 @@ async def test_asking_the_user_again_and_again_with_nothing_done_stops_the_run(
|
||||
assert asked == MAX_IDLE_ASKS + 1
|
||||
assert "asked" in result.state["warnings"][-1]
|
||||
assert "no work done in between" in result.state["warnings"][-1]
|
||||
|
||||
|
||||
async def test_a_reply_that_restates_what_is_agreed_is_an_answer_not_a_change(
|
||||
tmp_path,
|
||||
):
|
||||
"""Evaluation case 04: a requirement said again in reply to a later question."""
|
||||
from app.nodes.router import Readings, TurnView
|
||||
from app.states.ledger import Ledger
|
||||
|
||||
ledger = Ledger()
|
||||
ledger.start([K.REQUIREMENTS], actor="orchestrator")
|
||||
ledger.submit(K.REQUIREMENTS, 1, actor="requirements-analyst")
|
||||
view = TurnView(
|
||||
execution_id="e1",
|
||||
pre_approved=False,
|
||||
ledger=ledger,
|
||||
deliverables={
|
||||
"requirements": {
|
||||
"attachments": {"requirements.json": SAMPLE_DATA["requirements.json"]}
|
||||
}
|
||||
},
|
||||
pending_questions=[],
|
||||
outcomes=[],
|
||||
last_reply={"kind": "reply", "text": "99.9% availability.", "asked": []},
|
||||
)
|
||||
asks = {
|
||||
"asks_change": Answer(0.9, 0.8),
|
||||
"change_target": Answer("requirements", 0.95),
|
||||
"change_clarity": Answer(1.9, 0.8),
|
||||
}
|
||||
|
||||
restated, changed = Readings(), Readings()
|
||||
decider = FakeDecider({**asks, "already_agreed": Answer(0.85, 0.7)})
|
||||
await Router(decider, None, Thresholds())._read_message(view, restated)
|
||||
await Router(
|
||||
FakeDecider({**asks, "already_agreed": Answer(0.02, 0.96)}), None, Thresholds()
|
||||
)._read_message(view, changed)
|
||||
|
||||
assert restated.change == "close"
|
||||
assert changed.change == "act"
|
||||
state, questions, _ = decider.asked[0]
|
||||
assert "The agreed requirements:" in state and "NFR-1" in state
|
||||
assert "already_agreed" in questions
|
||||
|
||||
Reference in New Issue
Block a user