Commit Graph

4 Commits

Author SHA1 Message Date
1cf1234143 fix: an unimplementable Check is corrected everywhere it applies, in one Revision
The reason sent back to the target architect says to read every Check against
it, and the IaC author records every Check that cannot be met in one
submission. Records the investigation of why the Target architecture is
revised: two causes, about half each (iteration 6 step 12).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 11:51:46 +01:00
654370bb46 feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:32:57 +01:00
d41607f505 fix: read only the user's own words, and stop the router looping at a limit
Case 04, the first interactive evaluation, stopped at 100 Orchestrator
turns on two router faults:

- A reply that quoted the agent's question back was judged on that question.
  A reply now reaches the router with quoted questions and Q:/A: labels
  removed, and the questions are given to the decider as background.
- At a Revision limit the reply was read as another change request for the
  same Deliverable, and the refusal repeated. The user is now told once,
  the reply is taken as the answer, that Deliverable is not reopened again,
  and the Execution carries on with it as it stands.
- Any run that asks the user four times running with no work done in
  between now ends with a warning.

The validator is told to confirm with the user in an interactive Execution.

Plans record the fixes and the rerun, which did not finish: both fixes
held, and a third fault appeared that is not fixed here. A scripted answer
restating a requirement, given to a draft-confirmation question, reopens
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:05:03 +01:00
5bb22d984e fixed multile issues
Some checks failed
validation / verify (push) Failing after 8s
2026-10-07 17:33:36 +01:00