Commit Graph

4 Commits

Author SHA1 Message Date
7ee7054523 docs: record case 06 passing after the verdict and naming fixes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 15:23:34 +01:00
c4c8a1f94e fix: a Check is judged by its own words, and a wrong verdict can be disputed
Some checks failed
validation / verify (push) Failing after 10s
Case 06 reached its Revision limit because the validator failed a Check that
a role is granted, for the identity also holding other roles, and the IaC
author could only answer by recording the Check as unimplementable. The
validator is told what its own words require; the author is told to dispute
such a verdict and change nothing.

The evaluation's naming standard for case 06 accepted only a variable inside
each name; it now accepts names built from a local as well.

Records the reruns: case 04 passed at 12 of 12, case 06 ended with warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 14:40:31 +01:00
e5e1647ff1 feat: prove the lessons, check overlays in evaluation, and settle the cost steps
Some checks failed
validation / verify (push) Failing after 13s
- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:27:09 +01:00
567aede5ca feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)
Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
  Skill from the folders the deployment configures; recorded with its version.
  An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
  one without, and triage of evidence an existing lesson already covers.

Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
  its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
  the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
  real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
  the scores kept in its record. The evaluation takes its rubric from the same
  place.

Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 17:45:56 +01:00