Files
gcp_solution_architecture_a…/.agents/plans/iter-05-jev-decisions/steps/step-19.md
Jonathan Boniface 567aede5ca feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)
Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
  Skill from the folders the deployment configures; recorded with its version.
  An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
  one without, and triage of evidence an existing lesson already covers.

Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
  its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
  the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
  real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
  the scores kept in its record. The evaluation takes its rubric from the same
  place.

Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 17:45:56 +01:00

1.7 KiB

Step 19 — Implementation Detail

Built on 2026-10-09, on by default (JEV_ASSESS_EXECUTIONS). No Execution has run with it.

  1. app/tools/assessment.py holds the graded rubric the evaluation uses for design recommendations (specific, actionable, trade-offs, traceable) and the bar (a mean of 0.67). eval/scoring.py and eval/judge.py now take the rubric, the bar and the grades from there, so a live Execution and a benchmark case are asked the same questions.
  2. When an Execution ends, completed or with warnings, its design recommendations are put to the decider once. The four scores, their mean and whether it meets the bar are kept in the Execution's state and in execution.json under assessment. A mean below the bar is logged as a warning.
  3. It reports only. Nothing stops or changes, and a failure to assess leaves the Execution with no assessment.
  4. Risk flags (step 14) are already in execution.json under risks.

Differs from the plan:

  • Only the quality rubric is asked. Checklist coverage is already settled inside every Execution by the pillar checklist review (iteration 4), in code. The structural checks need a case's expected products, which a live Execution does not have.
  • Nothing is sent as feedback on a trace. That part waits on step 17 and the owner's decision on LangSmith.
  • No alerting. A low score is a log line and a field in the record; nothing watches for it.

The judge behind it was measured in step 16: Jev answered the same question the same way every time (1.00) and agreed with Gemini on 0.91 of questions. Agreement with a person still awaits the owner's 32 labels.

Tests: tests/execution/test_assessment.py (5).