Files
gcp_solution_architecture_a…/app/tools/assessment.py
Jonathan Boniface 567aede5ca feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)
Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
  Skill from the folders the deployment configures; recorded with its version.
  An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
  one without, and triage of evidence an existing lesson already covers.

Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
  its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
  the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
  real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
  the scores kept in its record. The evaluation takes its rubric from the same
  place.

Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 17:45:56 +01:00

67 lines
2.4 KiB
Python

"""Every Execution judged as it ends (ADR-0007, iteration 5 step 19).
The evaluation scores benchmark cases. This asks the same graded questions of a live
Execution's design recommendations when it ends, and keeps the answers with its
record. It reports; it never stops or changes an Execution.
Without a decision an Execution carries no assessment.
"""
from __future__ import annotations
import logging
from collections.abc import Mapping
from typing import Any
from app.adk.runtime import Decider, Question
logger = logging.getLogger(__name__)
GRADES = ("not at all", "weakly", "mostly", "fully")
QUALITY_BAR = 0.67 # mean of the graded rubric: "mostly" or better
QUALITY_RUBRIC: Mapping[str, str] = {
"specific": "Are the recommendations specific to this workload and its stated "
"requirements, as opposed to generic best practice that would fit any workload?",
"actionable": "Does each recommendation name the concrete setting, feature or "
"configuration to apply, so an engineer could act on it without more research?",
"trade_offs": "Where pillars pull in different directions (for example cost "
"against reliability), is the trade-off named and a resolution recommended?",
"traceable": "Can each recommendation be traced to a requirement or constraint "
"of the workload?",
}
_JUDGED = "design_recommendations"
async def assess(
decider: Decider,
deliverables: Mapping[str, Mapping[str, Any]],
*,
scope: str = "",
) -> dict[str, Any] | None:
"""The rubric's scores for the design recommendations, each from 0 to 1, with
their mean and whether it meets the bar. None when there is nothing to judge or
no decision."""
document = str((deliverables.get(_JUDGED) or {}).get("content", ""))
if not document.strip():
return None
answers = await decider.decide(
document,
{key: Question("score", text, GRADES) for key, text in QUALITY_RUBRIC.items()},
scope=scope,
)
if answers is None:
return None
top = len(GRADES) - 1
scores = {
key: round(min(1.0, max(0.0, float(answers[key].value) / top)), 2)
for key in QUALITY_RUBRIC
}
mean = round(sum(scores.values()) / len(scores), 2)
if mean < QUALITY_BAR:
logger.warning("[%s] design recommendations judged at %.2f", scope, mean)
return {
"recommendations_quality": scores,
"mean": mean,
"meets_bar": mean >= QUALITY_BAR,
}