feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)
Iteration 4: - Overlay Skills (step 12): an Execution names an organisation's standards Skill from the folders the deployment configures; recorded with its version. An example standards Skill and an evaluation case that uses it. - Lessons (step 10): a prove command that compares a run with lessons against one without, and triage of evidence an existing lesson already covers. Iteration 5: - Step 8: the references of a Specialist's Skills its task needs are named in its context as read_first. - Step 14: a remote MCP server's reply that instructs the agent is withheld; the user's own text is flagged as a risk and nothing more. - Step 15: in a Change request, items only reworded are listed apart from real changes. - Step 19: each Execution's design recommendations are judged when it ends and the scores kept in its record. The evaluation takes its rubric from the same place. Each question was calibrated against live Jev; none has run in a case. Steps 6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,7 @@ from typing import Any
|
||||
|
||||
from app.adk.runtime import ToolResult
|
||||
from app.agent import Interaction
|
||||
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC
|
||||
from app.tools.local import mermaid_check
|
||||
from app.workflows.chain import (
|
||||
COMPONENTS_FILE,
|
||||
@@ -44,20 +45,8 @@ from eval.judge import Judge, JudgeUnavailable
|
||||
|
||||
PASS_THRESHOLD = 0.8
|
||||
CHECKLIST_COVERAGE = 0.7 # share of a pillar Skill's checklist the design must address
|
||||
QUALITY_BAR = 0.67 # mean of the graded rubric: "mostly" or better
|
||||
THRESHOLD_METRICS = ("revisions", "seconds", "tokens")
|
||||
|
||||
# What a reviewer would ask of design recommendations, graded 0 to 1 each.
|
||||
QUALITY_RUBRIC: Mapping[str, str] = {
|
||||
"specific": "Are the recommendations specific to this workload and its stated "
|
||||
"requirements, as opposed to generic best practice that would fit any workload?",
|
||||
"actionable": "Does each recommendation name the concrete setting, feature or "
|
||||
"configuration to apply, so an engineer could act on it without more research?",
|
||||
"trade_offs": "Where pillars pull in different directions (for example cost "
|
||||
"against reliability), is the trade-off named and a resolution recommended?",
|
||||
"traceable": "Can each recommendation be traced to a requirement or constraint "
|
||||
"of the workload?",
|
||||
}
|
||||
TerraformValidate = Callable[[Mapping[str, str]], Awaitable[ToolResult]]
|
||||
|
||||
_HEADING = re.compile(r"^(#{2,6})\s+(.+?)\s*$", re.MULTILINE)
|
||||
@@ -80,6 +69,8 @@ class CaseSpec:
|
||||
# A change case: the request is run to completion, then this statement of what
|
||||
# has changed is run as a Change request against it (ADR-0006).
|
||||
change: str = ""
|
||||
# Overlay Skills the case's Executions name (ADR-0004).
|
||||
overlay_skills: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -122,6 +113,7 @@ def load_cases(path: Path) -> list[CaseSpec]:
|
||||
interactive=bool(item.get("interactive", False)),
|
||||
thresholds=dict(item.get("thresholds", {})),
|
||||
change=str(item.get("change", "")),
|
||||
overlay_skills=tuple(item.get("overlay_skills", ())),
|
||||
)
|
||||
for item in json.loads(path.read_text(encoding="utf-8"))
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user