feat: prove the lessons, check overlays in evaluation, and settle the cost steps
Some checks failed
validation / verify (push) Failing after 13s
Some checks failed
validation / verify (push) Failing after 13s
- Schema review: a dotted path is judged only when it starts at a name the resource type has, so a database flag or a service name is no longer read as a missing setting. Lesson L-04, which worked around that, is retired. - Skill routing: in a Change request the question says what changed. - Evaluation: a case may list code_standards the infrastructure code must meet; case 06 checks its overlay's regions, naming and labels. - Model tiers (iteration 5 step 6): a lighter model for a role's first attempt, named in the provider bundle. Measured on case 03 and left off: Flash saved five cents on the IaC author and took twice the requests. - Steps 7 and 9 decided against on the numbers, recorded in the plans. - Plans: the runs of cases 05 and 06, and the first proof of the lessons on case 03 (no Revision, 22 minutes against 28, quality held). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -43,9 +43,11 @@ def load_roles(
|
||||
specialist_budget: int = 15,
|
||||
include_preview_mcp: bool = True,
|
||||
lessons: bool = True,
|
||||
model_tiers: bool = False,
|
||||
) -> dict[str, RoleSpec]:
|
||||
"""`lessons` off leaves the lessons Skill out of every role, so an evaluation
|
||||
can run with and without what has been learned."""
|
||||
can run with and without what has been learned. `model_tiers` on gives a role
|
||||
the lighter model the bundle names for its first attempt."""
|
||||
specialists = {spec.specialist: kind for kind, spec in DEPENDENCY_GRAPH.items()}
|
||||
required = {ORCHESTRATOR, PILLAR_ADVISOR, REPLY_WRITER, *specialists}
|
||||
optional = {LESSON_CURATOR}
|
||||
@@ -74,6 +76,13 @@ def load_roles(
|
||||
roles[name] = dataclasses.replace(
|
||||
roles[name], skills=(*roles[name].skills, bundle.lessons_skill)
|
||||
)
|
||||
if model_tiers:
|
||||
for name, model in bundle.models.first_attempt.items():
|
||||
if name not in roles:
|
||||
raise RoleConfigError(
|
||||
f"'models.first_attempt' names {name!r}, which is not a role"
|
||||
)
|
||||
roles[name] = dataclasses.replace(roles[name], first_attempt_model=model)
|
||||
unexpected = sorted(set(roles) - required - optional)
|
||||
if unexpected:
|
||||
raise RoleConfigError(
|
||||
|
||||
@@ -34,6 +34,9 @@ class RoleSpec:
|
||||
model: str | None = None
|
||||
# Seconds one model request may take; None takes the runtime's default.
|
||||
model_timeout_seconds: float | None = None
|
||||
# A lighter model for the first attempt at a Deliverable, when the bundle
|
||||
# names one and model tiers are on (iteration 5 step 6).
|
||||
first_attempt_model: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
||||
Reference in New Issue
Block a user