feat: prove the lessons, check overlays in evaluation, and settle the cost steps
Some checks failed
validation / verify (push) Failing after 13s
Some checks failed
validation / verify (push) Failing after 13s
- Schema review: a dotted path is judged only when it starts at a name the resource type has, so a database flag or a service name is no longer read as a missing setting. Lesson L-04, which worked around that, is retired. - Skill routing: in a Change request the question says what changed. - Evaluation: a case may list code_standards the infrastructure code must meet; case 06 checks its overlay's regions, naming and labels. - Model tiers (iteration 5 step 6): a lighter model for a role's first attempt, named in the provider bundle. Measured on case 03 and left off: Flash saved five cents on the IaC author and took twice the requests. - Steps 7 and 9 decided against on the numbers, recorded in the plans. - Plans: the runs of cases 05 and 06, and the first proof of the lessons on case 03 (no Revision, 22 minutes against 28, quality held). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,6 @@
|
||||
"""Scoring a finished Execution against a benchmark case, without models or network."""
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
@@ -772,3 +773,49 @@ def test_a_functional_requirement_is_reached_through_its_components_elements():
|
||||
|
||||
assert by_name["change_is_recorded"].detail == "FR-1 changed"
|
||||
assert by_name["changed_requirements_reach_checks"].passed
|
||||
|
||||
|
||||
# --- standards an overlay Skill sets reach the code --------------------------------------
|
||||
|
||||
STANDARDS = (
|
||||
{"name": "approved regions", "forbid": r'"us-[a-z]+[0-9]+"'},
|
||||
{"name": "label owner", "require": r"\bowner\b\s*="},
|
||||
)
|
||||
|
||||
|
||||
def code(text: str) -> dict:
|
||||
return {"content": "guide", "attachments": {"terraform/main.tf": text}}
|
||||
|
||||
|
||||
async def test_code_that_follows_the_cases_standards_passes():
|
||||
case = dataclasses.replace(CASE, code_standards=STANDARDS)
|
||||
labelled = code('region = "europe-west1"\nlabels = { owner = var.owner }')
|
||||
|
||||
result = await score_execution(
|
||||
case, execution(deployment_guidance=labelled), HEADINGS, passing_terraform
|
||||
)
|
||||
|
||||
assert check(result, "code_follows_standards").passed
|
||||
|
||||
|
||||
async def test_a_broken_standard_is_named():
|
||||
case = dataclasses.replace(CASE, code_standards=STANDARDS)
|
||||
|
||||
result = await score_execution(
|
||||
case,
|
||||
execution(deployment_guidance=code('region = "us-central1"')),
|
||||
HEADINGS,
|
||||
passing_terraform,
|
||||
)
|
||||
|
||||
broken = check(result, "code_follows_standards")
|
||||
assert not broken.passed
|
||||
assert (
|
||||
broken.detail == 'approved regions: found "us-central1"; label owner: not found'
|
||||
)
|
||||
|
||||
|
||||
async def test_a_case_with_no_standards_has_no_such_check():
|
||||
result = await score_execution(CASE, execution(), HEADINGS, passing_terraform)
|
||||
|
||||
assert all(c.name != "code_follows_standards" for c in result.checks)
|
||||
|
||||
Reference in New Issue
Block a user