fix: a Check is judged by its own words, and a wrong verdict can be disputed
Some checks failed
validation / verify (push) Failing after 10s
Some checks failed
validation / verify (push) Failing after 10s
Case 06 reached its Revision limit because the validator failed a Check that a role is granted, for the identity also holding other roles, and the IaC author could only answer by recording the Check as unimplementable. The validator is told what its own words require; the author is told to dispute such a verdict and change nothing. The evaluation's naming standard for case 06 accepted only a variable inside each name; it now accepts names built from a local as well. Records the reruns: case 04 passed at 12 of 12, case 06 ended with warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -27,3 +27,21 @@ No evaluation check scores whether an overlay was followed; this was read from t
|
||||
## An evaluation check, later on 2026-10-09
|
||||
|
||||
A case may now carry `code_standards`: regular expressions the infrastructure code must contain or must not, scored as `code_follows_standards`. Case 06 has six: only approved regions, names carrying the environment from a variable, and the four labels. Run against the code case 06 produced earlier today, it passes.
|
||||
|
||||
## Rerun on 2026-10-10: case 06 failed, case 04 passed
|
||||
|
||||
Both rerun after the week's changes (`20261010T123446Z` case 04, `20261010T123450Z` case 06).
|
||||
|
||||
**Case 04, interactive: completed, 12 of 12 checks, 34 minutes, about $4.58,** one Revision of the Target architecture. Nothing in routing, intake, the guardrail or the writing of files as it goes broke it.
|
||||
|
||||
**Case 06, overlay: ended with warnings, 67%, 42 minutes, about $7.72.** The Target architecture reached its Revision limit of 3 and no guide was packaged. Two separate causes:
|
||||
|
||||
- **A verdict that was wrong, and no way to say so.** Check K-10 says a role is granted to the service identity. It was. The validator failed it because the identity also held three other roles that other Checks require. The IaC author, sent the failure, could change nothing, and recorded the Check as unimplementable with the reason "the validator insists on only this role". That sent the design back for a fault it did not have. The same Check passed in the validation between, on the same grants: the validator was not consistent. The other two Revisions were ordinary ones (labels and names the overlay asks for, and a resource whose name cannot contain hyphens).
|
||||
- **The evaluation's naming check was too narrow.** `code_follows_standards` wanted a variable with "env" in its name inside each resource name. The code built names from `local.name_prefix`, itself made from `var.workload` and `var.environment`, which follows the standard. The check now asks that names are built from a variable or local, and that the environment comes from a variable; this run's code and the earlier run's both pass it.
|
||||
|
||||
Changed, not yet run:
|
||||
|
||||
- The validator is told to judge each Check by its own words: a Check that something is granted or set passes when it is, whatever is beside it, unless it says "only". What else it thinks should hold is a proposed addition, never a failure.
|
||||
- The IaC author is told what to do when a Check failed and the code already meets its words: change nothing, keep the implementation entry, name the Check it disputes, and not record it as unimplementable.
|
||||
|
||||
An overlay also makes a design larger: this run ended with over 120 Checks, against 50 to 90 for the same request without one.
|
||||
|
||||
@@ -19,6 +19,7 @@ You produce the **deployment guidance** Deliverable: the Skill's task for genera
|
||||
- Take every environment-specific value (project, region, names, image references) from variables; embed no credentials or secrets.
|
||||
- Submit as soon as validation passes, with your content and no attachments: your working copy is attached for you. If a submission is refused, fix exactly what the refusal lists and submit again.
|
||||
- When `working_copy` lists files, they are your last attempt. When `feedback` lists failed Checks, read the files concerned and change only what those Checks need.
|
||||
- When `feedback` says a Check failed and your code already does what the Check's words say, the verdict is what is wrong, not the Check. Change nothing for it, keep its entry in `implementation.json`, and say in your content which Check you dispute and where the code meets it. Do not record it as unimplementable: that sends the design back for a fault it does not have.
|
||||
|
||||
Your working copy must hold `terraform/main.tf`, `terraform/variables.tf` and `implementation.json`, with `terraform/versions.tf` and `terraform/outputs.tf` when they help.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ includes: [specialist-protocol]
|
||||
You produce the **validation results** Deliverable: the Skill's pre-deployment validation, done statically without provisioning anything.
|
||||
|
||||
- Run the deployment dry-run with your tools: validate the Terraform, and preview it with a plan when you have permission to read live cloud resources. The tools already hold the submitted files, so pass them no files.
|
||||
- Give a verdict on every Check in the Target architecture, from the infrastructure code: pass, fail or not verifiable, each with the evidence you read. The Checks are the whole bar: judge the code against them and nothing else.
|
||||
- Give a verdict on every Check in the Target architecture, from the infrastructure code: pass, fail or not verifiable, each with the evidence you read. The Checks are the whole bar: judge the code against them and nothing else. Judge each Check by its own words. A Check that says a role is granted or a setting is set passes when it is; other grants or settings beside it do not fail it unless the Check says "only" or "no other". What you think should also be true is a proposed addition, never a failure.
|
||||
- Before you fail a Check because the code lacks a setting, confirm with `terraform_schema` that the resource type has that setting, when the tool is available. When it does not, the Check cannot be met by any code: give it not verifiable, with what the schema shows as the evidence. Give the same verdict to a Check that `implementation.json` records as unimplementable.
|
||||
- When `terraform_modules` is true the code may call approved published modules. Then judge a Check on a resource the module creates from the module input that sets it: read the value in the code, and confirm with `terraform_module` what that input sets or what its default is. The modules Skill, when you have one, says how.
|
||||
- Review the Target architecture's coverage once, completely. Go through every requirement and every design recommendation in turn; for each, list the conditions its text states and the Elements it applies to, and record a proposed addition for each one no Check verifies, naming the requirement or recommendation it serves. Report every gap in this pass: each later pass that finds one more costs a whole Revision of the design, the code and this validation. Propose nothing for what a requirement or recommendation lists under `outside_code`: that is application behaviour, an outcome measured in operation, or a process, and no Check on infrastructure code can verify it. Name those under your analysis heading as things to verify after deployment. A proposed addition is not a failure of the code.
|
||||
|
||||
@@ -448,8 +448,12 @@
|
||||
"forbid": "\"(?!europe-west[14]\")(?:us|europe|asia|australia|northamerica|southamerica|me|africa)-[a-z]+[0-9]+\""
|
||||
},
|
||||
{
|
||||
"name": "names carry the environment from a variable",
|
||||
"require": "name\\s*=\\s*\"[^\"]*\\$\\{(?:var|local)\\.[a-z_]*env[a-z_]*\\}"
|
||||
"name": "names are built from variables",
|
||||
"require": "name\\s*=\\s*\"\\$\\{(?:var|local)\\.[a-z_]+\\}-"
|
||||
},
|
||||
{
|
||||
"name": "the environment comes from a variable",
|
||||
"require": "var\\.environment\\b"
|
||||
},
|
||||
{
|
||||
"name": "label workload",
|
||||
|
||||
Reference in New Issue
Block a user