fix: a Check is judged by its own words, and a wrong verdict can be disputed
Some checks failed
validation / verify (push) Failing after 10s
Some checks failed
validation / verify (push) Failing after 10s
Case 06 reached its Revision limit because the validator failed a Check that a role is granted, for the identity also holding other roles, and the IaC author could only answer by recording the Check as unimplementable. The validator is told what its own words require; the author is told to dispute such a verdict and change nothing. The evaluation's naming standard for case 06 accepted only a variable inside each name; it now accepts names built from a local as well. Records the reruns: case 04 passed at 12 of 12, case 06 ended with warnings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -27,3 +27,21 @@ No evaluation check scores whether an overlay was followed; this was read from t
|
||||
## An evaluation check, later on 2026-10-09
|
||||
|
||||
A case may now carry `code_standards`: regular expressions the infrastructure code must contain or must not, scored as `code_follows_standards`. Case 06 has six: only approved regions, names carrying the environment from a variable, and the four labels. Run against the code case 06 produced earlier today, it passes.
|
||||
|
||||
## Rerun on 2026-10-10: case 06 failed, case 04 passed
|
||||
|
||||
Both rerun after the week's changes (`20261010T123446Z` case 04, `20261010T123450Z` case 06).
|
||||
|
||||
**Case 04, interactive: completed, 12 of 12 checks, 34 minutes, about $4.58,** one Revision of the Target architecture. Nothing in routing, intake, the guardrail or the writing of files as it goes broke it.
|
||||
|
||||
**Case 06, overlay: ended with warnings, 67%, 42 minutes, about $7.72.** The Target architecture reached its Revision limit of 3 and no guide was packaged. Two separate causes:
|
||||
|
||||
- **A verdict that was wrong, and no way to say so.** Check K-10 says a role is granted to the service identity. It was. The validator failed it because the identity also held three other roles that other Checks require. The IaC author, sent the failure, could change nothing, and recorded the Check as unimplementable with the reason "the validator insists on only this role". That sent the design back for a fault it did not have. The same Check passed in the validation between, on the same grants: the validator was not consistent. The other two Revisions were ordinary ones (labels and names the overlay asks for, and a resource whose name cannot contain hyphens).
|
||||
- **The evaluation's naming check was too narrow.** `code_follows_standards` wanted a variable with "env" in its name inside each resource name. The code built names from `local.name_prefix`, itself made from `var.workload` and `var.environment`, which follows the standard. The check now asks that names are built from a variable or local, and that the environment comes from a variable; this run's code and the earlier run's both pass it.
|
||||
|
||||
Changed, not yet run:
|
||||
|
||||
- The validator is told to judge each Check by its own words: a Check that something is granted or set passes when it is, whatever is beside it, unless it says "only". What else it thinks should hold is a proposed addition, never a failure.
|
||||
- The IaC author is told what to do when a Check failed and the code already meets its words: change nothing, keep the implementation entry, name the Check it disputes, and not record it as unimplementable.
|
||||
|
||||
An overlay also makes a design larger: this run ended with over 120 Checks, against 50 to 90 for the same request without one.
|
||||
|
||||
Reference in New Issue
Block a user