feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)

Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
  Skill from the folders the deployment configures; recorded with its version.
  An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
  one without, and triage of evidence an existing lesson already covers.

Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
  its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
  the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
  real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
  the scores kept in its record. The evaluation takes its rubric from the same
  place.

Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 17:45:56 +01:00
parent 844987587c
commit 567aede5ca
38 changed files with 1956 additions and 58 deletions

View File

@@ -6,18 +6,21 @@
status list every proposal and where it stands.
publish write every approved proposal (Status: ready-for-agent) into the
provider's lessons Skill.
prove compare an evaluation run without lessons with one run with them,
and write the comparison into every published proposal.
retire remove a lesson from the Skill, by id.
Usage:
.venv/bin/python -m eval.curate_lessons propose [--from <folder> ...] [--min 2]
.venv/bin/python -m eval.curate_lessons status
.venv/bin/python -m eval.curate_lessons publish
.venv/bin/python -m eval.curate_lessons prove --without <report.json> --with <report.json>
.venv/bin/python -m eval.curate_lessons retire L-03
A person approves a proposal by editing its file: set `Status: ready-for-agent`, or
`wontfix` to reject it. Nothing is published without that. After publishing, prove
the lesson: run the evaluation with LESSONS_ENABLED=false and again with it on, and
record the comparison under the proposal's `## Proof`. Retire a lesson that does
the lessons: run the evaluation with LESSONS_ENABLED=false and again with it on, on
the same cases, then run `prove` with the two reports. Retire a lesson that does
not hold quality and lower Revisions, time or tokens.
`propose` costs model usage. Lesson candidates are client data; they are read
@@ -31,9 +34,11 @@ import sys
from collections.abc import Sequence
from pathlib import Path
from app.agent import open_decider
from app.config import get_settings
from app.learning.curator import LESSON_CURATOR, curate
from app.learning.evidence import group_candidates, identifiers, read_candidates
from app.learning.proof import compare, record, render
from app.learning.proposals import ISSUES, load
from app.learning.skill import NotApproved, publish, published, retire
from app.providers import ProviderBundle, load_provider
@@ -79,14 +84,21 @@ async def propose_lessons(arguments: Sequence[str]) -> int:
skill_dirs={name: meta.path for name, meta in bundle.skills.items()},
model_timeout_seconds=settings.MODEL_TIMEOUT_SECONDS,
)
proposed = await curate(
runtime,
roles[LESSON_CURATOR],
groups,
executions=len(records),
identifiers=identifiers(records),
folder=ROOT / ISSUES,
)
decider = open_decider(settings)
try:
proposed = await curate(
runtime,
roles[LESSON_CURATOR],
groups,
executions=len(records),
identifiers=identifiers(records),
folder=ROOT / ISSUES,
decider=decider,
)
finally:
aclose = getattr(decider, "aclose", None)
if aclose:
await aclose()
for proposal in proposed:
print(f" {proposal.id} [{proposal.deliverable}] {proposal.title}")
print(
@@ -127,6 +139,23 @@ def publish_approved() -> int:
return 0
def prove_lessons(arguments: Sequence[str]) -> int:
without, with_lessons = (
_options(arguments, "--without"),
_options(arguments, "--with"),
)
if not without or not with_lessons:
print("Give both reports: --without <report.json> --with <report.json>.")
return 2
first, second = Path(without[0]), Path(with_lessons[0])
result = compare(first, second)
proof = render(result, first, second)
recorded = record(load(ROOT / ISSUES), proof)
print(proof)
print(f"\nRecorded in {len(recorded)} published proposals.")
return 0 if result["helps"] else 1
def retire_lesson(arguments: Sequence[str]) -> int:
if not arguments:
print("Name the lesson to retire, such as L-03.", file=sys.stderr)
@@ -145,6 +174,8 @@ def main(arguments: Sequence[str]) -> int:
return show_status()
if command == "publish":
return publish_approved()
if command == "prove":
return prove_lessons(rest)
if command == "retire":
return retire_lesson(rest)
print(__doc__, file=sys.stderr)