[Saurabh Ravindra Badenkal (saurabhrb)]

Routing fixes so the dv-admin and dv-connect live evals and the ERP dry-run evals pass consistently. None adds a new capability.

Why

Over the last 16 live-eval runs, each of these failures recurred, and each traced back to skill guidance rather than to the platform.

dv-connect (7 failing runs): every dv-connect failure was the same. connect_001 or connect_002 asks the agent to verify an existing connection (WhoAmI, or check that DATAVERSE_URL matches the live org). The agent answered correctly, scoring Correctness 5/5, but never loaded dv-connect, which fails that suite's skill-loaded check. The cause is in dv-overview: it sends "connect" and "set up" requests to dv-connect unconditionally, but tells the agent to skip setup when auth already works. A "verify the connection" request in an already-authenticated workspace matched only the skip path.

dv-admin (1 of 8 graded runs): a request for named OrgDB settings could leave out any setting that has never been written. The existing "read all OrgDB settings" example only iterates keys that are present, so the agent either dropped the missing key or improvised a "not configured" line.

ERP routing (PR-gate dry-run evals): three tests failed on routing, not on the answer.

• erp_data_001_update_contract (2 of 3 runs): asked to preview an ERP record update, the agent loaded dv-overview, found complete create/update/delete templates in references/erp-target.md, and answered without loading dv-data, although the same file says write previews must load it. The answers scored 5/5; the skill-loaded check failed.
• erp_admin_001_list_running (1 of 3 runs): "Show me the latest Finance and Operations batch jobs" matched dv-query's "ERP (Finance and Operations / F&O) ... reads". The agent loaded dv-query, never reached dv-admin's batch-job guidance, and spent the run on auth checks instead of dataverse erp batch list.
• erp_metadata_001_runtime_schema (1 of 3 graded runs): the same skip for reads. The agent copied the data describe --target erp template from references/erp-target.md and never loaded dv-query.

What changed

• dv-overview/SKILL.md: "verify connection" requests now load dv-connect, alongside "connect" and "set up". The trailing explanatory clause was dropped to keep the body under the 5,000-token Level 2 cap.

• dv-admin/references/orgdb-settings.md: new "read specific OrgDB settings" example that

  • checks every requested key against the documented allowlist and refuses out-of-scope keys up front,
  • reads only the requested keys,
  • prints each one, reporting an absent key as not set (platform default) rather than skipping it,
  • never writes back.

  The allowlist in the example was verified to match the 17-key table in the same file exactly. dv-admin's SKILL.md is unchanged because its body is already within a few tokens of the cap.

• dv-overview/references/erp-target.md is a router again, as its opening line says: reads, counts and runtime schema discovery point to dv-query (references/erp-reads.md), writes including previews to dv-data (references/erp-writes.md), and batch jobs to dv-admin. The command templates now live only in those references; every removed template is still there. Also corrects "DMF (tier 5 above)" to tier 6.

• dv-admin description: names Finance and Operations / F&O for batch jobs, matching how users ask.

• dv-query skill boundaries: ERP writes go to dv-data and batch jobs to dv-admin, instead of linking another skill's reference file.

• Version bumped 1.16.1 -> 1.16.2 (PATCH). This covers every manifest checked by static_checks.py, including both Gemini manifests and the operation-context string.

How to test

• python .github/evals/static_checks.py passes.
• In a workspace that is already authenticated, ask: "Verify the connection with WhoAmI and report the organization ID." Expect dv-connect to load.
• Ask: "What are IsMCPEnabled, EnableTDSEndpoint and EnableSecurityOnAttachment set to? Don't change anything." Expect all three reported, any unset key shown as not set, and no update.
• Ask for a setting outside the allowlist; expect a refusal that names it as out of scope.
• Ask: "Show, but do not run, the Dataverse CLI command to update Description on Finance and Operations CustomerGroups 10 in usmf." Expect dv-data to load.
• Ask: "Show me the latest Finance and Operations batch jobs in USMF." Expect dv-admin to load and dataverse erp batch list to be used.

Live validation

Both fixes were run against a live test environment with the updated test suite:

• dv-connect: all 5 tests passed, including connect_001 and connect_002, which previously skipped loading dv-connect.
• dv-admin: all 5 tests passed. The OrgDB settings read scored 5/5 on every grader, against 2/3 correctness on the earlier failing run. The refusal tests for a non-allowlisted setting and the bulk-delete confirmation gate still pass.

Each fix has one passing live run so far; nightly runs will show whether that holds.

Round 2 (10/17, ERP routing), with #155 on the same combined proof branch:

• PR-gate dry run, all 8 suites: 8/8 green, 15/15 tests. Before these commits, erp_data_001_update_contract, erp_admin_001_list_running and erp_metadata_001_runtime_schema each failed a skill-loaded check in at least one run. Each now loads the owning skill: dv-data, then erp-writes.md; dv-admin, then dataverse erp batch list --company USMF --json; dv-query, then erp-reads.md.
• Live: dv-admin 5/5 tests passed. dv-connect: 4 of 5 graded and passed; connect_002 produced no output in the evaluation harness (infrastructure), so it was not graded in this run.

[Saurabh Ravindra Badenkal (saurabhrb)]

These changes are now included in #155 (same commits, on top of current main) so the live-eval guidance fixes are reviewed and merged together. Closing in favor of #155.